WPS8312 Policy Research Working Paper 8312 Lessons from Reforming Financial Management Information Systems A Review of the Evidence Ali Hashim Moritz Piatti-Fünfkirchen Independent Evaluation Group January 2018 Policy Research Working Paper 8312 Abstract Financial management information systems are a sine qua diagnostic phase, (2) systems development lifecycle, and non in public financial management and play a founda- (3) coverage and utilization. The paper argues that reaching tional role in the execution of the budget. Recognizing their the financial management information systems production potential contribution to fiscal discipline, the strategic allo- frontier requires optimization across these dimensions, and cation of resources, and operational efficiency, significant that a programmatically coherent approach is required to time and resources have been invested by the World Bank realize fully the expected improvements in budget manage- and other development institutions into such systems across ment. The study identifies a set of lessons on the various the world. However, the reform of financial management stages that are mapped against the framework by triangu- information systems tends to be complex, and the evi- lating findings from a systematic review of the financial dence base of causal effects and mechanisms is thin. This management information systems literature, field-based study develops a framework that outlines the various steps project-level evaluations and protocol based case studies, and involved in reform that illustrate how change is expected a comprehensive desk review of the World Bank financial to happen. Three major dimensions were identified: (1) management information systems project documentation. This paper is a product of the Independent Evaluation Group. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank.org. The authors may be contacted at moritzpiatti@ worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Lessons from Reforming Financial Management Information Systems A Review of the Evidence Ali Hashim Moritz Piatti-Fünfkirchen Contact: Moritz Piatti-Fünfkirchen is the corresponding author and can be reached at mpiatti@worldbank.org. Key words: Governance and Public Sector Management, Public Finances, Public Financial Management, Public Expenditure Management, Financial Management Information Systems, FMIS, IFMIS. JEL codes: H61 (Budget systems), H83 (Public administration – public sector accounting and audits), O19 (Role of International Organizations), O33 (Technological Change). Contents OVERVIEW............................................................................................................................................. 5 1. INTRODUCTION AND MOTIVATION ..................................................................................... 11 An Overview of the World Bank FMIS Engagement ............................................................................................... 12 Motivation and Scope .............................................................................................................................................. 14 Methods and Limitations ......................................................................................................................................... 15 2. A FRAMEWORK FOR FINANCIAL MANAGEMENT INFORMATION SYSTEMS.................. 18 3. ANALYSIS AND FINDINGS .................................................................................................... 20 The Diagnostic Phase ............................................................................................................................................. 20 Policy and institutional underpinnings ................................................................................................................................. 23 Control protocols.................................................................................................................................................................. 25 The System Development Life Cycle ...................................................................................................................... 26 Project management............................................................................................................................................................ 26 Functional processes and systems design .......................................................................................................................... 27 FMIS procurement ............................................................................................................................................................... 30 System implementation ....................................................................................................................................................... 34 Ongoing systems operations and maintenance .................................................................................................................. 43 FMIS Coverage and Utilization ............................................................................................................................... 47 The World Bank’s Role in FMIS Reform ................................................................................................................. 51 4. CONCLUDING REMARKS ...................................................................................................... 56 REFERENCES...................................................................................................................................... 59 iii Acknowledgments This study is a product of the Independent Evaluation Group. The team was led by Moritz Piatti-Fünfkirchen under the supervision of Pablo Fajnzylber and Auguste Kouame. It was authored jointly by Ali Hashim and Moritz Piatti-Fünfkirchen with equal contributions. The team is grateful to Mark Sundberg and Guenter Heidenhof who sponsored the report. The team is thankful to extremely valuable peer review comments from Richard Allen, Allen Schick, and Gerardo Uña, who have improved the report significantly in both substance and structure. The report has gone through various iterations to carefully reflect these in the final version. Valuable comments and feedback was also received from the World Bank FMIS Community of Practice, with comments from Leah April, Jim Brumby, Cem Dener, Verena Fritz, and Eduardo Talero. Very important inputs were received from peer and panel reviewers of the various Project Performance Assessment Reports and case studies who include Iradj Alikhani, Guy Anderson, Salvatore-Schiavo Campo, Michael Stevens, and Clay Wescott. The team is also grateful to Jozef Leonardus Vaessen and Lodewijk Smets for providing methodological guidance. The team is particularly thankful to Arun Arya for his support throughout the process and his contributions on Ghana. iv Overview What is an FMIS and what is its purpose? A government’s capacity to manage its public finances is central to its ability to deliver services. Financial management information systems (FMIS) are among the basics that facilitate this as they “support management of public sector budgetary, accounting, treasury, and public debt management processes as well as generate corresponding reporting documents” (Uña and Pimenta 2015, p.282). FMIS systems are a set of automation solutions that allow government finance and accounting staff to carry out their day-to-day operational tasks. This enables them to plan, prepare and approve budgets, approve and verify commitments, issue payment orders and payments, monitor and report on financial resources collected, and develop appropriate resource allocation and borrowing strategies. As such, the potential of such systems to improve budgetary management outcomes in terms of fiscal discipline, allocative efficiency, and operational efficiency is widely recognized by the literature and practitioners alike. Recognizing the potential benefits, the World Bank invested significant resources into the development and implementation of such systems around the world. Motivation and Scope. FMIS projects tend to be complex and prone to a wide range of implementation challenges. Despite the relevance of FMIS systems to expenditure management the literature on causal effects and mechanisms from FMIS on budgetary outcomes is relatively thin and the quality uneven. The aim of this study is to identify practical issues that may help practitioners during the design, implementation, and operationalization of FMIS systems for the achievement of improved results, such as a reduced variance between authorized and actual expenditure, timely in-year accounts and annual financial reports, and improved information available to the legislative. Data and Methods. This study draws on a systematic review of the FMIS literature, field based project-level evaluations and protocol based case studies, and a comprehensive desk review of the World Bank FMIS project documentation. Through a triangulation process of the various sources, findings were identified and mapped against a theoretical framework. Case studies were used to inform causal mechanisms on what worked, why, and under what circumstances. The convergence of conclusions across sources and the at times axiomatic nature of arguments addresses concerns of external validity of the findings. An FMIS Framework. A theory of change was developed from the literature that visualizes the various steps involved in FMIS implementation, which is important as it illustrates how change is expected to happen. It also visualizes how the various 5 5 dimensions are interrelated and that a programmatically coherent engagement is required in order to fully realize the expected improvements in budget management. The three major dimensions identified were: (1) the diagnostic phase; (2) the systems development lifecycle; and (3) coverage and utilization. The report argues that reaching the FMIS production frontier requires optimization across all dimensions, and a focus on one alone may not be enough. The structure of the report mimics the framework, and the analysis of the portfolio, case studies and review of the literature led to the following findings: The Diagnostic Phase. It is important to clearly determine the rational for implementing an FMIS, and identify the problems that the proposed system intends to address as the nature and scope of the investment will largely depend on this. Here it is critical that the diagnostic is comprehensive covering all relevant aspects of budget management as a partial diagnostic could be misleading. A diagnostic focusing only on accounting issues for example may lead to solutions that do not adequately address larger relevant budget management deficiencies. Overall, maintaining a focus on effective budget management is critical, even if the diagnostic points to other immediate deficiencies. The effectiveness of an FMIS as a budget management tool depends, not only on its technical robustness, but also the policy and institutional environment under which it operates. In line with findings from the 2016 World Development Report, FMIS systems too need analog complements to make the FMIS effective, and protect against downside risks. These factors should be considered in a diagnostic as they are often referred to as preconditions for FMIS effectiveness. Even advanced systems may not facilitate desired budget management improvements without these analog complements. Similarly, a review of existing control protocols is essential. Setting up an automated FMIS without the necessary control functionalities could speed the hemorrhage of resources instead of controlling it. The System Development Life Cycle. There are multiple stages involved in getting an FMIS operational, which are commonly referred to as the systems development life cycle. These include process and system design, system procurement, actual systems implementation, and system maintenance. Throughout this process effective project management and strong government commitment, especially from the functional side is critical. Government commitment can be fostered through well designed project management structures. Further, training and change management considerations during the FMIS reform have been widely acknowledged as important. 6 An effective system design is one that is cognizant of larger budget management issues that follows functional and business process requirements of government. System designs that follow predominantly technical considerations, were found to be less effective for solving budget management problems. The procurement of FMIS systems is complex and warrants careful consideration of a number of issues. Systems specifications in tender documents, the design of the consultancy package, a consolidated systems implementation plan, and contract management experience, were among the most important factors in the process. Carefully weighing the benefits against risks and costs is important for selecting an appropriate strategy for application software development. Should this include changing technology platforms midway, this involves substantial risks, including loss of human capacity, diversion of reform focus, and neglect of potentially necessary improvements of the legacy system during the transition period. Experience suggests that system implementation strategies that are strategic and take a phased approach tend to be more successful than attempting simultaneous implementation of a wide set of functionalities as this may overstretch client capacity and dilute the reform momentum. The literature suggests that modules necessary for execution and reporting should be prioritized. Some countries have experimented with taking a modular approach in systems implementation, which holds promise in being more cost-effective. The implementation strategy may also benefit from prioritizing transaction processing before investing into financial operations and management and reporting layers, since transaction processing is foundational and a prerequisite for the proper functioning of the other layers. If all these are in place, policy makers can use FMIS reports to take strategic decisions regarding the allocation of resources whilst maintaining a fiscally prudent stance. For this however, the system requires full data integrity, which in turn necessitates a reliable transactions processing layer. The usefulness of the financial operations and reporting layer is contingent on this. The systems deployment strategy across levels of government is a critical phase in implementation. This review found a striking pattern in the transactions profile, with only few transactions making up the bulk of the volume of the budget. This information could be used strategically for early results in the implementation process. Once operational, adequate budgetary provisions for maintenance and updates are essential for the sustainability of the investment. In some cases, the lack of doing so has resulted in a failure to install timely upgrades, insufficient capacity to manage 7 7 transaction requirements, and system exposure to various security risks. Another important factor for sustainability is the continuous availability of technical expertise, though it may be difficult to attract at regular government pay scales. FMIS Utilization and Coverage. Benefits from the FMIS to budget management can only accrue to the funds that are actually routed through the system. For the reform process, it is important to be transparent about what transactions are subjected to FMIS internal controls, and what are only posted to the general ledger after they have occurred. The share of the budget that is routed through the FMIS, and benefits directly from its functionalities, could serve as a good proxy for FMIS contribution to improved budget management. The analysis also shows that the effectiveness of a system can be undermined if commitment management is not adequately exercised or budget releases are delayed. The World Bank’s Role in FMIS Projects. The World Bank can potentially play a transformative role across all these dimensions. The review identified the following key factors for a successful engagement: realism with regards to time and costs; having experienced specialists available; continuity of staff; the use of appropriate lending instruments; and adequate monitoring and evaluation to undertake mid-course corrections when necessary, and allow for attribution of FMIS investments to larger public financial management objectives. A detailed list of the findings is summarized below: Overview of the Findings The Diagnostic Phase • Finding 1: It is important to clearly determine the rational for implementing an FMIS, and identify the problems that the proposed system intends to address. The nature and scope of the investment will largely depend on this. Maintaining an overall focus on effective budget management is critical, even if immediate deficiencies may be elsewhere. • Finding 2: The effectiveness of an FMIS as a budget management tool depends - not only on its technical robustness - but also the policy and institutional environment under which it operates. These factors should be incorporated in a diagnostic and can be considered as preconditions as even advanced systems may not facilitate desired budget management improvements without these. Key preconditions include a functioning, comprehensive treasury single account, standard budget classification structures that are consistent across all levels of government, and an economic classification segment that is a subset of the chart of accounts. 8 • Finding 3: A review of existing control protocols is essential. FMIS systems expedite the speed of transactions. Setting up a system without adequate controls could result in a situation where it is easier to do the wrong things faster. The System Development Lifecycle Project Management. • Finding 4: Government commitment is an essential success factor for FMIS implementation. Appropriately designed project management structures can help garner government commitment. Functional Processes and Systems Design • Finding 5: A system design should follow functional requirements of government rather than technical considerations. • Finding 6: FMIS projects are likely to be more effective if framed as expenditure management projects, rather than accounting projects. Procurement • Finding 7: Lack of precise systems specifications in the tender documents could cause suppliers to build large risk mitigation factors into their pricing, which can result in excessively high bids. • Finding 8: It is important to weigh costs against benefits when choosing an application software strategy. System Implementation • Finding 9: Taking a phased approach in FMIS implementation is an important factor of success. Fundamental modules necessary for budget execution and reporting should be prioritized. • Finding 10: A small percentage of transactions make up a large share of the budget. Capturing such high-value transactions and subjecting them to budgetary controls may facilitate improved fiscal management early in implementation. • Finding 11: Deployment of FMIS systems is costly and logistically complex. Taking a sequenced approach and utilization of web-based technologies could help. • Finding 12: Reliable transactions processing is fundamental to the integrity of a system. It provides the data used for financial operations and management reporting. It should thus be prioritized in the implementation process. • Finding 13: Technical training and training for end users and managers is essential throughout the system lifecycle. End users need to feel comfortable using the systems. It is important to ensure that staff in the implementing agencies recognizes the inevitability of change. 9 9 Ongoing Systems Operation and Maintenance • Finding 14: Along with the investment costs incurred in first setting up an FMIS, provisions should be made to cover expected recurrent costs to keep the system operational. As a rule of thumb, this was estimated at 10-15 percent of capital cost. • Finding 15: Non-availability of technical expertise has become a key vulnerability for system sustainability in some countries. Options to ensure availability should be incorporated during project design. FMIS Utilization and Coverage • Finding 16: Benefits from an FMIS only apply to the funds routed through it. The share of the budget subjected to FMIS ex-ante controls could be used as good proxy for FMIS contribution to effective budget management. • Finding 17: Benefits from an FMIS will accrue only if the system’s control protocols are diligently applied. The practice of incurring budgetary commitments outside the system is prone to the accumulation of arrears. • Finding 18: Commitment management is essential for budgetary control. • Finding 19: Delayed budget releases may cause spending units to by-pass the system to avoid budgetary controls. Procedures to improve budgetary allocations from day one are important to maintain confidence and utilization of the system. • Finding 20: Implementing advanced budgeting methodologies without an operational budget execution system is unlikely to achieve significant results. The World Bank’s Role in FMIS Projects • Finding 21: FMIS projects can be costly and lengthy. World Bank projects should be realistic during appraisal and choose adequate lending instruments. • Finding 22: The World Bank team needs to have experienced specialists available to advise the client on important technical issues during the design and implementation phases. The specialists should also be familiar with procurement practices and procedures for information technology procurements and the World Bank rules under which they will need to be applied. Continuity of World Bank staff from project design throughout implementation is a crucial success factor. • Finding 23: Development policy financing, though underused, can provide critical leverage to address political economy constraints. • Finding 24: The attribution to improved PFM outcomes can be facilitated through good monitoring and evaluation frameworks. 10 1. Introduction and Motivation 1.1 A government’s capacity to manage its public finances is central to its ability to deliver services. Financial management information systems (FMISs) are among the basics that facilitate this, as they “support management of public sector budgetary, accounting, treasury, and public debt management processes as well as generate corresponding reporting documents” (Uña and Pimenta 2015, p.282).1 There are multiple definitions of FMISs varying in scope in the literature (see Dorotinsky and Watkins (2011) for a review), which have been summarized as “computerized systems that track government expenditures and payment processing, and report accordingly” (Schiavo-Campo 2017, p.187). As such, FMISs broadly consist of computer programs, databases, and associated processes, procedures, and technology platforms that enable government finance and accounting staff to conduct their day-to-day operational tasks.2 The information collected in the system databases as the transactions process enables government finance managers to plan, prepare, and approve budgets, approve payments, monitor and report on financial resources collected, and develop appropriate resource allocation and borrowing strategies. Government auditors can access this transactional data to audit operations. 1.2 The potential of FMISs to contribute significantly to budget management outcomes has been documented widely by the literature (see for example World Bank 1998, Dorotinsky and Matsuda 2001, Diamond and Khemani 2005, Hove and Wynne 2010, Dener et al 2011, Dener and Min 2013, Dorotinsky and Watkins 2013, Hashim 2014, and Uña and Pimenta 2015) and practitioners alike. Among the arguments that are put forward are that improved recording and processing of government financial transactions allows ready access to reliable financial data. This supports transparency and accountability of the executive to parliament and the general public. Further, an FMIS strengthens financial controls, potentially facilitating a comprehensive and current picture of commitments and expenditure on a continuous basis. Once a commitment is made, the system would be able to trace all the stages of the transaction processing from budget releases, commitment, purchase, payment request, reconciliation of bank statements, and accounting of expenditure. This then allows a 1FMISs are frequently and loosely referred to as Integrated FMISs or IFMIS, regardless of whether they are truly integrated and share common databases. 2These include originating, receiving, recording, and processing transactions related to budget authority; requests for financial resources; payment of bills, loan, and grants; assessments of taxes, duties, and other levies; recording receipts; and managing deposits in government bank accounts. 11 11 comprehensive picture of budget execution. Finally, it provides the information to ensure improved efficiency and effectiveness of government financial management. Generally, increased availability of comprehensive financial information on current and past performance assists budgetary control and improved economic forecasting, planning, and budgeting (Diamond and Khemani 2005). Dorotinsky and Matsuda (2001) eloquently map how FMISs can contribute to the high-level PFM objectives of macro-fiscal stability, strategic allocation of resources, and operational efficiency. 1.3 Given the rapid pace in technological change and innovations in the field, there are significant opportunities to reap greater digital dividends. For example, innovations in digital payments could fundamentally change the way governments transfer funds, which has the potential to expedite basic PFM functions and strengthen accountability (Cangiano et al. 2017). While this report embraces the enthusiasm of such opportunities, it also points out the foundational role of analog complements3 – the policy and institutional environment necessary to not only enable technological innovations, but also to safeguard against potential risks. An Overview of the World Bank FMIS Engagement 1.4 The World Bank has invested significant time and resources into FMIS operations. In the last 30 years (since 1985) about US$ 4,079 million has been committed through 134 operations across 74 countries.4 There has been a clear upwards trend in the engagement following an increased recognition of the potential role of ICT in government. By the turn of the century, about six operations were approved annually averaging about US$33.3 million each. FMIS projects have been prominent in the Governance Global Practice (GP) portfolio. Since 2000, 28 percent of all projects mapped to the Governance GP had at least one FMIS component, making up about 17 percent of the GP’s overall commitment. 3See the World Development Report 2016 (World Bank 2016) for a more expansive discussion of the role of analog complements as facilitating factors for ICT innovations. 4These figures reflect total project amounts. As FMIS are usually part of a larger PFM engagement in most cases not all project funds are dedicated to FMIS related activities. Operations include active, closed, and pipeline projects. 12 Figure 1. Trend of WB FMIS engagement 10 400 Commitment (US$ millions) Number of FMIS operations 360 8 320 280 6 240 200 4 160 120 2 80 40 0 - 1985 1990 1995 2000 2005 2010 2015 2020 Fiscal Year Number of operations Commitment amount Source: World Bank FMIS Community of Practice Database and Business Intelligence. 1.5 The Africa region has dominated the FMIS engagement both in terms of number of operations as well as total loan amounts. Close to two billion US$ were invested in the region, making up about 45 percent of total commitment amounts. In terms of numbers, the Africa region made up 38 percent of total projects, an indication that the average loan size was at about US$ 36.0 million, also the highest across all regions. Both in terms of lending and commitment the Africa region is followed by Latin America and the Caribbean and subsequently the Europe and Central Asia region. On the other hand, the Middle East and North Africa region stands out with relatively limited engagement. A regional overview of the engagement is provided in table 1. While there is a potentially important role for development policy finance (DPF) operations, the predominant vehicle for engagement has been through investment loans reflecting information technology (IT) infrastructure needs and technical assistance. Table 1 Regional breakdown of FMIS engagement (FY1985-FY2017) Region Total commitment (US$ m) Number of projects Mean project size (US$ m) AFR 1,833.5; (45%) 51; (38%) 36.0 LCR 685.5; (17%) 33; (25%) 33.4 ECA 635.2; (16%) 19; (14%) 32.4 SAR 486.1; (12%) 15; (11%) 30.2 EAP 332.6; (8%) 11; (8%) 21.2 MNA 105.9; (3%) 5; (4%) 20.8 Total 4,078.8; (100%) 134; (100%) 30.4 Source: World Bank FMIS Community of Practice Database and Business Intelligence. 13 13 1.6 Recognizing the potential payoffs, World Bank FMIS projects have had high- level objectives along the lines of improved service delivery, improved public sector effectiveness, better expenditure management, or improved transparency and accountability. Reviewing all objectives of the FMIS projects5 shows that the majority (40.5 percent) were focused on accountability, fiduciary responsibility, and oversight, followed by transparency (39.7 percent), efficiency (35.7 percent), and effectiveness (20.6 percent).6 This is summarized in table 2 below. Table 2 FMIS objectives by area of focus (FY1985-FY2017) Area of focus Number of projects Share Accountability, fiduciary responsibility and oversight 51 38.1% Transparency 50 37.3% Efficiency 45 33.6% Effectiveness 26 19.4% Capacity and coverage 20 14.9% Public service delivery and access to services 12 9.0% Credibility 9 6.7% Allocation 7 5.2% Quality and timeliness of reporting and information 5 3.7% Utilization 5 3.7% Equity in resource sharing and inclusiveness 3 2.2% Reliability 3 2.2% Source: Authors’ review and classification. Motivation and Scope 1.7 The above sections have established that there is broad agreement on the potential of FMISs to contribute to larger PFM objectives and that significant time and resources have been dedicated in the pursuit of these. The nature of FMIS implementation is however complex (Dener et al 2011, Hendricks 2012, and Hashim 2014) and the academic literature remarkably thin on causal effects of FMIS investments on intermediate or final outcomes (see table 2 for examples). A Boolean search in Web of Science found five studies which were however of relatively low quality and none were of experimental or quasi-experimental nature. Combaz (2015, p.3) refers to this state of the FMIS literature as a “dearth of rigorous evidence”. There are however a 5 Project objectives were obtained from the IEG DataMart and coded following a grounded theory approach. 6 Categories per project are not mutually exclusive and thus add up to more than 100 percent. 14 significant number of studies, working papers, and guidance notes issued by development institutions and think tanks that document some of the experience with FMIS implementation and draw conclusions. A targeted google scholar search that is more inclusive with regards to working papers and gray literature identified 951 documents though widely ranging in relevance and rigor. 1.8 The aim of this study is to identify practical and operational issues that may arise during design, procurement, and implementation of FMIS systems and summarize lessons on what has worked through the triangulation of the various data points. Secondly, it is hoped that carefully outlining the various implementation steps in a framework will stimulate a conversation around the various dimensions and encourage practitioners to take a more programmatically coherent approach. 1.9 This report is an effort to make the vast World Bank experience with FMIS operations more readily accessible to practitioners engaged in such reforms to allow them to make better informed decisions or take calculated risks. Further, identifying critical dimensions through a framework could provide for an input for project design and the assessment of the programmatic coherence of the FMIS engagement in a country. It could also provide for an input into the development of M&E frameworks going forward, which are critical for documenting the FMIS contribution to larger PFM objectives. This report could serve as a reference point for practitioners in the World Bank and other development institutions, as well as government counterparts and their advisors leading such reform efforts. It could also be useful to contractors, consultants, and in-house information technology staff tasked with implementing an FMIS system. Methods and Limitations 1.10 The report draws on the FMIS literature, five purposefully selected case studies, and desk reviews of World Bank project documentation. These are discussed respectively in more detail below. 1.11 To identify the FMIS literature, a systematic search strategy was used drawing first on two search engines and secondly on forward and backward tracing of references from seminal reports. For academic literature, Web of Science7 was used, which found five studies which were however of relatively low quality and none were of experimental or quasi-experimental in nature. Google Scholar was used to cast a wider net and capture reports issued by international finance institutions and think tanks as well as gray literature, guidance notes, various case studies, and relevant evaluations. 7 Web of Science is an online subscription-based scientific citation indexing service. 15 15 The Google Scholar search identified 951 documents, though widely ranging in relevance and rigor. For both engines, a Boolean search was used with the following search terms: ‘FMIS’, ‘IFMIS’, ‘Financial Management Information System’, or ‘Financial Management System’. Results were limited to publication date 2000 and after. The articles selected for inclusion in this review were restricted to English-language studies only. The review focused primarily on evidence from low-income and middle-income countries, though some evidence was drawn from high-income countries where this complemented available evidence from LMICs. False positives and duplicates were identified and excluded after a title and abstract review. Studies considered of insufficient quality through e.g. lacking peer review process or data were excluded. Forward and backward tracing was confirmatory. The final list of studies used for the analysis (44) is provided in the list of references. 1.12 This study draws on five rigorous FMIS project level evaluations (IEG 2016a-e) that serve as purposeful case studies (Cambodia, Ghana, Malawi, Vietnam, and Zambia). While the standard objective based evaluation protocol was employed in the assessment, a deliberate and structured effort was made to capture the FMIS implementation experience through dedicated annexes. Case studies were required to systematically reflect on the design, implementation, and utilization experience of FMIS as per protocol and document key success factors and failure points. The case study design follows the guidance of Yin (2013) and with the purpose of informing causal mechanisms (rather than causal effects) on what worked during a systems life cycle and why. The intent here was to explore depth rather than breadth and boundedness and is by definition exploratory rather than affirmative. An extensive discussion of the merits and drawbacks of qualitative work can be found in Gerring (2004). The five case studies are not (and cannot be) representative of the entire portfolio and all regions and were not selected with that objective. Rather the intent was to derive in-depth qualitative information on causal mechanisms of systems at different stages in their lifecycle and varying degrees of sophistication and utilization. While the findings of these are context specific, an effort has been made to assess convergence with the literature and broader documentation of World Bank experience to address external validity concerns. 1.13 The experience of FMIS operations as documented in World Bank self- evaluations and IEG validations thereof was drawn upon and systematically coded following a grounded theory approach. This complements an earlier effort of doing so by Dener et al 2011 and extends the exercise until 2017. In total, 80 ICRs and ICR Reviews were reviewed. Appraisal documents for projects to which these lessons relate were consulted on an as-needed basis. A database of all FMIS operations was kindly made available to the team by the World Bank FMIS Community of Practice. 16 1.14 Drawing on nonrepresentative case studies invites criticisms of external validity – namely that findings from one context cannot naturally be extrapolated to another. To address external validity concerns, an effort was made to triangulate information from the literature, World Bank experience and case studies to assess convergence of findings and provide important contextual factors that may inform policy makers on the relevance of the findings to their specific situation. While context plays an important role, focusing on mechanisms (as this report does) and then judging whether a mechanism is likely to apply in a new setting has a number of practical advantages for policy making. The interested reader is referred to Bates and Glennerster (2017), which provides a four-step schematic on how to assess whether particular findings can be applied to a specific setting. 1.15 It is important to note the following limitations to this study: • The report relies extensively on World Bank evaluations and self-evaluations to draw lessons on what has worked. Any assessment of effectiveness and associated lessons is a function of the project as a whole to which FMIS usually only partially contributes. If no explicit effort is made to document the FMIS experience explicitly (as was done for the case studies), lessons on attribution of FMIS to project outcomes can be unclear, as they are mixed with other interventions. • Case studies are not representative of the entire portfolio and do not test hypotheses or derive causal effects. While an effort has been taken to make a case for external validity through the use of axioms and convergence of findings across data sources, the practitioner is still urged to exercise caution when applying findings to his/her specific context. The relevance of findings will still likely vary by whether countries have an anglophone or francophone history. It will also vary by type, sophistication, and application of FMIS solution. As such, it is important to reflect on ‘under what conditions’ what has worked to better inform the relevance of lessons to the specific need. For this purpose, a reference to the countries with hyperlinks to the relevant project documentation is provided in appendix A.8 • An effort of triangulation of data was made, but this was at times difficult given that the literature is sparse and that lessons in ICRs and ICR reviews are at times excessively generic. 1.16 The underlying project data were drawn from the Community of Practice that captures FMIS projects in investment operations. The data do not however capture 8 Appendix A can be accessed at this link. 17 17 information on development policy finance (DPF) operations. To that end, an OPCS database of policy actions was mined for FMIS-related key terms, but these projects were not vetted for inclusion by the same quality assurance process. Further the possibility of type II errors remains, as only prior actions (rather than the entire program document) were reviewed. 2. A Framework for Financial Management Information Systems 2.1 The literature, case studies, and World Bank experience point to a number of factors that are critical for FMIS effectiveness. These can be mapped into dimensions from which a framework is derived. Visualizing a theory of change is important, as it illustrates how change is expected to happen. It also visualizes how all dimensions are important and that a programmatically coherent engagement is required in order to achieve the expected improvements in budget management. Conversely, it implies that outcomes are unlikely to be achieved if one of the dimensions is violated. The framework is outlined in figure 2, followed by a detailed justification and description of the dimensions. Figure 2. A framework for financial management information systems 18 2.2 The Diagnostic Framework. Prior to procuring and implementing it is important to determine the rationale for implementing an FMIS and identify the problems that the system is intended to solve. It is also important to identify whether legal and institutional prerequisites are in place. For countries that have an operational FMIS (which is most but not all), this stage may involve identifying key system deficiencies before engaging in a second/third generation reform. Such a diagnostic should facilitate a cost-effective engagement by determining binding constraints across the various stages of the system lifecycle and take into consideration coverage and utilization questions. Similar to public investment management diagnostics (Rajaram et al. 2010), in FMIS a diagnostic can provide a ‘gap analysis’ of the actual system relative to a benchmark so as to identify structural aspects that may be weak and in need of attention. The importance of a diagnostic prior to an engagement is firmly embedded in the information systems literature (see for example Dorotinsky and Matsuda 2001, Diamond 2013, Dorotinsky and Watkins 2013, and Hashim 2014). The importance of a diagnostic (or lack thereof) also comes out firmly from recent IEG project evaluations and case studies (IEG 2016a-e) and was also documented in some World Bank project documentations (see for example experience from Bolivia, Cameroon, Colombia, the Democratic Republic of Congo, Ecuador, Honduras, Kazakhstan, Kenya, Madagascar, the Russian Federation, Sierra Leone, the Slovak Republic, Pakistan and Ukraine). There have been some efforts in developing diagnostic toolkits, including Hashim and Moon (2004) and more recently Hashim and Piatti (2016). 2.3 The System Development Life Cycle. There are multiple stages involved in getting an FMIS operational, which are commonly referred to as the systems development life cycle (SDLC). These include process review and design, system design, system procurement, actual systems implementation, and system maintenance (Sheti and Sharma 2013). These stages are discussed in more depth in appendix B.9 Most studies recognize the importance of the SDLC and are focused on at least some of its aspects in their analysis (see for example Hashim and Allan 1999, Diamond and Khemani 2005, Khan and Pessoa 2010, Dener et al 2011, Hashim 2014, Uña and Pimenta 2015, Combaz 2015, and Laizer and Suomi 2016). As almost all World Bank FMIS investment operations are centered around the implementation of FMIS, they naturally address various aspects of the SDLC and present lessons accordingly in the project documentation. 2.4 Coverage and utilization. Once the system is up and running it is axiomatic that it has to be utilized to be functional. The coverage and utilization stage in the theoretical framework covers scope of the system (geographical and functional), actual budget 9 Appendix B can be accessed at this link. 19 19 coverage, and the actual application of control protocols. The literature as well as World Bank experience on this aspect are however relatively thin. This dimension draws extensively on IEG evaluation findings and case studies, as well as World Bank documentation from experience in Malawi, Indonesia, Pakistan, and Vietnam. The analysis and findings of the report are structured around the framework. Framing the discussion in terms of these dimensions is important, as this is likely to be a critical condition for learning. In their seminal paper, Hanna et al (2012) find that a ‘failure to notice’ is a key binding constraint.10 Practitioners may excel in the dimensions they are working on, but are unlikely to optimize dimensions they fail to notice and are thus also unlikely to reach the production frontier. In terms of the framework, reaching the production frontier requires optimization across all dimensions, and a focus on one alone (e.g. the system lifecycle) may not be enough. 3. Analysis and Findings 3.1 This section triangulates findings from the various data points and illustrates these with in-depth country examples. All findings are systematically mapped against the framework and phases of the systems lifecycle. The section concludes with findings on the role of the World Bank, which cuts across the various stages. The aim is to identify practical and operational issues that may arise during the various phases and summarize what has worked. The triangulation of sources and use of axiomatic arguments provides some confidence in external validity. The Diagnostic Phase Finding 1: It is important to clearly determine the rational for implementing an FMIS, and identify the problems that the proposed system intends to address. 3.2 A careful diagnostic study to identify problems in budget management that FMIS investments can address is crucial before starting any implementation work. The relevance of a comprehensive, consistent, and objective diagnostic has been raised in a number of studies (see for example Dorotinsky and Matsuda 2001, Dorotinsky 2003, Diamond 2013, and Hashim 2014), and lessons to that regard have been pointed out 10 The learning through noticing approach alters the standard intuition that experience guarantees effective technology use (see, for example, Nelson and Phelps 1966, Schultz 1975, Foster and Rosenzweig 2010f). 20 across a number of World Bank ICRs.11 The argument is that without clearly articulating how the FMIS would address these problems, investments could result in a system that does not match actual requirements, which would fundamentally undermine its relevance and effectiveness. It is also important for minimizing financial and institutional resources, as clearly articulated in the Bolivia ICR. 3.3 A detailed diagnostic assessment led to an effective engagement in countries where it preceded investments and then guided the solutions’ design. In Honduras, Kazakhstan, the Russian Federation, and Ukraine, for example, a detailed diagnostic of the underlying economic problems and their causes preceded work on setting up the treasury systems. The diagnostic emphasized a fiscal situation under stress, a deficit exceeding its targets and needing cash rationing, and a rapid arrears accumulation. Among the causes identified were weaknesses in the legal framework, institutional structures, and accompanying systems required for managing government finances. Following this, the reform directions had three distinct characteristics: • Implementing a centralized TSA, including closing all spending unit bank accounts. • Establishing a treasury-centric payment mechanism that routed all government payment and receipts through the treasury (or out-posted treasury staff). • Setting up a basic budget execution system with line item control. These three elements of the reform program led to the following: • Better fiscal control by ensuring that expenditures complied with budget appropriations, commitments and cash allocations, close monitoring of outstanding bills, cash in government bank accounts, arrears and fiscal deficits. • Better cash management by bringing all government accounts under the treasury’s control. • Timely and accurate reporting for economic management, and preparing statutory financial statements, and improved baseline data quality for budget preparation. 3.4 Lack of a thorough diagnostic led to ineffective engagement and disappointing results in several cases.12 Ghana and Zambia, for example, faced similar problems to 11 See project documentation for FMIS operations in Bolivia, Cameroon, Colombia, the Democratic Republic of Congo, Ecuador, Honduras, Kazakhstan, Kenya, Madagascar, Russia, Sierra Leone, the Slovak Republic, and Ukraine. 12 See project documentation for FMIS operations in Zambia, Ghana, Cambodia, Russia, Kazakhstan, and Ukraine. 21 21 countries above with fiscal discipline and cash management, but the focus on the engagement was fundamentally different: • They did not focus on establishing a TSA or centralizing the payment function even though World Bank accounts outside treasury control were recognized as a critical problem. • Even without a basic budget execution system with basic line item control, these countries embarked on advanced budget preparation methodologies (such as activity-based budgeting, program budgeting, and performance budgeting). • They adopted a black box approach that did not set priorities implementing core budget execution modules. • They took an FMIS technology-driven approach that assumed that a comprehensive off-the-shelf enterprise resource planning package would meet government and user requirements. 3.5 This approach was not based on a rigorous diagnostic and failed to recognize the importance of a TSA and a centralized payment system, even though these issues had in some cases already been apparent.13 As a consequence, the implementation of the FMIS did not did lead to the desired budget management and control improvements. Since then, the approach has been partially correct in some of these countries. 3.6 The diagnostic could underpin the formulation of the wider public financial management reform and FMIS implementation strategy. The lessons from the ICR for Jamaica suggested that the diagnostic could be used to determine the system’s feasibility and overall scope. It could provide a blueprint for implementing the system, including sequencing, approximate costs and timeline, project management and technical capacity, and other requirements. Such a preparatory step would determine the FMIS implementation strategy, and guide the process. The diagnostic could also underpin the formulation of a wider public financial management reform strategy to ensure adherence to new policies, procedures, and control structures and resolve problems associated with timely availability of information for economic management.14 13 In Zambia, the two adaptable program loans under which the FMIS work was done did not discuss the importance of a TSA or have a component related to its establishment, despite the IMF noting that idle balances were excessively high. In Ghana, no work in this area was done during 1998–2007. The follow-up initiative has since made modest progress, but large resources remain outside the treasury’s purview. 14Some authors, such as Peterson (2006) and Chêne (2009) however warn against using FMIS as an entry point for broader PFM reform to avoid unnecessary excessive complexity. 22 POLICY AND INSTITUTIONAL UNDERPINNINGS Finding 2: The effectiveness of an FMIS as a budget management tool depends - not only on its technical robustness - but also on the policy and institutional environment under which it operates. These factors should be incorporated in a diagnostic and can be considered as preconditions as even advanced systems may not facilitate desired budget management improvements without these. 3.7 Investments in FMISs yield the highest returns after laying appropriate policy and institutional groundwork.15 In the literature these issues are often referred to as preconditions or analog complements that need to be in place (Diamond and Khemani 2005, Dener 2011, Peterson 2011, Dorotinsky 2013, and Hashim 2014, World Bank 2016). The World Development Report 2016 notes that ‘when analog complements are absent, the development impact will be disappointing’ (World Bank 2016, p.5). As such, it is critical that these are considered in a diagnostic study. 3.8 Even a basic FMIS can facilitate significant progress in budget management when an enabling environment is provided.16 Countries such as Kazakhstan, Nepal, and Russia have reformed their policy and institutional framework including the enactment of enabling legislation, establishing the treasury organization, and introducing a treasury single account (TSA). These have led to significant budget management improvements, despite only rudimentary FMIS systems being in place. On the other hand, significant investments in advanced systems were unable to address basic budget management problems if the underlying policy and institutional structures were not in place.17 In Zambia, the lack of attention to this aspect had an adverse impact, and significant idle balances built up in line ministry bank accounts in commercial banks. Similarly, in Ghana this resulted in holding about 30–40 percent of the total financial resources separately in commercial bank accounts outside the treasury’s control. 3.9 A comprehensive TSA is a critical enabling condition for effective budget management.18 From a cash management perspective, having all government moneys in a TSA at the central bank is important to avoid large idle balances in commercial bank accounts outside the treasury and ministry of finance’s control. If money is outside the TSA and the central bank, the government cannot draw on these funds for investment (or for fund requests from other spending units). Furthermore, commercial 15See project documentation for FMIS operations in Bolivia, Chile, Colombia, Ghana, Kyrgyz Republic, Nepal, Philippines, Russia, Ukraine, Uganda, and Zambia. 16 See project documentation for FMIS operations in Nepal, the Russian Federation, and Ukraine. 17 See project documentation for FMIS operations in Ghana and Zambia. 18 See project documentation for FMIS operations in Bolivia, Ghana, and Zambia. xxiii banks that hold this money can use it to buy government borrowing instruments (such as treasury bills), which means they can lend the government its own money at interest. Although an FMIS could facilitate transactions for funds outside a TSA, the FMIS alone cannot address the associated fundamental budget management problems stemming from an ineffective or partial TSA. As the Bolivia FMIS ICR pointed out, “institutionalizing procedures for cash management and control requires… a tremendous change in the conceptual framework supporting financial management in the public sector.” 3.10 For the TSA to be effective, coverage would have to include all government funds. Therefore, it should include all budget resources, extra budgetary funds, internally generated funds, and donor funds. Although donor funds are often kept outside the TSA (such as in Ghana and Zambia), they can be placed within and still be ring-fenced, earmarked, and subjected to external control. If funds are kept outside the TSA, banks that hold them can use them to buy government borrowing instruments, and all problems as outlined above apply.19 3.11 Linking accounts improves transparency, but it does not address the basic associated budget management issues. Several countries, such as Ghana and Zambia, applied an intermediate arrangement of linking accounts. Although this arrangement improves transparency, it does not address the basic budget management problems associated with idle balances because the funds are still outside the treasury’s purview, the funds cannot be used for investments (or fund requests from other spending units), and funds will likely be used to buy government borrowing instruments. Therefore, linking accounts is not a different TSA modality, nor should it be considered a viable alternative. An IMF guidance note on TSAs makes this clear, stating that for a TSA to work effectively, accounts should operate on a zero-balance basis, and balances should sweep into the central bank unconditionally (Pattanayak and Fainboim 2011). 3.12 All levels of government should have a uniform budget classification structure to enable comprehensive countrywide reporting by the FMIS.20 The information requirements determine the design of the budget classification structure. In principle, 19 If the government finds it necessary to give line ministries access to financial resources without routing the transaction through the central treasury (project accounts, for example), it can establish zero-balance accounts with commercial banks that line ministries could access without reference to the central treasury. These accounts would clear to the TSA periodically, and idle balances would not build up. In such cases, the necessary financial control structure will need to be established in the line ministry itself by, for example, posting treasury staff to the line ministry, or delegating the financial powers and responsibility (along with accountability) to the line ministry financial staff. The government can make similar arrangements for donor-funded project accounts. 20 See project documents for FMIS operations in Chile, Kazakhstan, Moldova, Nigeria, Russia, Uganda, Ukraine, and Vietnam. Some countries (especially in Latin America) have a separate budget classification and chart of accounts, with a bridging table to compare the two classifications. 24 this structure should at least cover function, organization, and spending unit, and object of expenditure (economic classifications). Function codes and economic classification codes should follow internationally accepted classification schemes (Diamond 2013). Furthermore, codes such as source of funds and program and project codes could be added to track expenditures by fund source and for specific programs and projects. It would then be possible to break down the organization codes to a level where a specific unit is performing a specific subfunction. 3.13 It is fundamental that the economic classification codes are a subset of the account codes in the chart of accounts to ensure integration of budgeting and accounting. If expenditures are booked under line items in the chart of accounts (which is different from those in the budget classification system), then reports produced from the FMIS would not provide expenditure incurred by budget appropriation line items. 3.14 An overly complex budget classification structure and chart of accounts could impede implementation. A complex budget classification has frequently been justified to allow allocation of every transaction to the lowest level. However, the complexity needs to be balanced with coding requirements (the complete budget classification structure and chart of accounts need to be coded on every transaction). This is cumbersome and can lead to allocation errors. Therefore, there may be a trade-off between the level of detail required for the transaction and the quality of the data entry. Furthermore, a chart of accounts that is too complex could also increase transaction processing times on the server, especially when the number of active users is large, as was the case in Vietnam. CONTROL PROTOCOLS Finding 3: A review of existing control protocols is essential. Setting up an automated FMIS without the necessary control functionalities could speed the hemorrhage of resources instead of controlling it. 3.15 A review of control protocols is important for a comprehensive diagnostic. An automated system may increase the speed of disbursements. For example, the Philippines implemented reasonably effective automated systems to distribute budget allocations to spending ministries, but did not implement a system to monitor budget execution. In this situation, the government has an efficient mechanism in place to distribute its financial resources to the line agencies, but it has little control over whether the agencies spend resources as prescribed in the budget. 21 21 See project documentation for FMIS operations in Ghana, Philippines, and Zambia. 25 25 3.16 A payroll system automates and expedites the process, but that system does not provide any budgetary controls. Ghana and Zambia had a fully functioning payroll system in place, but the budgetary limits were exceeded because transactions were not subjected to FMIS budget checks, which caused significant budget overruns after substantial wage increases. This was also true for the domestic debt payment system in Ghana. Thus, this increases the efficiency in the process without controls. The System Development Life Cycle PROJECT MANAGEMENT Finding 4: Project management structures are important to achieve government commitment. 3.17 Government commitment to the reform agenda is critical. This has been highlighted almost universally by World Bank project documents22 and is widely cited by the literature (see for example Diamond and Khemani 2005, Dener 2011, Hashim 2014, and Combaz 2015) and is a generic lesson that applies to much of the World Bank engagement overall. A more nuanced point is that project management structures can help generate and maintain critical commitment.23 The following factors could help facilitate this: • Appointing a senior-level project sponsor from the ministry of finance (such as the minister or the permanent secretary) is required for introducing necessary policy changes and other decisions for project implementation. In Cambodia, for example, the minister of finance took on this role and showed continuing commitment to the FMIS implementation, which helped sustain government interest in the project despite a lack of progress for many years. It was also instrumental in overcoming resistance from more traditional quarters in the Cambodia Ministry of Economy and Finance and elsewhere. In Zambia, FMIS implementation progress suffered in the project’s early years (2005–08) because government support had waned. Project implementation was revived only after the permanent secretary of the Zambia Ministry of Finance and National Planning could get top management on board to address FMIS-related issues. Also in Zambia, the payroll management and establishment control system’s improved performance is partly attributable to government support for 22See project documentation for FMIS operations in Albania, Argentina, Bolivia, Brazil, Cabo Verde, Cambodia, Cameroon, Chile, China, the Democratic Republico of Congo, Ecuador, Georgia, Guatemala, Honduras, Indonesia, Jamaica, Kazakhstan, Kenya, Lao PDR, Madagascar, Mongolia, Nicaragua, Pakistan, Russia, Sierra Leone, Somalia, Turkey, the República Bolivariana de Venezuela, Vietnam, and Zambia. 23Conversely weak project management arrangements are often associated with project failure (see for example Beschel and Ahern 2012 on FMIS implementation in Iraq). 26 underlying reform measures related to the civil service and the project the system supported. • Appointing a steering committee that includes representatives from all major stakeholders helps provide policy guidance and facilitates consensus building. Major stakeholders include the ministry of finance, treasury, budget, central bank, line ministries, and revenue collection agencies. Many successful FMIS projects, such as those in Cambodia, Indonesia, Kazakhstan, and Russia, established steering committees and used them effectively. Cambodia, for example, took an approach that needed substantial input from end user departments, which had to sign off on functional requirements and systems specifications before starting the procurement process. This ensured that user department needs were adequately addressed, and it facilitated their support. • The project manager should be a senior official from the functional side with stature within the bureaucracy and adequate financial and administrative powers to cater to day-to-day operational, administrative, and financial requirements. FMIS projects are not information technology projects, and the project’s primary emphasis needs to be on functional objectives like fiscal control and cash management, which requires project management that is competent in these areas instead of technology. In Indonesia, Kazakhstan, Pakistan, Russia, and Vietnam, appointing such a person facilitated project implementation. • Selecting a core team or working group consisting of staff with in-depth knowledge of their functional areas ensures that the system design is responsive to functional requirements. A core team or working group can document the business processes and help design new functional processes. After the system is completed, they can be change agents during implementation. This worked in Pakistan, where the working group facilitated buy-in from end users. 3.18 It is important to note here that putting a well-designed committee structure in place is not, by itself, sufficient to ensure smooth and effective implementation of FMIS projects. Various change management aspects are important to neutralize or mitigate the opposition of stakeholders who are against the reform. FUNCTIONAL PROCESSES AND SYSTEMS DESIGN Finding 5: Functional requirements should drive system design. 3.19 Determining the overall information architecture is crucial for FMIS design.24 This requires assessing the associated government functional processes, the 24 This point has been developed by a number of studies including Diamond and Khemani (2005), Rodin-Brown (2008), Khan and Pessoa (2010), Hashim (2014), and Combaz (2015). Further, FMIS implementation experiences 27 27 underpinning regulatory framework, their information requirements, and functional responsibilities of implementing agencies. The main functional processes involved in government financial management include macroeconomic forecasting; budget preparation; budget execution, accounting, and fiscal reporting; cash management; position, payroll, and benefits management; pensions management for government retirees; debt management; revenue administration (customs and tax); and auditing (appendix B25 provides more details). Each of these processes requires specialized information systems to support them. 3.20 Designing a system that can support all these processes and share information should start with an analysis of the functional processes. The Guatemala ICR noted that “information systems need to be designed along functional rather than organizational lines, sharing a common data base.” This should include those associated with government financial management; the overall regulatory framework underpinning these processes; their information requirements; the functional responsibilities of agencies commonly responsible for the processes; information flows between the agencies; the nature, volume, and frequency of these flows; and the data characteristics of the information the processes use and create. This analysis produces the information architecture for the government financial management systems landscape. The information architecture addresses questions such as the following: • What are the different information systems modules needed to support government financial management functional processes? • What is the function, scope, scale, and type of a particular systems component? • What are the primary interfaces of the modules with each other—that is, how do these modules exchange and share information, and what are the characteristics of the primary information flows? 3.21 More details on FMIS design are provided in Appendix B.26 3.22 The information architecture can be a useful template to inform design. It can help to identify a project’s areas of priority, focus, scope, and coverage, and the interfaces between modules. The information architecture proved useful in identifying the focus areas and for the design and a roadmap for the FMIS implementation in several countries, such as Ghana, Liberia, and Pakistan. Lack of such analysis at the from the following countries have documented similar lessons: Argentina, Bangladesh, Cambodia, Colombia, Ghana, Guatemala, Hungary, Kazakhstan, the Kyrgyz Republic, Liberia, Madagascar, Pakistan, Uganda, and the Republic of Yemen. 25 Appendix B can be accessed at this link. 26 Appendix B can be accessed at this link. 28 design stage can result in adopting a black box approach. If the system design is based on technological considerations or does not clearly specify which functionalities and modules are the prime focus areas, the engagement can be counterproductive, as in the early stages of the Cambodian engagement or in Zambia. Finding 6: FMIS projects are likely to be more effective if framed as expenditure management projects, rather than accounting projects. 3.23 It is important not to lose sight of budget management issues during system design, even if the diagnostic points to immediate other deficiencies. Comparing projects designed as accounting projects to expenditure management projects illustrates this point: 3.24 In Pakistan, the Pakistan Improvement to Financial Reporting and Auditing system was originally an accounting project with an objective formulated around comprehensive reporting and accounting and data provision to enable auditing.27 Although the project objective was reportedly achieved (see ICR and the IEG ICR Review), the system did not focus on fiscal and cash management issues (critical concerns to the country at the time), which undermined its relevance to overall budget management. Comprehensive reporting was achieved by ensuring that periodic transaction summaries were included ex post in the system. The system, therefore, produced complete and comprehensive accounts—a critical requirement for an accounting project and in line with project objectives. However, the system would need to include all budget expenditures to determine the ways and means position accurately, and currently transactions from agencies with departmentalized accounts at both the federal and provincial levels do not go through the system. Consolidated statements from these agencies post to the general ledger only after the transactions occurred.28 As such, addressing the major public expenditure management issues confronting the country would require a paradigm shift and a retrofit of the deployment and transaction flows. 3.25 In Indonesia in the 1990s, the World Bank tried to establish systems aimed at improving the government’s accounting and reporting capabilities. These projects did not aim to reform the underlying policy and legal framework and the institutional arrangements for budget execution that mainly concerned the ministry of finance. The successor project prepared in 2003 recognized that the problem was primarily with 27 This project followed a diagnostic that was restricted to accounting issues in the audit department. However, the diagnostic did not cover the country’s larger fiscal or economic management issues and consequently the project’s relevance for budget management was inhibited. 28 This includes the self-accounting agencies with departmentalized accounts estimated to amount to more than 30 percent of the budget. 29 29 expenditure management and made fundamental changes in its engagement accordingly. 3.26 The ICR of the Pakistan PIFRA II project (p. 19) noted that “Projects such as PIFRA should be framed primarily as initiatives designed for reform of the expenditure management systems rather than merely accounting systems reform. If reform measures are framed in terms of resolution of an accounting problem… the danger is that it will be focused only on the reform of the organization and processes related to the accounting function. These are areas where it is difficult to get active involvement of senior policy makers in the MOF. Framing the project primarily as an expenditure management reform initiative raised the importance of the problem to a level where senior level policy makers in the country and their counterparts in donor organizations could relate to it.”29 FMIS investments framed as public expenditure management reform projects are likely to be more prominent on the political and economic management agenda. The Chile ICR noted that “FMIS are designed to extend beyond one political administration, such political transitions should be managed and accounted for in project design and supervision.” Adequately framing the project could thus be an important factor for the investment’s sustainability. FMIS PROCUREMENT 3.27 Appropriately designed systems procurement strategies and contracts packaging is crucial to ensuring timely availability of quality consulting services and technology at competitive prices. FMIS implementation requires procuring complex technology platforms, as the systems include computer hardware, telecommunications, systems and application software, information security systems, and other components that work together to deliver a functional system. Specialized consulting services are also required for the system design and implementation. Two main contracting assignments are normally involved in project implementation, as follows:30 • A diagnostic and design consultancy. A consulting firm with the required experience and expertise normally performs the work associated with diagnosing the existing situation and developing the new system’s design and specifications. This assignment also includes developing the tender documents for the system components and helping the government during the tender evaluation process. 29 This lesson was successfully picked up in the design of the Indonesia GFMRAP project. 30 The government could also make several supplementary and specialized consulting assignments for areas such as training, change management, and restructuring the technical organization required to support system implementation. 30 • A contract for system implementation. The main contractors selected through an international tendering process perform the systems implementation work. 3.28 Appendix G31 discusses options for procuring the technology platforms (single responsibility contract and multi-tranche procurement) and highlights important issues to consider when configuring the two major contracting assignments and other procurement related issues. Appendix H32 provides further detail on procurement considerations. Finding 7: Lack of precise systems specifications in the tender documents could cause suppliers to build large risk mitigation factors into their pricing, which can result in excessively high bids. 3.29 It is important for tender documents to specify exactly what is required and expected from the system.33 Without these details, it is difficult for a potential contractor to understand what is expected. If the FMIS suppliers do not recognize the risks associated with an extensive list of requirements and do not have a focused description of the real requirements’ priorities and details in a procurement document, they might build large risk mitigation factors into their pricing, resulting in excessively high bids. This occurred in Cambodia—the first FMIS tender yielded bids ranging from $28–35 million, but the available budget was only about $10–12 million. This delayed the procurement for several years, and was eventually canceled. 3.30 Taking specific measures can help preempt these risks. After the first tender failed in Cambodia, the World Bank took a proactive approach in the second tender and advised the government to take the following steps that facilitated appropriately priced bids: • The bid documents clearly specified core which functional processes and information flows are covered, which departments and entities will be connected online, and the number of users in each department and entity; which transactions need to be recorded; and which departments will send transactions offline and how the system will capture them. The bid clearly documented the estimated transaction volume and high-volume sites. • The functional specification document specified the buyer’s overall business objectives. The specification adopted a process-based approach instead of 31 Appendix G can be accessed at this link. 32 Appendix H can be accessed at this link. 33See project documentation for FMIS operations in Cambodia and the Kyrgyz Republic. The Kyrgyz Republic ICR noted that “the expectations of vendors should be made explicit through the tender process and during contract negotiation, in particular as regards the degree of customization and the presence of technical staff in the field .” 31 31 feature-based, focused on the processes the system would support, and derived the functionality required from them. • Systems procurement used a two-stage international competitive bidding (ICB) process. The government used stringent screening criteria in the first stage to avoid spurious bids—for example, firms lacking relevant public sector experience, and software packages that were unsuccessful for similar systems (in both functionality and scale). The government designed the screening criteria to screen firms based on financial viability and previous, relevant public sector experience. It also screened proposals based on successful use of the proposed application software for implementing FMISs and treasury systems, and whether the firm proposed staff with experience in implementing the software. • The government and consultants developed broad cost estimates for the systems procurement package to use as a guideline for assessing bids. The bid specified the total amount of resources available under the loan to restrict spurious and excessively high bids. Finding 8: It is important to weigh costs against benefits when choosing an application software strategy. 3.31 Choosing an appropriate application software strategy for the FMIS is crucial when configuring the technology platform.34 Strategy options include using in-house or custom-developed application software or using commercial off-the-shelf (COTS) software packages. Those deciding which strategy to use should consider the various modules’ complex functional requirements and the critical requirements for integration and data sharing so that the systems can deliver the required outcomes.35 Here, local capacity is a critical determining factor. 3.32 Different strategies may be appropriate for different phases of systems implementation. The initial phases of the reform might require only a subset of features to support core FMIS elements, and a basic locally developed system may suffice (see for example Georgia). A full-function system needing an advanced, off-the shelf solution may be required after the reform stabilizes and the legal and institutional framework is in place and functioning. For example, projects in Kazakhstan and Russia initially used custom software to implement core treasury functionality, but later 34This has been discussed extensively by Dener (2011), Beschel and Ahern (2012), Hashim 2014, Combaz (2015) and Uña and Pimenta (2015), and is well reflected in project documentation for FMIS operations in Bangladesh, Brazil, China, Georgia, India, Indonesia, Kazakhstan, the Philippines, Russia, and Ukraine. 35 Appendix F contains a list of items required for the technology platform. It can be accessed at this link. 32 replaced it with COTS software. Similarly, the new project in Ukraine planned to use COTS software (Hashim 2014). 3.33 Building in all COTS features36 into a custom-developed package requires extensive software development that can be costly and time consuming. Furthermore, any extensive in-house application software development effort requires significant technical and project management expertise—without it, custom development can be risky and the outcome unpredictable. Advanced countries such as Brazil, China, and India, are more likely to have readily available software development expertise, in which case locally developed custom software could be a medium-term option. However, the need for such expertise would essentially extend throughout the life of the software to ensure adequate maintenance and upgrades as required. In practice, it has often been difficult to retain highly qualified technical staff in government for such long periods. Software development is a specialized business and not within a government’s core competencies. Outsourcing options can be expensive and less reliable. 3.34 Some countries (Bangladesh and the Philippines, for example) have made significant investments in custom-developed software, but progress in implementation has been slow. In Brazil, China, India, and the Philippines, attempts to develop and implement a comprehensive integrated FMIS based on custom development remains work in progress (as of 2016), despite the availability of technical expertise for systems development. The slow progress is attributable to a whole host of factors, and not only the choice of application software strategy. 3.35 On balance, it can be more practical and cost effective and less risky to use a COTS solution when software development expertise is not available. It is relatively easy to develop the expertise required for operating such a solution through specific in- country training programs. Software maintenance is the supplier’s responsibility and is available with an annual license agreement. 3.36 A more detailed discussion on what it takes to move from an FMIS to a broader integrated system is discussed in appendix E.37 However, there are considerable risks associated with changing technology platform mid-way during the reform process. These are discussed in detail in appendix I.38 36 Such as functionality, controls, and integration. 37 Appendix E can be accessed at this link. 38 Appendix I can be accessed at this link. 33 33 SYSTEM IMPLEMENTATION Finding 9: Taking a phased approach in FMIS implementation is an important factor of success. Fundamental modules necessary for budget execution and reporting should be prioritized. 3.37 Budget execution is central to the information architecture. Although each system in the information architecture performs important functions in a government’s financial management process, the collection of modules that provides support to budget execution, accounting, cash management, and fiscal reporting (known as the budget execution or the treasury system) are regarded as the core elements of this architecture, considering their central place in the government’s financial management cycle (see World Bank 1998, Dorotinsky 2003, Hove 2010, or Hashim 2014 on definition). These systems are the centerpiece of the government financial management systems network and are used to conduct the budget execution process and monitor and evaluate overall budget implementation. They are the primary repository for financial data that are the basis of government financial management. The systems include information on resource levels expected to be available during the year and how to allocate these resources as the macroeconomic forecasting and budget preparation systems specify, and the balances available for use at a given time during the fiscal year. Other systems, such as fixed asset management, debt management, tax and customs administration, payroll, and pensions systems, though of vital importance to government financial management, support the core system by maintaining detailed data related to specific areas and providing these data to the core according to specified requirements for timeliness and detail. 3.38 FMIS projects are more effective if implemented in phases.39 The Mongolia ICR notes that “expectations must be adapted to appropriate pace, sequencing and sophistication of the reforms. Basics should be considered a priority before more sophisticated reforms are attempted. If they take longer than envisaged, then the introduction of the sophisticated reforms should be delayed.” As such it is essential to implement modules catering to core budget execution processes first to achieve significant progress on budgetary control and cash management early in a project. For practical system implementation, it is crucial to sequence and develop various system elements in a modular way. Systems sequencing is vital for a successful implementation program. Simultaneously implementing all systems modules is in most cases neither practical nor required and instead likely to overstretch client capacity. After the initial 39 Combaz (2015) notes that “there is a consensus in the literature and among contributing experts that sequencing implementation is crucial to avoid the high risks associated with implementing too many components at once. ” This finding is also corroborated by World Bank project documentation for FMIS operations in Afghanistan, Azerbaijan, Bangladesh, Chile, Colombia, Ghana, Honduras, Kazakhstan, Mongolia, Nicaragua, Pakistan, Sierra Leone, Russia Somalia, Tajikistan, Ukraine, and Zambia. 34 framework is established and the integration prerequisites and criteria are detailed and incorporated into the implementation plan, actual implementation and module integration could be sequenced and phased according to requirements. Further, the literature suggests that the system should be pre-tested with real data and assessed and only after a successful pre-testing phase should the system be rolled out (Uña and Pimenta 2015, Chêne 2009). 3.39 The importance of sequencing implementation is illustrated in Russia, Kazakhstan, Pakistan, and Vietnam where a focused effort on core budget execution processes has facilitated good progress in a short time period. On the other hand, in Cambodia and Zambia initial attempts at implementing all modules without any phasing failed and subsequently a more sequenced approach had to be adopted to finally deliver a working system. The ICR of the Ukraine PFMP clearly describes the problems associated with abandoning an approach which focused on core budget execution processes in favor of an all-encompassing approach. On the other hand, the Chile ICR makes interesting references to benefits of a big-bang approach.40 Finding 10: A small percentage of transactions tend to make up a large share of the budget. Capturing such high-value transactions and subjecting them to budgetary controls may facilitate improved fiscal management early in implementation. 3.40 The transaction profile tends to be skewed with few transactions making up the majority of the budget.41 Typically, a small share of expenditure transactions (by number) account for a large share of the budget (by volume). A rough estimation of this proportion is 20: 80, even though in some large countries (such as China and Vietnam) this distribution was found to be even more skewed (with 8-12 percent of number of transactions covering 90-95 percent of the budget). Thus, if a transaction threshold could be determined defining high-value transactions which cover 90-95 percent of the budget, it would be possible to target ex ante control to these transactions. Such an approach could yield a very high budget coverage despite focusing only on a limited number of transactions. Figure 3 shows this pattern for a sample country. 40Chile ICR notes: Both big-bang and gradual implementation processes for FMIS systems present challenges, and there does not seem to be an easy answer as to which is the better approach. In the case of Chile, the decision to proceed gradually allowed it to initiate implementation (and deal with the issues that arose) without risking impacting the entirety of the government, and shift towards a more aggressive strategy only when such problems had been addressed. Nevertheless, this strategy has its costs requiring that resources continue to be allocated to both the old system and the new one and may diminish individual institutions ’ incentives to transition to the new system. 41 See project documentation for FMIS operations in Vietnam. 35 35 Figure 3 Sample transaction profile by number of transactions and volume of budget 100 99.5 99.9 100.0 93.2 96.1 Cummulative share (percentage) 90 80 82.0 70 65.3 60 50 52.7 47.4 40 36.3 30 31.6 20 18.4 15.7 10 11.4 6.7 3.5 0 0.0 0.0 0.1 0.4 1.0 Value of transactions in LCU Number of transactions Value of budget Source: Authors. 3.41 This information could be used to formulate systems deployment and control strategies that focus initially on these high-value transactions. Such a strategy would yield useful results early in an FMIS rollout phase, and address some of the cost-benefit issues raised by Glenday (2015). This could be done by stressing that transactions above this amount be routed through the treasury system directly. It is important to recognize that maintaining accountability for low-value transactions and capturing them in the treasury system continues to be important. For fiscal discipline, capturing high-value transactions is key. But in order to produce comprehensive financial statements, full transaction coverage, including low-value transactions, is necessary. However, the modality through which these are captured may differ. 3.42 In practice, the spending units that generate these large transactions are few, and capturing transactions from these units would result in the same high coverage. Thus, the treasury could exert control by focusing on spending units that generate high- value transactions. This approach is akin to focusing on large taxpayers on the revenue side, which is also a risk-based approach. Identifying these spending units can enable the treasury to focus control on them and transactions above this threshold instead of wasting effort across the network. This could also be useful for determining future cash requirements and developing borrowing strategies because they determine the bulk of the cash requirements. 3.43 If a spending unit that is not part of this list generates a high-value transaction, the rules could then specify that the transaction must be routed through the central 36 system. The remaining transactions could be paid through the banking network in which branches of commercial banks act as fiscal agents for the central bank and operate zero-balance accounts. These banks would need to be instructed not to honor payments above the calculated transaction threshold. After agreeing to such a selective control principle, then the following three steps could be taken in practice: 1. Make it mandatory to route all transactions generated at the central ministry of finance (such as fiscal transfers, subsidies, and debt service payments) through the central system. 2. Route all payroll and civil service pension payments calculated by a central system through the central system (these are typically about 30-40 percent or more of the total budget). If the payroll is not automated, route the corresponding bill or payment request through the central system after the line ministry or spending unit calculates the employee payroll. 3. Route all payments above the transaction threshold from a line ministry or spending unit through the central system. 3.44 Such a strategy could be a useful first step in implementing a budget execution system in countries that do not yet have one. For example, in the Philippines, the large number of spending units and the lack of a network of treasury offices charged with ex ante transaction control (which line ministries resist for all transactions) hindered all previous efforts at implementing such a system. In this environment, line ministries and spending units can open bank accounts in designated commercial banks. The government transfers their allocations to these accounts periodically, and the ministries and spending units make payments through these banks. The treasury single account at the central bank controls the budgetary resources, and the commercial banks operate zero-balance accounts. However, ex ante budget control is not practiced on these transactions, which leads to the problems described earlier. Therefore, if transactions above a transaction threshold are required to be paid through a central treasury payment system (which currently regulates only budget transfers to spending unit bank accounts), then an effective budget execution system could be established quickly without deploying the treasury system to all spending unit sites (which can number in the thousands). The banking network would still handle transactions below the threshold (and during the transition period), as is currently proposed for the Philippines. This strategy could also be useful for countries where state of the art systems have been put in place, but the budget coverage remains low. 37 37 Finding 11: Deployment of FMIS systems is costly and logistically complex. Taking a sequenced approach and utilization of web-based technologies could help. 3.45 Deploying a budget execution system to all spending units involves a high deployment cost and complicated logistics due to the large number of spending units in the country. The following two strategies were able to address this issue in some countries42: 1. Implement a treasury-centric system first, and then decentralize to spending units at a later stage, as necessary. Choose a systems deployment architecture that enables comprehensive capture of all payment and receipt transactions and application of ex ante control to these transactions while limiting the deployment to nodes that are essential for this task. A deployment across the treasury offices (or out-posted treasury staff in line ministries) through which these transactions are routed could be sufficient for this purpose. Projects in Indonesia, Kazakhstan, and Russia adopted this approach, extending the central system to spending units after it was stabilized. 2. A web portal can be cost effective and provide easy access. A spending unit– based system deployment is preferable in principle, but it can be costly, impractical, and sometimes violate a treasury law requiring the treasury to be responsible for payment control. However, a web portal can give spending units access to the system while containing costs and retaining the treasury’s role of exercising control over payments. In this case, spending units can access a web- based portal to send a transaction to the system electronically, but this transaction would not update the system databases and would create a file that the treasury office would later use to review the transaction and update and process it for payment. Web portals can also give budget administrators direct access to the system to perform their budget management responsibilities, and for line ministries to access budget execution figures and reports. Several countries used this approach successfully. Spending units and other end users in Indonesia and Russia use web portals to send transactions to the system electronically instead of physically bringing them to the treasury offices. The Vietnam treasury started a pilot to provide such access. 3.46 Using a treasury-centric system also has challenges. The lack of spending unit access inhibits ownership at the line ministry and end user levels, which often leads them to develop independent financial systems (as was the case for example in 42See Diamond (2013) and project documentation for FMIS operations in Indonesia, Kazakhstan, Russia, Somalia, and Vietnam. 38 Vietnam). Secondly, in some cases line ministries have resisted such an approach, fearing that the central treasury would delay payment requests. Strategies to get around these issues are developing standards (e.g., monitoring processing times) or setting a web portal to provide basic access to the central treasury system, as was done in Indonesia, Russia, and Vietnam. Finding 12: Reliable transactions processing is fundamental to the integrity of a system. It provides the data used for financial operations and management reporting. It should thus be prioritized in the implementation process. 3.47 FMIS implementation can be viewed as a layered approach to process basic transaction data. A financial operations layer is next for producing budget execution reports, determining the government bank accounts’ cash position and the ways and means position, and producing monthly financial statements. The top layer is the management and statutory reporting layer, which produces overall financial reports, audited financial statements, and statutory financial reports (figure 4). Figure 4. Budget execution and treasury systems: a layered implementation approach Economic Management / Overall Fiscal Reports & Audited Management/ Statutory Reporting Statutory Reporting/ Statutory Financial Statements Layer Auditing Budgetary Control Budget Execution Reports Accounting, Cash Cash Position, Ways and Means Position Financial Operations Management Monthly Financial Statements Layer Budget Commitment Payments Receipts General Cash Bank Transaction Systems Modules Management Management Management Management Ledger Management Reconciliation Processing Layer Vendor Invoices, Goods Receipts, Budget Apportionments Commitments, Payment requests Transaction Tax and non tax Receipts, Allotments, Releases Purchase Orders Debt servicing payments Data Bank balances Transfers, Supplementary Contracts Payroll, Pension payments, Subsidies, Budgets Fiscal transfers Systems Users / Data Providers Ministry of Line Ministries Debt Finance Revenue External and Treasury & Management TSA Bank Budget Agencies Internal Auditors Spending Units Department Department Source: Reprinted with permission from Hashim 2014. Note: TSA = treasury single account. 39 39 3.48 It is important to implement the transaction processing layer first to get credible and complete information for financial operations and management reporting.43 The handbook for public expenditure management stipulates that “information is the life blood of budgetary, resource allocation and financial management” (World Bank 1998, p.59). At a most fundamental level, this requires the data to have integrity and completeness, which can only be achieved if the transaction processing layer is intact and reliable. The quality of information available from the financial operations and management reporting layers will depend on the quality, timeliness, and comprehensiveness of the transaction data captured by the transaction processing layer. Therefore, even though implementing the transaction processing layer is the most difficult and time consuming, it should be prioritized (Hashim 2014). Once this is done, the other layers can be implemented with relative ease. For example, it took five to six years to implement a comprehensive transaction processing layer in Russia, but afterward, the treasury implemented strong analytical and reporting tools within six months to provide the government with useful information for monitoring financial operations, management, and statutory reporting. 3.49 In Malawi, the transaction processing layer was weak because transactions were done outside the system, and spending units kept a second set of books. This was found to undermine the data integrity in the transaction processing layer, which makes any functional and economic reports coming out of the system questionable. In other countries (such as Ghana and Zambia), the transaction processing layer did not include large portions of government financial resources and is thus not comprehensive. Thus, financial operations and management reporting will be based on incomplete data. 3.50 An alternative modular approach discussed by Gerardo Uña, in a recent blog post, draws on recent ICT developments in the private sector and holds promise for making systems implementation more effective and easier to maintain. The idea is to establish a common financial information architecture, which is interfaced with all core and non-core modules. These may be developed separately, but can communicate with one another. The state of São Paolo in Brazil is spearheading this approach44 (IMF 2017). Regardless of the approach, it remains paramount to ensure the integrity of the financial transactions as all other layers or modules will draw on this. If data from the financial transactions are compromised, so will be all other layers or modules. 43 See project documentation for FMIS operations in Ghana, Malawi, and Zambia. 44More information on this approach can be found at the following IMF PFM blogpost: http://blog- pfm.imf.org/pfmblog/2017/03/a-new-path-to-improve-fmis-performance.html. 40 Finding 13: Training and change management are essential for systems implementation. 3.51 The training is essential is among the most widely cited lessons in the FMIS projects portfolio.45 The literature and project documentation suggest that staff should receive training in the following categories: • Technical training in the application software, system software databases, wide area and local area networking systems, and information security systems selected for systems implementation • End user training in systems use • Training for managers in using information from the system’s databases. 3.52 The government should provide technical training for a select group of technical staff who are or will be responsible for maintaining and operating the system’s technical aspects. After selecting the application software, the government should identify staff from the technical and user departments who need training in key areas and send them for training. To ensure that government technical staff become well versed in the functionality and other technical aspects of the new system, the project could hire and finance one or more software specialists, who would have a dual role: (1) Specialists would help train government staff and coach them through software certification courses offered online so that a group of trained specialists is available to operate and maintain the FMIS application based on the selected software. (2) The specialists would help the government team assess whether the configured software as delivered meets the prescribed government requirements and configuration standards. This strategy was useful in the Cambodia and Pakistan implementations. 3.53 End user training requires the most attention because of the large number of staff involved in operations of the system. Adequate planning can prevent serious disruption of ongoing staff activities. Many governments successfully provided training to staff through training institutions. Matching the training program to requirements is essential, along with focusing the training and training materials. Most staff need to learn only specific features of the system, and training could initially be limited to those features. Here it is helpful for training to be coordinated closely with implementation plans focused on the specific requirements of a given site, and provided just before site 45See Diamond and Khemani (2005), Dener (2011), and Combaz (2015) and project documentation for FMIS operations in Argentina, Burkina Faso, Cabo Verde, Cambodia, Cameroon, Chile, The Gambia, Guatemala, Jamaica, Lao PDR, Mongolia, Nigeria, Pakistan, Sierra Leone, and the República Bolivariana de Venezuela. 41 41 implementation. Other important training steps could include setting up a help desk, conducting hand-holding clinics, and thoroughly training a group of power users to be trainers or technical resources for other staff. Experience suggests that operational end user staff can pick up automated systems with a well-managed training program, as was the case in Pakistan. 3.54 Training managers to use the system helps them understand the system’s capabilities and use them adequately. This training category is important because government senior finance managers can be unacquainted with using computer-based systems and are reluctant to learn from junior staff. They thus depend on junior staff to produce the required information, which imposes a limitation. At the least, managers should be familiar with what the system can and cannot do, so they become sovereign users and apply the system to its full potential. 3.55 Effective change management programs, as noted in many World Bank ICRs, are important.46 There is voluminous literature available on change management and its application to organizational change in public sector organizations in general. Central to this literature is the role of political leadership, outreach, and communication to mitigate opposition to reform. The interested reader is referred to a 2015 guide for practitioners, which also includes a comprehensive literature review (World Bank 2015).47 The study identifies the following twelve key challenges: high level political support; developing adaptive leadership skills; stakeholder mapping; development of a shared vision; an effective communications strategy; readiness of reform teams and project leaders; a well-defined deployment plan; a capacity building plan and appropriate institutional framework; system integration and transition capacity; managing resistance; assessing risks; and taking into account the political context. 3.56 With respect to FMIS, a recent World Bank study derives important lessons from the experience in Cambodia and Indonesia, among others, also on the role of political economy (World Bank 2017). Lessons from Chile’s FMIS reform are carefully documented by Dorotinsky and Watkins (2013) and stress the importance of internal capacity to developing a conceptual model; establishing user committees to validate a preliminary design; and giving priority in the development phase to the core functions (based on a 2011 presentation by Gerardo Uña). However, while an in-depth review of change management is beyond the scope of this study, the following aspect was identified as a common thread across countries: that staff in implementing agencies 46See Chêne (2009), World Bank (2017), World Bank (2015), and project documentation for FMIS operations in Cambodia, Chile, El Salvador, Kenya, Kyrgyz Republic, Malawi, Moldova, Nigeria, Russia, Slovak Republic, Tanzania, and Vietnam. 47 The experience of six OECD countries is presented by Huerta Melchor (2008). 42 recognize the inevitability of change and phase out the legacy system entirely. Some steps that countries have taken to do so include: • Ministry of finance management is fully convinced of the new system’s advantages and confident that implementing the system will proceed as planned. Management could explain and lead the transition process and assure the affected staff will not be eliminated although the nature of their work will change. • Governments should take special measures to reduce the stress of change by providing staff with more formal training and mentoring support during the transition to the new system. • Ministry of finance management could ensure that the legacy system is decommissioned as soon as possible after the new system goes online so that staff knows there is no possibility of going back. • Management needs to ensure that only the new system is used going forward. For example, for payroll, staff will process payments only through the system and pay only staff whose data are on the payroll file. Similarly, staff will process all bills only through the system. Despite initial resistance, the review noted that staff can adapt to the changed environment when management clarifies the rationale and inevitability. 3.57 Some governments adopted such change management policies, including Kazakhstan, Russia, and Vietnam. They mandated payments to be processed only through the new system once it was operational, which ensured that line agencies transitioned quickly from the older systems. Adequate support and help desk facilities ensured a smooth transition. Conversely, in Malawi, it was possible for line agencies to process transactions manually even after the system was operational, resulting in a situation in which line agencies continued to bypass the system and its controls. ONGOING SYSTEMS OPERATIONS AND MAINTENANCE Finding 14: Along with the investment costs incurred in first setting up an FMIS, provisions should be made to cover expected recurrent costs to keep the system operational. 3.58 Governments often make adequate provisions for capital investment for an FMIS, but do not set aside sufficient budget to cover ongoing recurrent expenditures.48 One factor for this may be that external donors often finance the initial investment. Subsequent, recurrent expenditures need to integrate fully into the budget and compete against other priorities. Maintaining and upgrading the technology 48See Diamond and Khemani (2005) and project documentation for FMIS operations in Bolivia, Burkina Faso, Colombia, Ghana, Jamaica, and Malawi. 43 43 infrastructure (software and hardware) can be costly, but it is important from a security and continuity perspective. If government fails to recognize the importance of expenditures related to recurring cost elements and fails to make adequate budget provisions to finance them, systems performance could degrade in the medium term and, in extreme cases, lead to suspension of system operations with grave consequences for the day-to-day financial operations the system supports. 3.59 The main recurring cost elements were estimated at about 10–15 percent of the initial investment.49 These include the following: • Hardware maintenance or replacement: Maintenance can be estimated to cost between 10–15 percent of the installed hardware costs per year. Hardware life cycles vary from 3–5 years. • Application software and database management system, and tools license fees: This highly significant element could easily cost 20–22 percent of the initial license fees per year. • Telecommunications costs: These are associated with use of the wide area network for communicating between remote and central sites. The costs vary with the telecommunications tariff operational in various countries, but they can be significant. • Stationary, utilities, and office premises. • Staff costs: Specialist technical positions are needed to ensure systems operation and maintenance, and this is a critical cost element. In many countries, these positions are filled by contract staff hired directly from outside the normal civil service structure because the government pay scales cannot accommodate the pay required. • Ongoing training costs: These costs continue throughout the life of the system because in most governments, trained end users rotate out of their positions after a few years, and governments fill these positions with new, untrained staff. Technical skills may also need upgrading as new versions of the application software and hardware are implemented. 3.60 Ghana and Malawi did not adequately provide for recurrent costs. Failure to install timely upgrades in Malawi led to system deterioration with long-outdated software, insufficient capacity to manage transaction requirements, and system exposure to various security risks. This situation alienated stakeholders and led to a lack of trust in the system. Budget provisions for use of the telecommunication network 49This is discussed in Hashim (2014). For this calculation, the initial investments were considered hardware, cost of software, installation, telecommunication infrastructure, and site preparation. 44 in Ghana were inadequate, which led to an arrears accumulation. Such service disruptions are a major risk to connectivity. Finding 15: Nonavailability of technical expertise has become a key vulnerability for system sustainability in some countries. In many low-income countries, the preferred option is to create an organizational unit within government that attracts professionals on long-term contracts. In more advanced countries, consideration should be given to outsourcing technical operations and maintenance using cloud services. 3.61 Technical expertise is critical for systems operation, but it is often difficult to attract at regular government pay scales.50 The FMIS becomes indispensable for budget management once it is operational and widely applied. It underpins the day-to-day functionality of all government accounting and budget entities. Therefore, any disruption in the system operation can cause significant disruption in the government’s financial operations. Establishing and staffing the supporting organization is crucial for the system’s long-term operational sustainability and the continuity of the government’s budget and financial management operations and reporting. Typically, a central organization is responsible for maintaining database integrity and ensuring that the systems network operation is smooth and uninterrupted. Mostly technical experts should staff this organization, which also provides technical support for end users. However, in many countries the technical skills required for these jobs are not generally available at the normal government salary scales. Government pay for technical staff for such an organization, combined with the lack of career prospects for technical information technology staff within government, is a major area of concern for FMIS project sustainability (in Maldives, Nepal, Pakistan, and Zambia, for example). 3.62 Countries used several strategies to incentivize staff retention. Government must plan the central institution’s organizational structure to perform the key technical tasks associated with the FMIS’ operation and maintenance and develop strategies to attract and retain the technical skills this unit requires. Retaining the highly skilled technical personnel required for system operation and maintenance is a common problem many governments face. Several options countries have used to address this issue include the following: • Option A - Create an organizational unit: Establish an organizational unit within government and staff it with employees paid at special market-based salary scales (for external personnel and qualified civil servants) to attract professionals on long-term contracts. 50 See project documentation for FMIS operations in Afghanistan, Bangladesh, Guatemala, Hungary, Kazakhstan, Maldives, Moldova, Nepal, Nicaragua, Pakistan, Turkey, Uganda, and Zambia. 45 45 • Option B - Hire technical skills directly: Hire the technical skills directly from the market on year-to-year contracts and manage them through regular, mainstreamed civil servants. This is a short-term measure usually adopted in the development phases of projects such as the Pakistan Improvement to Financial Reporting and Auditing Financial Accounting and Budgeting System implementation, but this arrangement might not be a suitable long-term solution. Without competent in-house technical management expertise, governments might hire short-term consultants to fulfill immediate needs, which can limit development of a long-term vision and plan for systems, the technology platform, and requirements. This can leave serious gaps in the skills and expertise required to run a stable, mission-critical system and plan for its future growth and enhancement. • Option C – Outsource technical maintenance and operations: Outsource the systems technical maintenance and operation to specialized firms. Countries with a developed market for services and firms that regularly cater to these needs (such as the United States) mainly use this option. This option could be the easiest to implement, but it could also be the most expensive because firms charge a significant overhead for providing the needed technical skills and expertise. This option requires sufficient in-house technical contract management capacity and considering issues such as information security. 3.63 Regarding option C, cloud services are increasingly attractive because they are readily available and reduce the need for technical staff within the government agency. Cloud technology makes it possible to house primary and disaster-recovery servers on external sites. Many firms host cloud services, but application development and maintenance is still the government agency’s responsibility. However, an advantage is that cloud service firms will upgrade or replace the technology periodically to respond to changing needs or outdated technology and can supply the necessary computing power, memory and disk storage, and bandwidth when needed, freeing the government agency from the responsibility. 3.64 Option A has been the preferred option for many low-income countries. It can build permanent in-house capacity within government and provide a career path for government employees appointed to the organization. Here government should develop job descriptions for the positions listed in the chart and specify the technical qualifications and experience required for the positions. Officers within the government who fulfill the technical qualifications and experience criteria could fill these positions. The key requirement is to match individuals’ skills with the positions’ required skills and experience profile. In this scenario, government may wish to review the organizational structure to validate the requirements for the central technical 46 organization to ensure that it caters to all aspects and technical requirements for running the systems. FMIS Coverage and Utilization Finding 16: Benefits from an FMIS only apply to the funds routed through it. The share of the budget subjected to FMIS ex ante controls could be used as a good proxy for FMIS contribution to effective budget management. 3.65 Transactions related to all budgetary, extra budgetary, and internally generated funds need to be routed through the system in ex ante mode, in order to benefit from the system.51 This is necessary to ensure treasury control over all government financial resources, application of control protocols (such as budget check and commitment and cash control) to all expenditure transactions before they are paid, and to promptly capture and recognize all receipts in the treasury books.52 Malaysia and Thailand, for example, implemented comprehensive transaction processing much better than many other countries in the region. Vietnam is on track to achieve similar status, as is Indonesia under the recently completed Government Financial Management and Revenue Administration Project and the SPAN (Sistem Perbendaharaan dan Anggaran Negara) project. The Indonesia and Pakistan examples illustrate this point: 3.66 In Indonesia, the two projects designed in the 1990s envisioned capturing transaction in ex post mode after the payment offices made the payment. These attempts were unsuccessful because the only way to guarantee that the system would capture all transactions after the payment was made was to ensure that no payments were made outside the system and that the system captured the transaction before initiating a payment. Therefore, in the Government Financial Management and Revenue Administration Project, the budget administrators and treasury offices were connected to the system, and the system captured all stages of the transaction in ex ante mode (before payment was made) at the payment order stage (with a resulting commitment), followed by the invoice, the goods report, and the payment. Today, the system (SPAN) is fully functioning and captures most of the budget, providing a sound basis for budget management. 51 See project documentation for FMIS operations in Indonesia, Pakistan, and Vietnam. 52 To be consistent with the requirement for developing systems along functional and not organizational lines, individual line ministries should not develop separate payment systems and should use the central treasury system for all payments and receipts. 47 47 3.67 In Pakistan, the Improvement to Financial Reporting and Auditing (PIFRA) system has gaps because government agencies with departmentalized accounts at both the federal and provincial levels do not use the system for detailed ex ante transaction entry and control. This includes the self-accounting agencies with departmentalized accounts, such as the defense department, railways, the post office, and others at the federal level, and the public works department, forest, irrigation, and others at the provincial level. The system includes only periodic summaries of transactions and the corresponding accounts ex post in the Financial Accounting and Budgeting System databases. Because Pakistan records a large portion of budgetary transactions ex post, there was no way to ensure that all transactions are recorded and that the system’s financial reports are complete. Finding 17: Benefits from an FMIS will accrue only if the system’s control protocols are diligently applied. The practice of incurring budgetary commitments outside the system is prone to the accumulation of arrears. Benefits from FMIS necessitate a diligent application of system protocols. 3.68 The FMIS needs an underlying commitment to manage public expenditures efficiently and with integrity, accountability, and adherence to the budget law to be effective. This requires significant political capital to overcome political economy issues and resistance from those with potentially vested interests.53 Automated financial management systems offer much stronger control over public finances, but benefits from an FMIS will accrue only by diligently applying the system’s control protocols (Diamond 2013 and Hashim 2014). 3.69 A functioning FMIS in Malawi that did not deliver because of a lack of political commitment illustrates this point.54 The government successfully implemented an FMIS with a set of core modules, but expenditure management was still lacking because of weaknesses in the control environment. Security firewall protocols were dismantled without authorization and other control mechanisms were ignored because of a breakdown in the accountability chain. This led to a major corruption episode. Disregard for existing systems and processes caused this situation, not an FMIS technical failure.55 Therefore, it is essential to communicate these aspects to counterparts in government and donor partners and obtain government commitment to address these as required before starting an FMIS project. 53 See Abdulbaqi and Ahmad (2012) for the relationship between FMIS and corruption. 54 See project documentation for FMIS operations in Malawi. 55 Like the FMIS, the scope of the human resource management information system must be comprehensive, and its controls should cover all civil service positions to avoid introduction of ghost workers. Similarly, the human resource management information system remains vulnerable to weaknesses in the control environment or to collusion by officials at the central level with those at the line ministry level, as was evidenced in Malawi. 48 Finding 18. Commitment management is essential for budgetary control. 3.70 Satisfactory budgetary control cannot be exercised by only checking for budget availability at the payment stage of an expenditure transaction. 56 At the payment stage this is too late to be effective, as goods and services were already received, and the government is legally obligated to pay the vendor. Commitment control moves the budget-checking control from the payment process to the time when goods and services are requested and ensures that spending units do not enter into contractual or other binding liabilities beyond their authorized budget (Hashim 2014). Along with the recording all accounts payable transactions, commitment control ensures that the government has complete information on its total expenditure liabilities and any build- up of arrears. However, effective commitment control requires closing all avenues often used to undermine it. A more detailed discussion of other controls that should be incorporated is provided in Appendix D.57 Finding 19: Delayed budget releases may cause spending units to bypass the system to avoid budgetary controls. Procedures to improve budgetary allocations from day one are important to maintain confidence and utilization of the system. 3.71 Investing in financial management information systems (FMISs) needs accompanying improvement in general budget management practices to derive significant benefits.58 Procedures for budget release and warrants should be improved so that allocations are available to line ministries to start their programs from the first day of the new fiscal year. A partial release system based on historical trends in previous years could be put in place so that line ministries can proceed with their day- to-day operations at the start of the fiscal year. Evidence has shown that without timely budget release procedures, managers develop informal arrangements to bypass the FMIS’ commitment and budget control procedures (Ghana and Malawi are examples). Thus, the line ministries reportedly entered into informal credit agreements with suppliers for goods and services delivery, for which payment is made after the release actually takes place. Then the FMIS produces a purchase order, and commitment and invoice transactions are entered. This defeats the commitment system’s purpose. Sometimes payment is from funds from other sources (such as internally generated funds and the like) that are not banked in the treasury single account (TSA) and are not 56 See project documentation for FMIS operations in Malawi, Pakistan, and Vietnam. 57 Appendix D can be accessed at this link. 58See project documentation for FMIS operations in Bolivia, The Gambia, Ghana, Malawi, Mongolia, the Slovak Republic, and Tajikistan. 49 49 under control of the treasury. The laxity in budget release procedures also builds up resistance in line ministries for including these funds in the TSA. 3.72 The treasury should have enough funds to cover the invoices generated once a budget release takes place (a warrant was issued). If the treasury does not have enough funds to cover budget releases, this may lead to delays in payment of invoices, and accumulation of arrears. It was observed (in Ghana, for example) that this is not always the case, and payments on invoices have to be delayed, which causes a build-up of arrears. If this becomes regular practice, then suppliers start building in margins in their invoices to cover the cost of delayed payments. 3.73 Implementing FMISs in cases with late budgetary releases or insufficient funds at the treasury would involve implementing only the mechanics of these systems and give the appearance of improved budget management without substance. Finding 20: Implementing advanced budgeting methodologies without an operational budget execution system is unlikely to achieve significant results. “Effective expenditure control is the sine qua non of good public financial management (PFM). Fiscal rules, medium-term budget plans, and annual budgets are meaningless if expenditure cannot be controlled during execution.” (Pattanayak 2016, 1) 3.74 Implementing a functioning FMIS is an essential prerequisite for implementing advanced budget methodologies. The role of advanced budgeting methodologies has been discussed in a number of FMIS operations.59 Program budgeting facilitates tracking the budget resources allocated to government programs, subprograms, and activities instead of against inputs (line items) only. Including program, subprogram, and activity codes in the budget classification structure can accommodate this. However, if codes against which resources are allocated are too detailed and budget control is exercised at this level, the number of transactions associated with budget apportionment and allotment will increase and raise the numbers of budget releases and transfers to unmanageable levels.60 The transaction traffic related to budget management and budget execution could increase ten-fold compared with a system that allocates and controls the budget at a line-item level. This would be difficult to manage without an automated FMIS. In Cambodia, for example, 59See project documentation for FMIS operations in Bolivia, Cambodia, The Gambia, Ghana, Mongolia, the Slovak Republic, Tajikistan, and Zambia. 60 One way the burden on budget execution could be mitigated is through controlling resources at a higher level but reporting expenditure at the detailed level. 50 the number of activities for the ministry of agriculture alone is about 450 and more than 10,000 for all ministries combined. Budget control is exercised at this level of detail in this case. Similar detailed programs were established in Ghana and Zambia, among other countries. Therefore, implementing a budget execution system is an essential prerequisite for implementing an advanced budgeting methodology such as program budgeting, and it may not be possible to achieve the full benefits of program budgeting without a fully functioning FMIS. 3.75 Prioritizing is important. The importance of sequencing in public financial management reform and the paramount role of budget execution are broadly recognized by the literature (see for example Schick 1998a, Diamond 2010, and Peterson 2011). Sequencing of these reforms has been noted as important in The Gambia. The ICR notes that “the original plan to simultaneously develop an IFMIS and MTEF was overambitious. Without decisive steps to strategically focus first on IFMIS, the project could have ended with neither reforms completed.” The role of sequencing is discussed in more detail in appendix C.61 The World Bank’s Role in FMIS Reform Finding 21: FMIS projects can be costly and lengthy. World Bank projects should be realistic during appraisal and choose adequate lending instruments. 3.76 FMIS projects in medium and large countries have had slow disbursement rates in the early years and could take 10 years or more to complete because of complex procurement, implementation, and rollout procedures. Slow disbursement rates in the early years were spent on system design, testing, and piloting—activities requiring relatively small outlays, but long time frames. The large disbursements occur toward the end of the project after completing design and testing at the pilot sites and the system is ready for rollout.62 3.77 The standard investment project financing lending instrument is not well suited for FMIS projects. The IPF instruments requires a relatively short completion period and a frontloaded disbursement schedule. This does not cater well to FMIS realities. Thus, task managers may consequently make unrealistic projections, which undermines project performance. The Cambodia Public Financial Management and 61 Appendix C can be accessed at this link. 62See Diamond and Khemani (2005), Dener et al (2011), and Hashim (2014) and project documentation for FMIS operations in Albania, Algeria, Argentina, Bangladesh, Cabo Verde, Chile, Ethiopia, Honduras, Indonesia, Kenya, Kyrgyz Republic, Lao PDR, Malawi, Moldova, Uganda, Russia, and Zambia. 51 51 Accountability Project’s design and quality at entry received significant criticism for this reason. 3.78 An adjustable program loan (APL) might be an attractive option to cater for the medium term. An appropriately phased project could deliver a usable part of the system at the end of each phase, which is necessary to maintain the project’s credibility during the long development period. An APL however carries the risk that financing may not materialize in later stages, and it is important to take appropriate precautions such that the initial investment is not lost. A carefully phased engagement could for example deliver a usable part of the system in each phase to maintain the project’s credibility. The Russian Federation Treasury Development Project (P064508) used this strategy, where the first phase of the APL provided sufficient financing to establish a working core treasury system, even though the second phase did not materialize.63 3.79 Adequate costing is important, and project design should draw on the wealth of costing information already available. FMIS implementation costs vary with the implementation’s scope and scale. However, it is possible to estimate FMIS implementation costs with a given scope by using existing data on completed projects to plot the contracted cost of actual systems implementation against the number of end users connected to the system (a proxy for system scale). It is necessary to use comparable numbers. Thus, the cost elements should cover the same elements for all projects, and if this expense was incurred in different project phases (pilot implementation and rollout, for example), then these costs should be summed. Hashim (2014) estimated costs for World Bank–financed FMIS projects involving implementation of a commercial off-the-shelf (COTS) package (figure 5), plotting the contracted costs (which cover hardware, software, implementation services, wide area and local area networking, and training) against the number of COTS user licenses acquired (a proxy for number of end users). These data can help predict the approximate cost per user at about $15,000, which is similar to an estimation by Dener et al (2011). This metric can identify significant outliers in both planned projects and evaluation of completed projects, as was done for projects in Ethiopia and Lao PDR, and an efficiency assessment for Malawi. 63 Zambia also adopted a two-step implementation under the Public Service Capacity Building Project and the Public Expenditure Management and Financial Accountability Program to start with an interim, basic, custom- developed system (FMS) until a full-function FMIS was procured and implemented (which can take several years). This was practical and reasonable. However, a key difference between the Zambia interim system and those implemented in Kazakhstan and Russia is that Zambia’s interim FMS system and the commitment control systems are both based on ex post recording of expenditure transactions after payment, which undermines basic functionality and only gives the illusion of control. Instead of correcting this problem during the APL’s second phase, the Public Expenditure Management and Financial Accountability Program institutionalized the approach by using the FMIS only at the central and provincial levels. 52 Figure 5. FMIS implementation costs plotted against number of users 60 50 Vietnam 40 US$ millions 30 Pakistan Indonesia 20 Azerbaijan Kazakhstan Moldova 10 Mongolia Albania Maldives Kyrgyz 0 Republic 0 1,000 2,000 3,000 4,000 5,000 6,000 Number of users (COTS licenses) Note: COTS = commercial off-the-shelf. Source: Hashim 2014. 3.80 Costing data for custom-developed systems and recurrent costs is unavailable. The results of a comparable exercise for custom-developed implementations may not be as reliable because a combination of local government staff and consultants often develops the software and does not capture the associated costs accurately during the long systems development life cycle. In such cases, the implementation is often incremental and is constantly changing, with many iterations done during several years, each yielding a slightly better functionality. Similarly, comparable data for recurrent costs are unavailable for the actual recurrent costs associated with ongoing system operation and maintenance. Finding 22: The World Bank team needs to have experienced specialists available to advise the client on important technical issues during the design and implementation phases. The specialists should also be familiar with procurement practices and procedures for information technology procurements and the World Bank rules under which they will need to be applied. Continuity of World Bank staff from project design throughout implementation is a crucial success factor. 3.81 It is important to take a proactive role in project design and implementation.64 Having experienced World Bank specialists available to advise clients on important technical contract management and procurement issues during the design and implementation phases were identified as a critical success factor in many cases. In Cambodia for example the availability of expertise was a turning point in 64See project documentation for FMIS operations in Bolivia, Cambodia, Cameroon, China, Indonesia, Kenya, Timor-Leste, and Zambia. 53 53 implementation. The Zambia project also highlights the importance of good technical supervision from the World Bank team, where lacking technical expertise in the initial stages, allowed the contractor to take advantage of the government’s lack of experience. This caused significant losses and jeopardized implementation. 3.82 A tendency for high turnover in World Bank project staff negatively affects performance.65 Continuity in the World Bank team was an important success factor in projects in Kazakhstan, Russia, and Vietnam because key staff involved in the project design were also instrumental during implementation. However, often key members of the team that designs and appraises the project leave implementation support to other staff who were not involved in the discussions leading up to project approval. Similarly, on the government side counterparts often change further compounding the problem, leading to a situation where both sides are not fully informed about the project’s underlying rationale and implementation strategy. While it may be necessary that task team leaders change frequently, projects would likely benefit if key technical staff remain on the team even if their regional affiliations change. The frequent change of task team leaders that happened in Cambodia and Malawi had a serious, adverse impact on implementation. Finding 23: Development policy financing, though underused, can provide critical leverage to address political economy constraints. 3.83 Policy actions related to public sector governance are most widely used in development policy financing (DPF), but include few actions related to FMIS implementation. Since the first FMIS project in 1984, there have been 6,987 policy actions related to public sector governance, which is about 28 percent of all policy actions. However, only 47 of these related directly to FMIS implementation, and only 56 related to treasury single account (TSA) implementation. Thus, only 0.7 percent of public sector governance actions relate to FMIS and 0.8 percent to the TSA. This is a small fraction of the total actions.66 3.84 Development policy financing lends itself better for addressing political economy bottlenecks than for supporting investment infrastructure. Investment lending supplemented with technical assistance has been the main lending instrument 65See project documentation for FMIS operations in Bangladesh, Burundi, Cabo Verde, Cambodia, Colombia, Hungary, Kazakhstan, Malawi, Pakistan, Timor-Leste, Russia, the República Bolivariana de Venezuela, and Vietnam. 66 This analysis is based on a prior action database maintained by World Bank Operations Policy and Country Services. All actions with a public-sector governance theme code (25, 26, 27, 28, 29, 30, 90, and 94) were considered public sector governance. If a prior action explicitly mentioned the FMIS or the treasury single account, it was tagged as such. 54 type for supporting FMIS and TSA infrastructure. However, in cases where the infrastructure is not adequately used, a DPF could be an effective accompanying measure.67 A DPF operation could incentivize countries to adopt measures that would make the FMIS infrastructure investment more effective for budget management through policy actions. For example, through requiring the counterpart to minimize the transactions occurring outside the system, introduction of the chart of accounts, or facilitating an increase in TSA and FMIS coverage. 3.85 Although some cross-conditionalities have been found successful, there are many missed opportunities. In Pakistan, for example, the World Bank team and the IMF included conditionality related to adopting and implementing the newly designed chart of accounts and budget classification structure as one of the triggers for a policy- based lending operation. This contributed significantly to the effectiveness of FMIS. However, in a missed opportunity, in Malawi, parallel development policy operations could have incorporated conditionality related to establishing an effective control environment, which could have improved FMIS use and led to better results in budget management. In Ghana, where many transactions continued to occur outside the system, budget support could have provided the necessary leverage to facilitate improved coverage. An attempt to expedite the TSA implementation process through budget support operations failed because of political resistance. Finding 24: The attribution to improved PFM outcomes can be facilitated through good monitoring and evaluation frameworks. 3.86 FMIS projects have often had high-level objectives along the lines of improved service delivery, improved public sector effectiveness, better expenditure management, or improved transparency and accountability. Reviewing all objectives of the 126 FMIS World Bank projects that were approved since 1984 shows that the majority (40.5 percent) were focused on accountability, fiduciary responsibility, and oversight, followed by transparency (39.7 percent), efficiency (35.7 percent), and effectiveness (20.6 percent). An overview of FMIS objectives is given in table 2 in the introductory section. 3.87 M&E Frameworks are frequently lacking, making attribution to objectives difficult to establish. M&E frameworks of FMIS projects tend to be inadequate. The Independent Evaluation Group (IEG) considered monitoring and evaluation of 77 percent of all closed FMIS projects to be of modest or worse quality. This gives little confidence that progress in investments can adequately be tracked to intermediate or 67See project documentation for FMIS operations in Bolivia, Ghana, Kyrgyz Republic, Malawi, Nicaragua, Pakistan, and Turkey. 55 55 final outcomes. This is problematic from a reputational perspective as it makes it difficult to attribute improvements in outcomes to the FMIS investment. 3.88 Strong M&E systems can help with course correction.68 While often M&E is lacking, there is evidence that good M&E design and utilization can support the project during implementation and lead to better results. The Nigeria ICR made this point explicit, noting that “an adequate M&E arrangement from the onset, which regularly informs on the gaps between expected and realized activities, is critical not only for measuring progress toward PDOs, but also for making decisions that keep outputs aligned with the original PDOs. Because of the absence of an adequate M&E, tracking implementation status of all components and thus actual progress toward achievement of the PDOs turned out to be more difficult than anticipated.” 4. Concluding Remarks 4.1 Investing in financial management information systems can lead to a wide array of budget management improvements. Some of the benefits include readily and reliable financial information to guide policy decisions; facilitating a prudent fiscal stance, through the application of budget controls; improved accountability and transparency; better cash management and efficiency gains; comprehensive and reliable fiscal reporting to guide macroeconomic management; regular budget execution reports for spending units; improved expenditure tracking that can guide service delivery and help reallocate funds to priority sectors; increased efficiency in bill tracking and payment processing; better government program management, better management of capital projects; efficiency gains in transaction audits allowing for more emphasis on performance; certified annual accounts produced earlier and placed before the legislature sooner; and facilitating the reliance on country systems by donor and rating agencies. 4.2 A programmatically coherent approach will make for a most effective engagement. To achieve results the report argues that attention to a diagnostic, the system lifecycle, and coverage and utilization should be given: • A careful and comprehensive diagnostic is important as it guides the investment. Without assessing gaps in capacity, coverage, and scope, the investments may be ineffective for budget management. For example, the returns for additional investments in technological aspects will be limited if the coverage and 68 See project documentation for FMIS operations in Cambodia, Ghana, Malawi, Nigeria, Vietnam, and Zambia. 56 utilization of the system is the problem. Thus, the reform agenda should critically be informed by a comprehensive diagnostic that covers all relevant aspects of budget management. • The system lifecycle covers the steps necessary to make a system operation. This includes a process and system design, procurement, implementation, maintenance, and change management aspects. Problems in this phase can lead to delays with few tangible results that undermine the reform momentum. • Benefits to the investment necessitate that the system is used. Even systems that are of a technically high standard, but only cover a limited share of the budget will not yield the desired budget management benefits. Thus, it is important to monitor system coverage and utilization even after the system is implemented and operational. 4.3 This report identifies practical and operational issues that may arise during design, procurement, and implementation of FMIS systems and summarizes lessons on what has worked. It is an effort to make the vast World Bank experience with FMIS operations more readily accessible to practitioners engaged in such reforms to allow them to make better informed decisions or take calculated risks. Lessons are mapped against the three stages in the FMIS framework: (i) the diagnostic phase; (ii) the system lifecycle; and (iii) implementation and utilization. The individual lessons are summarized in the overview. 4.4 A conversation around how system deficiencies affect budget management is important. Most countries around the world have implemented FMIS systems, and they vary in quality, capacity and scope. They are set in different institutional contexts and differ in their ability to support various elements of budget management. It is important to have a conversation about how system deficiencies affect budget management, and involve stakeholders from government, civil society, and donor agencies. As not all parties can be expected to be fully conversant in the many intricacies of FMIS, it is important that such a conversation be guided by a thorough and objective diagnostic that is cognizant of larger budget management issues. Such a conversation could take many directions. For example, if there are problems in the policy and institutional environment, it is important to transparently discuss how these affect the system’s ability to support budget management. As the FMIS expedites the speed of transactions, can an insufficient policy environment lead to the a hemorrhage of funds? Similarly, which budget items are routed through the FMIS, which ones are not, and why? What consequences does it have for budget management if large transactions happen outside 57 57 the system? How reliable is transaction processing, and what are the consequences for the integrity of FMIS reporting? 4.5 Involve sector experts and economists in the dialogue. Given that FMIS data form the backbone for government financial reporting, confidence in FMIS data is paramount. Since economists and sector experts draw on FMIS reports to inform analytical work and the policy dialogue, it is important to involve them in the conversation, such that they are cognizant of the implications of FMIS deficiencies on their work. 4.6 Technological change presents opportunities and risks. Technological change presents a host of opportunities that may challenge a traditional approach to budget management. Innovations in the ICT sector are happening at a rapid pace and hold promise in strengthening accountability, transparency, and efficiency throughout the budget cycle. While it will be important to stay abreast new developments, it is equally important to caution against security risks and leaping forward without having adequate analog complements in place. The policy and institutional environment will remain paramount in determining the systems’ effectiveness in supporting budget management. Setting up an FMIS system may be a necessary prerequisite for effective budget management, but it is not sufficient. 58 References Andrews, Matthew. 2010. How Far Have Public Financial Management Reforms Come in Africa? HKS Working Paper No. RWP10-018 Ameen, A. A., and Ahmad, K. 2012. “Towards Harnessing Financial Information Systems in Reducing Corruption: A Review of Strategies”. Australian Journal of Basic and Applied Sciences. 6(8), 500-509. Bates, Mary Ann, and Rachel Glennerster. 2017. The Generalizability Puzzle. Stanford Social Innovation Review. Beschel Jr, R. P., and Ahern, M. 2012. Public financial management reform in the Middle East and North Africa: An overview of regional experience. World Bank Publications. Chêne, Marie. 2009. The Implementation of Integrated Financial Management Systems (IFMIS). U4 Helpdesk, with Transparency International & Chr. Michelsen Institute. Cangiano, Marco, Alan Gelb, and Ruth Goodwin-Groen. 2017. Digitization of Government Payments as a Key Tool of Public Financial Management. Conference Paper. Combaz, Emilie. 2015. Implementing Integrated Financial Management Information Systems. GSDRC (Governance and Social Development Resource Centre) Helpdesk Research Report 1229. Birmingham, UK: GSDRC, University of Birmingham. Cullen, R. 2017. “The Role of ICTs in Public Finance Management in Pacific SIDs: a Case for Good Governance. In Achieving Sustainable E-Government in Pacific Island States”. Springer International Publishing. Dener, Cem, Joanna Watkins, and William Leslie Dorotinsky. 2011. Financial Management Information Systems: 25 Years of World Bank Experience on What Works and What Doesn’t . A World Bank Study. Washington, DC: World Bank. Dener, Cem, and Saw Young Min. 2013. Financial Management Information Systems and Open Budget Data: Do Governments Report on Where the Money Goes? A World Bank Study. Washington, DC: World Bank. Diamond, Jack. 2013. Good Practice Note on Sequencing PFM Reforms. Washington: PEFA Secretariat. Diamond, Jack and Pokar Khemani. 2005. Introducing Financial Management Information Systems in Developing Countries. IMF Working Paper. WP/05/196 Dorotinsky 2003. Implementing Financial Management Information System Projects: The World Bank Experience. Preliminary Results. World Bank, PRMPS. Dorotinsky, William. And Yasuhiko Matsuda. 2001. Financial Management Reform in Latin America: An Institutional Perspective. Revista del CLAD Reforma y Democrácia, 23, 1-15. Dorotinsky, William, and Joanna Watkins. 2013. “ Government Financial Management Information Systems.” In The International Handbook of Public Financial Management, edited by Richard Allen, Richard Hemming, and Barry H. Potter, 797–816. New York: Palgrave Macmillan. Foster, A.D., and M.R. Rosenzweig. 2010. “Microeconomics and Technology Adoption”. Annual Reviews of Economics 2 (1): 395-424. Gerring, John. 2004. What is a Case Study and What is it Good for? The American Political Science Review, Vol 98, No 2 (2004) pp. 341-354. 59 59 REFERENCES Glenday, Graham. 2015. Economic Analysis Guidance for Governance and Public Sector Management Operations in Select Areas of Public Financial Management . Duke Center for International Development. Hanna, Rema, Sendhil Mullainathan, and Joshua Schwartzstein. 2012. Learning Through Noticing: Theory and Experimental Evidence in Farming. No. w18401 National Bureau of Economic Research. Hashim, Ali. 2014. A Handbook on Financial Management Information Systems for Government: A Practitioners Guide for Setting Reform Priorities, Systems Design, and Implementation. Africa Operations Services Series. Washington, DC: World Bank. Hashim, Ali and Bill Allan. 1999. Information Systems for Government Fiscal Management. World Bank Discussion Paper. World Bank, Washington, DC. Hashim, Ali, and Bill Allan. 2001. “Treasury Reference Model.” World Bank Technical Paper No. WTP505, World Bank, Washington, DC. Hashim, Ali, and Allister Moon. 2004. Treasury Diagnostic Tool Kit. World Bank Working Paper No. 19. Washington, DC: World Bank. Hashim, Ali, and Moritz Piatti. 2016. “A Diagnostic Framework to Assess the Capacity of a Financial Management Information System as a Budget Management Tool”. IEG Working Paper 2016/1. World Bank, Washington, DC. Hendriks, Christoffel. 2013. “Integrated Financial Management Information Systems: Guidelines for Effective Implementation by the Public Sector of South Africa”. South African Journal of Information Management 15(1), 1-9. Hove, M. R., and Wynne, A. 2010. The Experience of Medium Term Expenditure Framework and Integrated Financial Management Information System Reforms in Sub-Saharan Africa: What Is the Balance Sheet? Occasional Paper No. 9. African Capacity Building Foundation. Huerta Melchor, Oscar. (2008). Managing Change in OECD Governments: An Introductory Framework. OECD Working Papers on Public Governance, No. 12, OECD publishing. IEG (Independent Evaluation Group). 2016a. Project Performance Assessment Report. Cambodia Public Financial Management and Accountability Project. Washington, DC: World Bank. Forthcoming. ______. 2016b. Project Performance Assessment Report. eGhana Project. Washington, DC: World Bank. ______. 2016c. Project Performance Assessment Report. Malawi Financial Management, Transparency, and Accountability Project. Washington, DC: World Bank. ______. 2016d. Project Performance Assessment Report. Vietnam: Evaluation of Public Finance Management Reform Project and Multi-Donor Trust Fund for Public Financial Management Modernization Project. Washington, DC: World Bank. ______. 2016e. Project Performance Assessment Report. Zambia Public Sector Management Program Support Project. Washington, DC: World Bank. IMF (International Monetary Fund. 2017. A New Path to Improve FMIS Performance, by Gerardo Uña. IMF Public Financial Management Blog, March 2017. Accessed on November 5 th, 2017. http://blog- pfm.imf.org/pfmblog/2017/03/a-new-path-to-improve-fmis-performance.html IMF (International Monetary Fund). 2014. Malawi: Restoring Financial Control and Accountability. IMF Fiscal Affairs Department. Washington, DC: IMF. Khemani, P., and Diamond, M. J. 2005. Introducing Financial Management Information Systems in Developing Countries. International Monetary Fund Working Paper No. 5-196 60 Khan, Abdul, and Mario Pessoa. 2010. Conceptual Design: A Critical Element of a Government Financial Management Information System Project. Technical Notes and Manuals. Fiscal Affairs Department. International Monetary Fund. Laizer, Ebenezer, and Reima Suomi. 2016. Evaluation of Integrated Financial Management Information System (IFMIS) in Tanzania. University of Turku, Finland. Nelson, R.R. and E.S. Phelps. 1966. “Investment in Humans, Technological Diffusion, and Economic Growth,” American Economic Review. 1966, 56 (1/2), 69–75. Njonde, J. N. and Kimanzi, K. 2014. Effect of Integrated Financial Management Information System on Performance of Public Sector: A case of Nairobi County Government. International Journal of Social Sciences and Entrepreneurship, 1 (12), 913-936. Pattanayak, Sailendra. 2016. Expenditure Control: Key Features, Stages, and Actors. IMF Technical Notes and Manuals, Fiscal Affairs Department. Washington, DC: IMF. Pattanayak, Sailendra, and Israel Fainboim. 2011. Treasury Single Account: An Essential Tool for Government Cash Management. IMF Technical Notes and Manuals, Fiscal Affairs Department. Washington, DC: IMF. Peterson, Stephen. 2006. Automating Public Financial Management in Developing Countries. John F. Kennedy School of Government Working Paper No. RWP06-043. Peterson, Stephen. 2007. Imperfect Systems: IFMISs in Africa. In World Bank and CABRO Conference on Budget Management and Public Financial Accountability: Pretoria, South Africa. Peterson, Stephen. 2010. Reforming Public Financial Management in Africa. HKS Faculty Research Working Paper Series RWP10-048, John F. Kennedy School of Government, Harvard University Rajaram, Anand, Tuan Minh Le, Nataliya Biletska, and Jim Brumby. 2010. "A Diagnostic Framework for Assessing Public Investment Management". World Bank Policy Research Paper 5397. Rodin-Brown, E. 2008. Integrated Financial Management Information Systems. A Practical Guide. USAID, Report prepared by The Louis Berger Group, Inc. and Development Alternatives, Inc. Schiavo-Campo, Salvatore. 2017. Government Budgeting and Expenditure Management: Principles and International Practice. Taylor & Francis. Schick, Allan. 1998a. “A Contemporary Approach to Public Expenditure Management.” World Bank Institute. Working Paper 35116. Schick, Allan. 1998b. “Why Most Developing Countries Should Not Try New Zealand Reforms.” The World Bank Research Observer 13 (1): 123–131. Schultz, T.W. 1975. “The Value of the Ability to Deal with Disequilibria”. Journal of Economic Literature 13 (3): 827–846. Sewell, D. 2004. Lessons on Sequencing Financial Management and Accountability Systems in Decentralized Post-Conflict/Low Capacity Contexts: The Case of Sierra Leone. World Bank, PRMPS. Sethi, Monika, and Anju Sharma. 2013. “Information System and System Development Life Cycle." Software Development Techniques for Constructive Information Systems Design 118 (2013). Uña, Gerado and Carlos Pimenta. 2015. “Integrated Financial Management Information Systems in Latin America: Strategic Aspects and Challenges in Public Financial Management” In Latin America. The Key to Efficiency and Transparency, edited by Carlos Pementa and Mario Pessoa, 281-323. Inter- American Development Bank. Washington DC. World Bank. 2017. Change Management that Works. Lessons from FMIS and PFM Reform in Cambodia and Indonesia. Washington, D.C.: The World Bank. 61 61 REFERENCES World Bank. 2016. Digital Dividends. World Development Report 2016. Washington, D.C.: The World Bank. World Bank. 2015. Managing Change in PFM System Reforms: A Guide for Practitioners. June 2015, Washington, D.C.: The World Bank World Bank. 1998. Public Expenditure Management Handbook. Poverty Reduction and Economic Management. Washington, D.C.: The World Bank. Yin, Robert. 2013. Case study research: Design and methods. Sage publications. Zakaria, W. Z. W., S. F. Rahman, and M. Elsayed. 2011. “An Analysis of Task Performance Outcomes Through E-accounting in Malaysia”. Journal of Public Administration and Governance, 1(2), 124-139. 62