WPS5397 Policy Research Working Paper 5397 A Diagnostic Framework for Assessing Public Investment Management Anand Rajaram Tuan Minh Le Nataliya Biletska Jim Brumby The World Bank Africa Region, Public Sector Reform and Capacity Building Unit & Poverty Reduction and Economic Management Network Public Sector Unit August 2010 Policy Research Working Paper 5397 Abstract This paper provides a pragmatic and objective diagnostic and controls (linked at appropriate stages to broader approach to the assessment of public investment budget processes) that are likely to yield the greatest management systems for governments. Since weaknesses assurance of efficiency in public investment decisions. in public investment management can negate the The approach does not seek to identify best practice, but core argument that additional fiscal space allocated to rather to identify the "must have" institutional features public investments could enhance future economic that would address major risks and provide an effective prospects, attention to the processes that govern public systemic process for managing public investments. The investment selection and management is critical. The authors also develop a diagnostic framework to assess the paper begins with a description of eight key "must-have" main stages of the public investment management cycle. features of a well-functioning public investment system: In principle, the identification of core weaknesses will (1) investment guidance, project development, and allow reforms to focus scarce managerial and technical preliminary screening; (2) formal project appraisal; (3) resources where they will yield the greatest impact. independent review of appraisal; (4) project selection In addition, the framework is intended to motivate and budgeting; (5) project implementation; (6) project governments to undertake periodic self-assessments of adjustment; (7) facility operation; and (8) project their public investment systems and design reforms to evaluation. The emphasis is placed on the basic processes enhance the productivity of public investment. This paper--a product of the Public Sector Reform and Capacity Building Unit, Africa Region; and the Public Sector Unit, Poverty Reduction and Economic Management Network--is part of a larger effort to support countries to enhance public investment management and efficiency. Policy Research Working Papers are also posted on the Web at http://econ. worldbank.org. The author may be contacted at arajaram@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team The World Bank Washington D.C. A Diagnostic Framework for Assessing Public Investment Management Anand Rajaram Tuan Minh Le Nataliya Biletska Jim Brumby This paper is an output from work undertaken in 2005-07 on Fiscal Policy for Growth and Development and part of an on-going work program by the Public Sector and Governance Unit of the Poverty Reduction and Economic Management (PREM) network to assist clients in strengthening Public Investment Management (PIM). Acknowledgements This paper benefited from the extremely useful comments provided by William Dorotinsky, Para Suriyaarachchi, Paul Wade, Jong Wook Lee, Kai Kaiser, Murray Petrie, and Lorena Viñuela that are gratefully acknowledged. The paper does not reflect the official views of the World Bank, its Executive Board, member governments, or any other entity mentioned herein. The views expressed in this paper belong solely to the authors. Contacts: Anand Rajaram, Africa Region PREM, arajaram@worldbank.org Tuan Minh Le, Africa Region PREM, tle@worldbank.org Nataliya Biletska, Public Sector Governance, PREM, nbiletska@worldbank.org Jim Brumby, Public Sector Governance, PREM, jbrumby@worldbank.org ii Contents I. I ntr oduction ........................................................................................................................1 II. Defining " M ust H ave" F eatur es for an E fficient Public I nvestment System ...............3 1. Investment Guidance, Project Development & Preliminary Screening.....................3 2. Formal Project Appraisal ...........................................................................................3 3. Independent Review of Appraisal..............................................................................7 4. Project Selection and Budgeting ................................................................................7 5. Project Implementation ..............................................................................................8 6. Project Adjustment.....................................................................................................9 7. Facility Operation ......................................................................................................9 8. Basic Completion Review and Evaluation ..............................................................10 III. Diagnostic Questions for E valuating Public I nvestment E fficiency ............................12 1. Investment Guidance, Project Development, and Preliminary Screening ...............12 2. Formal Project Appraisal .........................................................................................12 3. Independent Review of Appraisal............................................................................12 4. Project Selection and Budgeting ..............................................................................12 5. Project Implementation ............................................................................................13 6. Project Adjustment...................................................................................................13 7. Facility Operation ....................................................................................................14 8. Basic Completion Review and Evaluation ..............................................................14 IV. C onclusion ........................................................................................................................15 R efer ences .....................................................................................................................................16 A nnex........................................................................................... 17 I. Introduction 1. Many of the arguments for fiscal space are explicitly about the need to boost public investment in physical assets such as public infrastructure or in health or education facilities that contribute to improvements in human capital. 1 This argument is often weakened by evidence of low efficiency of public investment in a number of dimensions, including: · Poor project selection, including wasteful "white elephant" projects · Delays in design and completion of projects · Corrupt procurement practices · Cost over-runs · Incomplete projects · Failure to operate and maintain assets effectively so that the benefits are less than they should be. 2. The impact of such failures is to negate the core argument that additional fiscal space allocated to public investments could enhance future economic prospects. In considering the case for creating additional fiscal space for investment therefore, it is important that an assessment be made about the relative efficiency of public investment. Additionally, it would be desirable to identify the specific weaknesses that contribute to poor outcomes and suggest appropriate institutional and technical remedies that could correct such failures. 3. The approach taken in this guidance note is to begin with a description of key "must-have" features of a well-functioning public investment system. With regard to the "must have" features, the emphasis is on the basic processes and controls (linked at appropriate stages to broader budget processes) that are likely to yield the greatest assurance of efficiency in public investment decisions. The approach does not seek to identify best practice, as exemplified perhaps by a sophisticated high-level OECD or Chile-like system, but rather to identify the bare-bones institutional features that would minimize major risks and provide an effective systemic process for managing public investments. 4. A second feature of this approach is the use of diagnostic indicators of inputs, processes and outputs that would enable us to assess the functioning of actual public investment systems. These indicators should provide objective measures of inefficiency that can also help identify the decision nodes at which existing processes might be failing. 2 Thus, for example, an indicator of cost-over-runs relative to appraisal estimates may point to problems of unrealistic appraisal, poor project design or procurement or various combinations of the above which may need to be confirmed with more specific assessment. 1 The notion of "fiscal space" is best interpreted as an argument for investments whose fiscal impacts on the inter-temporal budget constraint are non-negative. 2 An obvious comment that might be made at this stage is that the economic rate of return (ERR) would provide the most appropriate measure of public investment efficiency. Where available, the ERR would be a valuable indicator and should be used. But this is often not calculated or, in many instances, is only available as an ex ante estimate. Some studies (Florio, 1997, 1999; Florio and Vignetti, 2005) suggest that ex post ERRs are systematically lower because of optimism in ex ante estimation and/or poor project implementation. The approach taken here is to develop a range of indicators which include indicators about process and institutional failures, which then provides the basis for corrective actions. Similarly, a very low rate of project completion often provides a robust indication that gate- keeping functions that prioritize public investment project selection consistent with resource availability are not effectively enforced. Poor project planning and management and procurement delays may also contribute to this outcome. 5. Using this approach in the case study countries, we suggest a diagnostic "gap- analysis" of the actual system relative to this basic system so as to identify structural aspects of the public investment decision and management process that may be weak and in need of attention. The gap analysis will be supplemented by diagnostic indicators to identify the particular areas of weakness that are likely to contribute to low public investment efficiency. As with any diagnostic, good judgment regarding the underlying incentive and capacity problems will be necessary to supplement the gap analysis and diagnostic indicators. The recent Ukraine Public Expenditure Review (PER) provides a good example of a case study that applies many of the suggestions in this note embedded in a broader understanding of public finance management practices. 3 6. In principle, a good diagnostic would allow reforms to focus scarce managerial and technical resources where they will yield the greatest impact. The approach is based on a clearly defined institutional framework and recognition of the role of institutions, capacity, and incentives. It is broadly consistent with the approach taken in the Public Expenditure and Financial Accountability (PEFA) initiative which addresses broader issues of public expenditure management. Like the PEFA framework, the diagnostic makes use of well- defined symptomatic indicators which can be objectively assessed and which provide information that can be used to identify problematic areas. This identification will enable (and typically require) more detailed assessment to develop institutional remedies to the identified problems. Thus they can motivate governments to undertake periodic self- assessments of public investment efficiency and design reforms to improve government systems. 7. This framework acknowledges that adequate data collection capacity and systems of accounting, auditing, and oversight underscore most of the identified `de minimis' features of an efficient public investment management system. 8. Section II below provides a schematic description of the "must-have" features of the public investment system. Section III then proposes some diagnostic questions and indicators that would help assess the functioning of the existing system. Section IV concludes and indicates the next steps to implementing this indicator-based approach in particular country case studies. 3 See chapter 5, "Capital Budgeting in Ukraine", in World Bank (2006). 2 II. Defining "Must Have" Features for an Efficient Public Investment System 1. Investment Guidance, Project Development & Preliminary Screening 9. First, some broad strategic guidance for public investment is often an important way to anchor government decisions and to guide sector-level decision-makers. Such guidance may be derived from a national plan or other medium to long term strategic document that establishes economy-wide development priorities at the highest decision-making levels. The Poverty Reduction Strategic Paper (PRSP) may serve as such a document in some countries. In other countries longer term vision documents may provide the necessary directional guidance. In many countries, this may be supplemented by a sector level strategy or even sub-sector strategy level that provides more detailed translation of the overarching priorities. 4 The existence of credible strategic guidance to public investment, which can be meaningfully interpreted at sector or sub-sector levels, is a basic requirement and may be referenced in annual budget preparation instructions. 5 10. Further, there should be a formal process for project development. Line ministries and spending agencies initiating projects for public investment should prepare a project profile with basic project information, including relevant strategic priority and sub-program or program, specific problem to be addressed, project objective, main activities, expected results and estimated budget. In addition, it is important at this stage that options for addressing the problem with and without a project are considered, and demand, supply and gap analysis is undertaken. 11. First level screening of all project proposals should be undertaken to ensure that they meet the minimum criteria of consistency with the strategic goals of government, and meet the budget classification tests for inclusion as a project rather than as a recurrent spending item. A project that fails to meet this consistency test should be rejected, making it unnecessary to subject it to further evaluation. An appropriate institutional arrangement to ensure that all major project proposals are screened is a critical must-have feature so that resources are not wasted in more detailed project appraisal; this function may on occasion rely substantively on responsibilities delegated to line ministries and spending agencies. 2. Formal Project Appraisal 12. Projects or programs that meet the first screening test should be subject to the appraisal of their viability which requires undertaking feasibility analysis. Its objective is to answer the essential question of whether a spending agency or line ministry should proceed with a project even though it is consistent with government priorities. This process requires 4 The U.K. requires departments to prepare Departmental Investment Strategies to guide investment decisions. Some countries produce plans for subsectors where the subsector is characterized by long planning and building cycles, such as roads, hospitals and schools. 5 Five year plans in many countries proved to be non-credible to the extent that they proposed targets that were not grounded in realistic resource projections. 3 a regulated set of project preparation steps, such as pre-feasibility study and feasibility study, including preliminary design, environmental and social impact assessments, that must be completed before a project can be approved for funding. 13. The pre-feasibility study helps identify relevant alternatives before undertaking a full- fledged feasibility study, and to find out early-on whether a proposed project is feasible. The feasibility study takes pre-feasibility analysis further by compiling all relevant data, refining project outputs and outcomes, outlining and analyzing in-depth the selected alternative of achieving project objectives, as well as undertaking various background assessments including environmental and social impact analysis. It helps to narrow the scope of a project to identify an optimal option for preliminary design. Box 1. Key Components of Feasibility Analysis Prefeasibility Study Feasibility Study Data gathering (geographic, climate, socio- Compilation of all relevant data economic, and technical) Project alternatives Alternative technologies for project Major risks (including institutional and Detailed estimate of costs and benefits for a budgetary) selected alternative Comparison of alternatives (engineering, socio- Preliminary design economic costs and benefits) Recommended project alternative Detailed risk assessment Preliminary estimate of project costs and benefits Detailed sustainability assessment Regulatory requirements Environmental impact assessment Identifying lacking information for Feasibility Social impact assessment Study 4 14. As part of feasibility analysis, projects or programs should undergo more rigorous scrutiny of their cost-benefit or cost effectiveness. The project selection process needs to ensure that projects proposed for financing have been evaluated for their social and economic value. To do so effectively, governments should have formal and well publicized guidance on the technical aspects of project appraisal appropriate to the technical capacity of ministries and departments. The guidance should describe techniques of economic evaluation that are appropriate to the scale and scope of the project ­ with larger projects requiring more rigorous tests of financial and economic feasibility and sustainability. The project appraisal process should consider project proposals of different scales and take into account the key macro, sectoral and project-specific uncertainties, such as inflation, cost overrun, change in output and key input prices over the project life. New investments should occur only when rehabilitating existing assets is not as cost effective as undertaking investment in a new asset. 15. Further, the value of ex ante project evaluation depends very much on the quality of the analysis which, in turn, depends on the capacity of staff with project evaluation skills. Upstream investment in training in project evaluation techniques is an important aspect of an effective public investment system. 6 Whether the government has an established process for training staff in project evaluation technique would be a useful indicator of capacity creation. Quantifying the number of staff in government with project evaluation skills would be another relevant indicator. Identifying whether such staff are in line positions that makes effective use of their skills is another related indicator. In many countries, technical assistance is used to train staff in skills which are then not effectively deployed. Nonetheless, attention must be paid to create systems and incentives concurrently to assure that the acquired project evaluation skills are actually applied. Having sufficient numbers of trained personnel is a necessary but not sufficient condition to improve practices of formal project appraisal. 16. It is worth noting that a full-fledged feasibility assessment that employs complex techniques of cost benefit and cost effectiveness analysis is often poorly developed and implemented in low capacity environments. Therefore, the emphasis should be made on basic elements of formal project appraisal: · the need for a project is well justified; · project's objectives are clearly specified; · broad alternative options to meet project's objectives are identified and comparatively examined; · the most promising option is subject to detailed analysis; · project costs are fully and accurately estimated; and · project benefits are assessed qualitatively as likely to justify the costs. 17. It is helpful to maintain a portfolio of the appraised projects. Such portfolio is expected to help not only to track how many projects have been selected but also allows 6See Fontaine (1997) for a description of the sustained effort undertaken by Chile to train a number of generations of public officials in project evaluation techniques. 5 revisiting rejected projects later on when underlying project circumstances change and they are likely to generate net positive benefits. Hence, all appraised projects should be recorded in a project database ranked by priority for budget consideration. 18. Projects involving non-standard procurement, such as public private partnerships (PPPs) and bundled "resources for infrastructure" projects, should be subject to the same appraisal process as standard public investment, and the costs and benefits of such projects should be compared against a public sector comparator project. 19. An important step after a project has been selected and before it is included in the budget is development of a detailed project design to ensure that the project is accurately costed and can be tendered and implemented (a "ready-to-go" check). Moreover, to facilitate project implementation, the project design should also provide a full risk assessment, performance indicators and an implementation strategy which should be used by an implementing agency. 6 3. Independent Review of Appraisal 20. It is always sound practice to subject project appraisals to an independent review. Optimism bias amongst those developing project proposals ­under-estimation of costs and over-estimation of benefits- is well documented. This function can be performed by the ministry of finance, a planning ministry, or other specialized agency. Where departments and ministries (rather than a central unit such as the ministry of planning) undertake the appraisal, an independent peer review might be necessary in order to check any subjective, self-serving bias in the evaluation. This function can be performed by the ministry of finance or by a designated specialized agency. In countries where donor-financed projects are significant, upstream aid-coordination can help channel resources to priority areas but should be subjected to the same appraisal stages as government funded projects. In this context, clarity of specific responsibilities is important. A multiplicity of players with unclear accountabilities can overburden the appraisal system. A formal set of delegations is necessary to keep minor projects away from clogging up appraisal. Box 2 Excerpts from U.K.'s "The Green Book" "Appraisal, done properly, is not rocket science, but it is crucially important and needs to be carried out carefully. Decisions taken at the appraisal stage affect the whole lifecycle of new policies, programmes and projects. Similarly, the proper evaluation of previous initiatives is essential in avoiding past mistakes and to enable us to learn from experience." "The first step is to carry out an overview to ensure that two pre-requisites are met: firstly that there is a clearly identified need' and secondly that the proposed intervention is likely to be worth the cost. The second step is to set out clearly the desired outcomes and objectives of an intervention in order to identify the full range of options that may be available to deliver them. The third step is to carry out an option appraisal. This is often the most significant part of the analysis. Following option appraisal, decision criteria and judgment should be used to select the best option or options, which should then be refined into a solution. Consultation is important at this stage...... Procurement routes should also be considered, including the role of the private sector. Issues that may have a material impact on the successful implementation of proposals must be considered during the appraisal stage, before significant resources are committed. (The main) purpose (of evaluation) is to ensure that the lessons are widely learned, communicated, and applied when assessing new proposals." Source: H.M. Treasury, "The Green Book: Appraisal and Evaluation in Central Government." 2003 4. Project Selection and Budgeting 21. It is essential that the process of appraising and selecting public investment projects is linked in an appropriate way to the budget cycle even though the project evaluation cycle may run along a different timetable. There is clearly a two way relationship between the budget cycle and the project selection cycle. The fiscal framework and the annual budget 7 need to establish envelopes for public investment (on an aggregate and/or sectoral basis) so that a sustainable investment program can be undertaken. 7 The key to efficient investment is both good decisions in choice of investments, and active management of the asset portfolio (including through disposals) and a budgetary process that ensures recurrent funding to operate and maintain existing assets. The latter is especially important for donor funded projects that create assets while operation and maintenance costs are assumed to be borne by government. Efficient investment also depends on whether the recurrent budget adjusts to reflect the impact of the capital projects. For instance, additional costs may be incurred to maintain and operate existing assets; in these circumstances, there should be consideration as to how these costs should be funded. Forward costs of investment projects, and their funding, should be reviewed systematically by both sector ministries and the ministry of finance during budget preparation. An example of how the link can be made to a revenue generating investment is shown below in Box 3. Box 3 Provision for a recurrent cost change following completion of a project The impact on the budget of O&M costs associated with a given project is shown in the following table. Approve the following changes to appropriations to implement Project ABCD, with a corresponding impact on the fiscal balance. $m ­ increase/(decrease) Year 0 Year 1 Year 2 Year 3 Outyears Vote Name1 Minister of Portfolio1 Program Name Operating Expenditure: 0.500 0.750 0.750 0.750 0.750 (Specified items) Development Expenditure: (Project Name ABCD) 1.000 2.000 0.500 - - Total Operating 0.500 0.750 0.750 0.750 0.750 Total Development 1.000 2.000 0.500 - - Fiscal Balance (Cash basis) (1.500) (2.750) (1.250) (0.750) (0.750) Source: Adapted from practice in New Zealand. 5. Project Implementation 22. Projects should be scrutinized for implementation realism. Project design should include clear organizational arrangements and a realistic timetable to ensure the capacity to implement the project. It is critical to establish and develop effective measures, such as efficient procurement plans, guidelines and institutional capacity to manage and monitor project implementation, total project cost management system and multi-year budgeting. 7 A medium term budget framework can provide some forward visibility regarding resource availability and predictability for long gestation investments. 8 Managing the total cost of projects over their life requires an accounting system that captures and reports all project costs, rather than accounting by separate contracts or stages and tracking against annual appropriations. Multi-year budgeting facilitates allocation of funds for project implementation over a project's life cycle. Project proposals, especially for a large infrastructure project, should also present organizational arrangements for running the project once the construction is finished. Problems in implementation can at times be related to poor project selection and budgeting. 6. Project Adjustment 23. The funding review process should have some flexibility to allow changes in the disbursement profile to take account of changes in project circumstances. For instance, if events transpire to make a project no longer incrementally beneficial, there should be a device via the funding approval process or the monitoring process to request project sponsors to recast the project, or even to halt disbursements. This suggests that funding should be carried out in tranches, with the tranches relating to the discrete phases of the project. Each funding request should be accompanied by an updated cost-benefit analysis and a reminder to project sponsors of their accountability for the delivery of the benefits. 24. These funding mechanisms can reinforce the nature of the monitoring process, making it an active rather than passive form of monitoring. Governments need to create the capacity to monitor implementation in a timely way and to address problems pro-actively as they are identified. Monitoring project implementation would minimally involve comparison of project progress relative to the implementation plan. Implementing agencies should be required to submit progress reports to identified monitoring agencies that may then need to audit both financial and physical implementation. 7. Facility Operation 25. Once a project is completed, there should be a process to ensure that the facility is ready for operation and services can be delivered. This requires an effective mechanism for handover of management responsibility for future operation and maintenance of the created assets and adequate budget funding of service delivery agencies to operate and maintain these assets. But the completed assets may still lie idle if they are not suitable for service delivery. Thus it is also important to verify the extent to which the newly completed facility requires post-completion adaptation or ancillary investment before the assets can be utilized. 26. In addition, asset registers need to be maintained and asset values recorded. Ideally, countries should require their operating agencies to compile balance sheets, on which the value of assets created through new fixed capital expenditure would be maintained. Whether there is accrual accounting or not, agencies should maintain asset registers which are exhaustive in their records keeping, and where necessary, legal title to property is affixed. 27. Active monitoring of service delivery is a desirable element of ensuring that the new assets serve the purpose over their useful life. This suggests that the quantity and quality of 9 service delivery associated with facility operation should be tracked through time. Moreover, agencies responsible for service delivery should be held accountable for results. 8. Basic Completion Review and Evaluation 28. Finally, a desirable but often missing feature of government systems is a basic completion review and ex-post evaluation of finished projects. Basic completion review should apply to all projects in a systematic way. It comprises an examination by a responsible agency or line ministry sometime after project completion of whether the project was finished within the original (and amended) budget and time frame, and whether the outputs were delivered as specified. As a supplement to this basic element, a supreme audit institution should periodically conduct a compliance audit of a sample of investment projects. 29. Ex post project evaluation should focus on the comparison of the project's outputs and outcomes with the established objectives in the project design. It is usually carried out two to three years or more after project completion on a highly selective basis. Good practice suggests that the project design should build in the evaluation criteria and that learning from such ex post evaluations is used to improve future project design and implementation. We include this as a "must have" feature in order to underline the need for governments, even in a basic way, to ensure that there is some learning and feedback from projects that will create a positive dynamic for improvement over time. 30. The box summarizing the approach suggested by the U.K. Green Book on appraisal and evaluation is cited as an example of good practice. While the rigor of any of the steps in public investment selection will need to be adapted to country capabilities, the emphasis here is on ensuring that attention is paid to the various aspects of good decision making, even if it is initially a rudimentary discipline. 10 Figure 1: Synchronizing Public Investment Evaluation with Budget Process 11 III. Diagnostic Questions for Evaluating Public Investment Efficiency 31. The following questions might provide the basis for a diagnostic assessment of the efficiency of a public investment management system. 1. Investment Guidance, Project Development, and Preliminary Screening 1. Is there well-publicized strategic guidance for public investment decisions at central/ministerial/provincial levels? 2. Is there an established process for screening of project proposals for basic consistency with government policy and strategic guidance? Is this process effective? What proportion of projects so screened is rejected? 2. Formal Project Appraisal 3. Is there a formal appraisal process for more detailed evaluation (whether at line ministry or central finance agency level) of public investment project proposals for costs and benefits? If yes, is appraisal mandatory for all projects or for projects above a certain monetary value? Is project appraisal undertaken only for specific sectors and if so which sectors? What proportion of public investment projects is formally appraised for costs and benefits? 3. Independent Review of Appraisal 4. Are project appraisals formally undertaken by the sponsoring department or by an external agency? What is the quality of such appraisals? 4. Project Selection and Budgeting 5. What proportion of the public investment program (PIP) is donor financed? Are donor financed projects subject to the same or different rules for appraisal and inclusion in the budget as government financed projects? If different, describe the difference. Does the government review project appraisals undertaken by donors? 6. Are appraisals screened by an external agency or department for quality and objectivity of appraisal? 7. Is final project selection undertaken as part of the budget process or prior to the budget process? Does the government maintain an inventory of appraised projects for budgetary consideration? 12 8. Is there an effective process to control the gates to the budgeted public investment program, i.e. the collection of projects that are formally approved for budget allocation and implementation? Is the number of oversight agencies limited, and their key roles clearly specified? Do delegation levels exist for bringing projects to the center? Is there an established but limited process for including projects for emergency or politically imperative reasons? 8 a. What proportion of projects enter the PIP by "climbing the fence" by avoiding the gate-keeping process? b. What proportion of projects that "climb the fence" is donor financed? 9. What is the average the value of new projects relative to the: · ongoing public investment program? · projects completed (use three year moving average)? 5. Project Implementation 10. What is the completion rate of the public investment program (annual average over the past 5 years), defined as the annual public investment budget divided by the estimated cost to complete the current public investment program? 9 How does this differ across key sectors ­ education, health, water supply and sanitation, roads and power, for example? 11. Do ministries undertake procurement plans in line with good practice (e.g. use competitive tendering)? And, if so, do they implement procurement plans effectively? 6. Project Adjustment 12. Has the government undertaken a rationalization of its public investment program in the recent past? Did the rationalization improve the prioritization of the public investment program? Did it result in the cancellation or closure of ongoing projects? If yes, what is the percent of the PIP that was cancelled or closed? Indicate if projects were merely "deferred" rather than cancelled. 8 The chart indicates the possibility of "unscreened projects" entering the budgeted program, essentially climbing the fence. An effective gate-keeping process would limit this to a very small number of emergency projects. 9 To illustrate, if the residual investment to complete the current program is $1000 and the annual investment budget is typically $100, the completion rate is 10 percent, implying 10 years to complete. A low completion rate may confirm a poor gate-keeping process that allows too many projects into the budget or it may reflect cost-escalation that causes the cost of completing projects to exceed initial estimates. 13 13. Are project implementing agencies required to prepare periodic progress reports on projects? Does this include an update on the cost benefit analysis? Are the sponsoring departments accountable for changes recorded in either costs or benefits and for the delivery of net benefits? What mechanisms exist to ensure that this occurs? Is this record of investment management used in subsequent budget discussions with the MOF or Ministry of Planning? 14. For a representative subset of the public investment program (including Bank- supported projects), what is the average percentage cost over-run (in inflation- adjusted terms) on major projects in key sectors? 7. Facility Operation 15. Are projects commissioned to private contractors and, if so, are contracts awarded on the basis of competitive bidding? Are international firms permitted to bid on contracts? If other methods are used, describe the methods. Is there any evidence from Country Procurement Assessment Reviews (CPARs) or other reviews of procurement contributing to cost escalation or fraud? 8. Basic Completion Review and Evaluation 16. Is there a process for handover of management responsibility for future operation and maintenance of the created assets to service delivery agencies? Do service delivery agencies have an adequate budget funding to operate and maintain these assets? Is service delivery associated with facility operation tracked through time? Are agencies held accountable for the delivery of services? 17. Does the government maintain an asset register or inventory of public sector property, equipment, vehicles, etc.? Is legal title to assets maintained? Are assets valued according to sound accounting principles, such that the accounting definition of an asset is met, depreciation is deducted from the asset value and where feasible, asset values are updated to reflect changed prices? 18. For a representative subset of the public investment program, what is the delay in project completion relative to initial estimated time and what is the deviation from the original (and amended) budget on major projects in key sectors? 19. Is the actual net present value (NPV) of completed projects measured, and is a project end evaluation undertaken to review the nature of differences relative to the estimated NPV at appraisal? What alternative methods, if any, are used to undertake ex post evaluation of completed projects? 14 32. The attached table provides a structured layout for an assessment of public investment management and likely efficiency of public investment. IV. Conclusion 33. PIM reforms require not only the alignment of incentives to improve project design and selection (`must have' features 1 to 3), but also credible commitments and long investment in administrative capacity to improve project implementation (`must have' features 4 to 7). The typology presented in Table 1 combines these two dimensions. The preliminary cross-country evidence shows that while a larger number of countries in the sample have strived to meet some of the required features to improve project implementation, only a handful of developing countries have been able to progressively move to the top left cell (cell A: good design and implementation). T able 1 Stylized T ypology of PI M -System Per for mance Project Implementation Well executed Poorly executed Project Design Good projects A C and Selection Poor projects B D 34. This paper provides guidance to country teams relevant to forming a pragmatic and objective assessment of the quality of public investment efficiency in a context where governments are seeking to mobilize additional fiscal resources for investment. An indicator-based approach provide a basis both for objective assessment as well as serving to highlight weaknesses that should be addressed if the use of fiscal resources is to enhance public sector assets and economic growth. The use of indicators must be accompanied by good country-specific judgments on the functioning of institutions and the underlying incentives for public sector performance, including political economy considerations. 15 References Allen, R. E. and D. Tommasi (ed.) (2001). Managing Public Expenditure ­A Reference Book for Transition Countries, OECD. Florio, M. (1997). "The Economic Rate of Return of Infrastructures and Regional Policy in the European Union", Annals of Public and Cooperative Economics, Vol. 68, No. 1, March, pp. 39-64. Florio, M. (1999). "An International Comparison of the Financial and Economic Rate of Return of Development Projects", Department of Economics Working Paper No. 99.06, University of Milan. Florio, M. and S. Vignetti (2005). "Cost-Benefit Analysis of Infrastructure Projects in an Enlarged European Union: Returns and Incentives", Economic Change and Restructuring, No. 38, pp. 179-210. Fontaine, E. R. (1997). "Project Evaluation Training and Public Investment in Chile", American Economic Review, Vol. 87, No.2, pp. 63-67. H. M. Treasury (2003). The Green Book: Appraisal and Evaluation in Central Government, U.K. H. M. Treasury, "Fiscal Policy: A New Framework for Public Investment", U.K., available at www.hm-treasury.gov.uk. World Bank (1998). Public Expenditure Management Handbook. World Bank (2006). Ukraine - Fiscal Space for Growth: A Public Finance Review, World Bank, see chapter 5 on Capital Budgeting. 16 Annex A nnex T able 1 Stages, I nstitutional A r r angements and Diagnostic I ndicator s of Public I nvestment M anagement Key Feature Stage of Public Investment Desirable Institutional Arrangement Diagnostic Indicator(s) (i) actionable strategic guidance; Strategic Guidance and Preliminary Published development strategy or vision Assess "realism" of strategy relative to and (ii) adoption of first level Screening. statement which has unambiguous authority. resource availability ­ is it actively used screening of all project to prioritize budgetary decisions? proposals relative to this guidance Various. Centralized approval by planning or Evidence of inadequate process for finance ministry (or delegated) for developing screening proposals - major projects proposals. Explicit ministry level justification inconsistent with government strategy with strategy. or vision Clarity of project objectives in terms of Sampling of proposals. outputs and outcomes. Consideration of alternative approaches to Sampling of proposals. objectives. projects or programs that meet Formal Project Appraisal, executed Publicized and transparent guidance; backed Quality of published guidance on the first screening test undergo by appropriately skilled staff (or by effective training and deployment of staff appraisal number of staff with training more rigorous scrutiny of their consultants). for project design and appraisal (including in project appraisal in line positions. cost-benefit or cost effectiveness stakeholder consultation in project design). (the value of ex ante project evaluation depends very much Application of guidance in project appraisal. Sampling of appraisals. on the quality of the analysis which, in turn, depends on the capacity of staff with project evaluation skills.) an independent peer review Independent Review of Appraisal Independent checks to ensure objectivity and Rate of rejection of project appraisals 17 checks any subjective, self- quality of appraisals (including donor funded). serving bias in the evaluation Disciplined completion of project appraisals Evidence to the contrary - appraisals prior to budget. "hurried" to meet budget timetables or downstream project design issues. Identifying and maintaining an inventory of Existence of a portfolio of appraised appraised projects ranked by priority for projects by ministries. budgetary consideration. Clarity of roles between projects which are Multiplicity of actors, with lack of minor and may be dealt with at the clarity about specific responsibility for departmental level, and those requiring proposals. additional scrutiny. Issue delegations, allowing the central appraisal system to be free of considerations of relatively important projects. the process of appraising and Project Budgeting and Selection Transparent criteria for selecting projects with Lack of transparent criteria. selecting public investment reference to policy objectives at ministerial projects is linked in an level. appropriate way to the budget cycle Well structured budget preparation process Disciplined budget calendar, with clear with scope to integrate investment and requirements for consideration of recurrent implications of projects. recurrent implications. Effective gate-keeping to ensure that only Small percentage of projects in the appraised and approved projects are selected budget that evaded established for budget financing. appraisal and selection process. High value of new project starts Ensuring adequate financing for selected relative to ongoing total capital budget projects, including recurrent needs on and to finishing projects. completion. projects are scrutinized for Project Implementation Published guidelines for project Review quality of guidelines for clarity, 18 implementation realism and then implementation. requirements for efficiency and implemented with regard to accountability. efficiency Cost-effective implementation through Evidence of competitive project procurement and contracting. tendering. Timely implementation in line with guidelines. Sample for delays in project implementation relative to appraisal estimates ­ sector specific indicators. Large stock of incomplete projects ­ data by vintage of projects by sector (look into reasons: lack of technical capability, unrealistic time-table, or under-funding). Timely implementation reports on major Review sample of reports for projects. timeliness problems of procurement on major projects by sectors. Effective budgeting for selected projects. Compare capital budget to outturn for several years. Existence of multi-year budget allocation system. Evidence of under-funding of major projects relative to actual requirements (shortfalls in budget allocations, unpaid bills, disputes over payment, etc.). Existence of total project cost management system to prevent imprudent cost increase. Estimates of cost-over-runs on major projects ­ sector specific indicators. 19 funding review has sufficient Project Adjustment Active monitoring Estimated costs and benefits updated flexibility to allow changes in the to reflect material changes in disbursement profile to take circumstances. account of changes in project circumstances Consequences of changes in estimated costs and benefits included in operating budgets, where relevant. Mechanisms exist to prevent continuing to spend money on a project when it (net of sunk costs) the benefits are not positive. process for ensuring that a Service Delivery Facility Operation Evidence of adequate funding for facility is ready for service service delivery agencies for operation delivery should be in place, and and maintenance. asset registers are maintained and asset values recorded Service delivery associated with facility operation tracked through time. Asset registers Agencies held accountable for the delivery of services. Evidence that complete asset registers maintained. The records management system facilitates valuation and custodianship. basic completion review and ex- Basic Completion Review and Formal institutional arrangements for basic Timeliness of project completion. post evaluation of completed Project Evaluation completion review and ex post evaluation of projects projects/programs with feedback into future Deviation from the original (and project designs. amended) budget. Compliance audit by SAI for a sample of projects. 20 Where ex-post evaluation exists, useful indicators may include: · Share of public investment projects subject to ex post evaluation. · Quality of evaluation and recommendations. · Evidence of response to the evaluation findings. 21