2016 Simplified Methodology for Public Investment Project Selection and Prioritization in the Kurdistan Region 15th May 2016 1 List of Acronyms and Abbreviations CBA Cost–Benefit Analysis CEA Cost-Effectiveness Analysis EA Efficiency Approach EIRR Economic Internal Rate of Return ENPV Economic Net Present Value FS Feasibility Study HSS Highly-Satisfactory Score IBP Integrated Bank of Projects KPIs Key Performance Indicators LAR Land Acquisition and Resettlement LFA Logical Framework Approach LFM Log-Frame Matrix MCA Multi-Criteria Analysis M&E Monitoring and Evaluation NDP National Development Plan NDS National Development Strategy DAC-OECD Development Assistance Committee (Organization for Economic Co-operation and Development) O&M Operation and Maintenance PF Prioritization Framework PIF Public Investment Framework PIM Public Investment Management PIMS Public Investment Management Systems PFS Pre-Feasibility Study PPP Public–Private Partnerships PSS Partly-Satisfactory Score RF Report Form SS Satisfactory Score USS Unsatisfactory Score WBG World Bank Group 2 Table of Contents 1. NOTE ON THE MANUAL ..................................................................................................................... 4 1.1 What is this manual? ............................................................................................................ 4 1.2 Who should use this manual? .............................................................................................. 4 1.3 How should this manual be used? ........................................................................................ 4 1.4 What is this manual intended to do? .................................................................................... 4 2. INTRODUCTION ................................................................................................................................... 4 2.1 Project prioritization overview............................................................................................. 7 2.2 Project selection and inclusion in the budget ....................................................................... 9 2.3 Objectives of prioritization methodology .......................................................................... 11 3. ESTABLISHING PROJECT PROFILES .................................................................................................... 12 3.1 Project lifecycle ................................................................................................................. 12 3.2 Planning and identification phase ...................................................................................... 13 3.3 Pre-investment phase ......................................................................................................... 13 4. USING PROJECT APPRAISAL TO STRENGTHEN THE SELECTION CRITERIA......................................... 14 4.1 What are we appraising? .................................................................................................... 15 4.2 Appropriate evaluation criteria and indicators ................................................................... 15 4.3 Evaluation criteria formulation .......................................................................................... 18 4.4 Establishing a rating scale .................................................................................................. 20 4.5 Establishing a project weighting scheme ........................................................................... 20 5. PROJECT SELECTION .......................................................................................................................... 23 5.1 Check list for ex-ante evaluation ....................................................................................... 24 5.2 Scoring for ex-ante evaluation ........................................................................................... 30 5.3 Ranking of selected projects .............................................................................................. 30 6. CONSOLIDATED APPLICATION ON PROJECT SELECTION................................................................... 31 6.1 Score indicators.................................................................................................................. 31 6.2 Tabulated results ................................................................................................................ 31 6.3 Prioritization and selection ................................................................................................ 32 7. FINAL COMMENTS ............................................................................................................................ 35 8. REFERENCES ...................................................................................................................................... 36 9. ANNEX N°1 – Practical Worksheet V 1.0. User Manual .................................................................... 37 3 1. NOTE ON THE MANUAL 1.1 What is this manual? This simplified manual is intended over the coming years to serve the government as a guide to fully developing its methods of selecting and prioritizing investment projects. It contains a methodology for project selection and prioritization and practical guidance on how to proceed with an appropriate methodology. This manual supplements all other budget manuals applicable for the relevant government administrative bodies at both central and governorate levels. 1.2 Who should use this manual? This manual on public investment project selection and prioritization criteria is intended for different types of user. First, it serves as a technical reference for public-sector managers and other relevant government officials who are responsible for making public-sector investment decisions at both central and local levels. But it also serves relevant technical staff in the Ministries of Finance and Planning, line ministries, other public agencies and local governments involved in the formulation, selection, implementation and evaluation of public investment projects. 1.3 How should this manual be used? The manual serves as a baseline tool to assist the government to implement necessary fiscal management reforms. This manual was developed with the aim of serving both as a desk reference for government officials already trained in public investment management (PIM), at both central and decentralized levels, and as a training tool for structured capacity- strengthening programs. 1.4 What is this manual intended to do? This manual should assist government officials involved in capital budget preparation to understand clearly and manage efficiently the use of project selection and prioritization criteria in the project cycle. A formal system of project selection should provide the basis and conditions for the government to forward only those initiatives that are demonstrably the most economically attractive for society. 2. INTRODUCTION One of the most important challenges for a government is how to allocate scarce resources between infinite needs and good investment projects. Developing countries, of course, are not the exception; in those countries, public investment plays an important role in terms of country- development process. In such cases, the improvement of public investment management systems (PIMS) should be an important priority for governments. Among other issues, to introduce better practices for conducting project selection and prioritization processes is a key matter in this context. A rigorous PIMS would enable the systematic scoring and ranking of public investment programs and projects on quality. However, to introduce a new and modern system for dealing with public investment programs and projects takes time and resources. In this case, in order to deal with shortcomings in the public investment project process, the governments should put the focus on developing key evaluation criteria, together with definitions and specific guidelines on how to apply them. 4 Thus, it is necessary to set up a systematic and comprehensive monitoring and evaluation (M&E) procedure from (i) investment viewpoints: planning, reviewing and approving investment proposals; (ii) planning, appraising and approving programs/projects; and (iii) implementation; to (iv) project operation. M&E has emerged as one of the most critical stages, which needs to be strengthened by scientific and objective solutions. Also, these criteria should follow international standards and best practices, in accordance with the project cycle, shown in Figure N°1. Phases the inN°1 Figure – The Project Project Cycle Cycle •Idea PROJECT •Project Preparation and Feasibility Studies (project cycle) •Project Financing •Project Implementation PRE-INVESTMENT PHASE •Operation INVESTMENT PHASE •Ex-Post Evaluation OPERATION PHASE •Project End •Project Identification •Appraisal FINANCING •Negotiation and Loan Disbursement •Portfolio Monitoring •Ex-Post Evaluation Source: Glenn P. Jenkins and Edgardo S. Mimica (2010). The Fundamental Institutional Pillars of an Efficient Capital Budgeting System. South Africa. An improved PIMS using M&E tools will lead to better public investment project results in terms of investment efficiency and effectiveness in achieving intended outcomes and development impacts. Better PIMS results will contribute to the economic growth of the country and to the wellbeing of the people. In a development context, evaluation is defined as a systematic and objective assessment of the design, implementation and outcome of an ongoing or completed intervention with socioeconomic development objectives. The main purposes of evaluation are:  To improve expected results from public investment projects through feedback of lessons learned; and  To provide a basis for accountability, including provision of information to stakeholders. In this context, the use of evaluation criteria for project selection will help to determine the relevance and fulfillment of objectives, efficiency, effectiveness, sustainability and impact of an intervention, using both quantitative and qualitative data. 5 Box 1: Common findings in developing countries Usually, public officials at central government level (Ministry of Planning, Ministry of Finance, line ministries, other public agencies, state-owned enterprises, etc.) are highly professional and possess a significant level of knowledge and practical skills. There is, however, a shortage of economic skills and knowledge. The current practice is only to remove projects from the pipeline if there is a shortage of public funds to implement the project. At the project proposal stage, no attention is paid to the estimation of the economic benefits of a project. The costs of publicly financed projects are considered to be justified simply because the benefits cannot be measured. Government officials, however, are aware of the capacity gap and are keen to get international assistance to address the issue. In second place, there is evidence of inefficiency in public investment management regardless of country income level in terms of: 1) Poor project selection: wasteful “white elephants” that do not transform into productive assets; 2) Unrealistic time schedules in ex-ante appraisal and consequent delays to completion; 3) Chronic under-execution of capital projects; 4) Cost overruns; and 5) Neglect to operate and maintain created assets. Thirdly, the common complaint is the absence of a management information system – or integrated bank of projects (IBP) – to keep track of projects in the pipeline. International experience shows that it is important to start tracking the project prior to the final approval stage. The project should enter the system when information is starting to become available. Information is continuously added to the project as it moves through the pipeline. The analysis at the prefeasibility and feasibility study stages will include an identification of the small number of key variables that must be monitored to determine the eventual performance of the project. The economic value of many variables will be the same for many projects. It is critical to capture and archive this information from the appraisal of all projects if we are to reduce the time and cost of appraising future public investment projects. If the project is rejected at any stage, it should remain in the database for future access. In the context of a fast- growing economy, the swift growth in demand for public services may eventually make a project, or a substitute project that addresses the same need, economically feasible. It is important, therefore, for a country to retain the knowledge gained about the project. In addition, from international best practices it is recommended that the following be considered:  Economic analysis to assess the project from the point of view of society as a whole: o Appraises project’s contribution to the economic welfare of the region or country. o It is based on a rationale following the three postulates of Harberger; the first postulate establishes that the demand curve represents the maximum willingness to pay for successive units of a good; the second postulate establishes that the supply curve represents the minimum prices that suppliers are willing to accept for successive units of a good or service. These minimum prices represent the opportunity cost of the goods. Postulate three concerns the distributional aspects of a project and how they should be incorporated into the economic analysis. This third postulate indicates that the costs and benefits of the project must be added, regardless of who wins and who loses. In other words, "a dollar is a dollar, regardless of who pays and who receives it".  Positive economic net present value should be a minimum standard. Other factors might be considered in order to select projects that pass this minimum criterion.  Success lies in political commitment and in the incentive for the bureaucracy to make high- quality appraisals of project ideas. 6  It is important to make a professional economic cost–benefit analysis (CBA), or alternatively a cost-effectiveness analysis (CEA), at the start. Such an analysis will quickly demonstrate its value through the improved quality of the projects selected for implementation.  The reform process should start with the building up of the skills of government staff through basic training and on-the-job experience.  Standard project approval criteria should result from project appraisal: Economic Net Present Value (ENPV) or Economic Internal Rate of Return (EIRR).  Projects that provide the largest gains to society should be financed first. However, establishing an effective PIM system is a process that takes time, since officials need to develop skills through training and practical experience. In this context, the investment appraisal system will usually only be fully implemented once policy makers see the benefits of the application of modern investment appraisal techniques:  Project appraisal should be based on quantitative measures. o CBA is based on economic theory and it has a longer academic pedigree than a multi- criterion analysis (MCA).  A project involves combining resources that are carefully defined and programmed over time (costs) in order to bring about an improvement in the well-being of society (benefits).  The aim of financial and economic analysis is to determine and qualify the costs and benefits of projects in order to facilitate certain decisions that have to be made following the project cycle. o However, CBA cannot always quantify all relevant costs and benefits; at such times, MCA can incorporate any dimension deemed important and bring together quantitative as well as qualitative factors. Its weakness is related to its potential manipulation by political authorities (e.g. by weights); to avoid this, it is key to figure out how to decide which factors are important and justifiable. 2.1 Project prioritization overview The prioritization framework involves the application of sequential filters. The first filter provides project profiling; ergo, the qualitative validation and classification of project proposals to determine investment profiles eligible to be part of the Executive Development Plan or the National Development Strategy (or similar document). It means to align project profiles to strategic priorities and asses their compatibility with national strategic priorities. The second filter involves the quantitative validation and classification of project proposals as eligible to receive funds for the conducting of pre-feasibility studies (PFS) and, after that, to receive funds for the conducting of feasibility studies (FS). This second filter provides a rigorous quantitative assessment of demand, as well as a technical analysis, legal and institutional analysis, environmental manpower analysis, financial analysis, economic analysis and risk analysis of the project. Once feasibility has been determined, the third filter provides a qualitative analysis to enable the selection of infrastructure investments between a portfolio of “good prepared projects”. At this stage it is important to mention that project budget allocation for investment projects is usually a political decision but in every case should be a well-informed one. 7 These filters, taken together, form a prioritization framework (PF) that contributes to the operationalization of the public investment plan. The following figure shows a schematic outline of the framework. Figure N°2 – Schematic Overview of the Prioritization Framework Profiling Appraisal Selection • Strategic Priorities • General & Sector • Ranking • Assess specific criteria • Selects and compatibility to • Evaluates and sequences projects national strategic appraises individual to be financed priorities projects against demand, technical, legal and institutional, environmental manpower, financial, economic and risk criteria Source: World Bank (2013). MPI Workshop: Session 1 Methodology for Selecting and Prioritizing Critical Infrastructure Investments. The PF provides a channel through which to implement the EDP, improving the efficiency of public investment projects in order to deliver a set of complete and profitable projects, as shown in Figure N°3. Figure N°3 – Improving Public Investment through Project Prioritization Framework Multi sectorial, SOE, 1. Profiling Local Government Pipeline 2. Appraisal 3. Selection Selected projects, “Wish list” ready to be (project profiles) implemented Source: World Bank (2013). MPI Workshop: Session 1 Methodology for Selecting and Prioritizing Critical Infrastructure Investments. In this context, the PF can be presented as a sequential steps funnel, like the one shown in Figure N°4. 8 Figure N°4 – Sequential Steps in Prioritization Framework Strategic Priorities General & Sector Specific criteria Project Selection Infrastructure Project Pipeline Source: World Bank (2013). MPI Workshop: Session 1 Methodology for Selecting and Prioritizing Critical Infrastructure Investments. Also, this process is totally consistent with the principles of an integrated approach, based on international best practices (shown in Figure N°5). Figure N°5 – Integrated Approach to Project Prioritization Strategic Conforms to the strategic priorities of the country Evidence driven Based on quality data and rigorous analytical tools Transparent Aligned Open, clear, Alignment of all auditable stakeholders processes Source: World Bank (2013). MPI Workshop: Session 1 Methodology for Selecting and Prioritizing Critical Infrastructure Investments. 2.2 Project selection and inclusion in the budget Project selection involves the approval of a project for financing and its eventual inclusion in the annual budget. Typically, approval takes place 2–3 times a year when a group of projects that have passed through the appraisal process are prioritized and considered for financing against the available investment resource envelope. Projects that are to be funded often come to the selection stage with the source of funding already identified. In this case, the role of the selection process is to endorse (or reject) the project appraisal and confirm that the project remains a priority for funding. The project 9 selection process is typically overseen by a high-level committee, often chaired by the Minister of Finance. Following approval of the project, the financing arrangements are finalized and the project included in the budget proposal. For international resource-funded projects, this involves negotiating a financing agreement; for domestically financed projects, it requires earmarking funding for the full investment cost and over the lifetime of the project. Inclusion in the next annual budget can then follow. While the budget approves funding for a single year, the initiation of an investment project also implies a multi-year funding commitment. For this reason, it is considered good practice for the annual budget to include an annex providing more detailed information on the capital investment projects being funded, including their total estimated cost and phasing of expenditure (Box 2). Box 2: Capital investment project annex to the budget The key information that would need to be maintained in the capital investment project annex to the annual budget comprises:  A brief narrative summarizing the objectives of the project, the investment to be undertaken, the location of the proposed investment, and the body responsible for implementation of projects.  The total estimated cost of the project (as approved by the Public Investment Screening and Selection Committee).  The sources of financing for the project,  A table showing actual expenditures incurred in previous years, a revised estimate of expenditure for the current year, budgeted expenditure for the coming fiscal year, and forecasted expenditure for a further two years, and the balance of expenditure required to complete the project. It can be useful for this table to show a breakdown of expenditure by funding source.  Estimated annual recurrent costs arising on completion of the project. In the context of the project cycle, this is shown in Figure N°6. Once again, this procedure is totally consistent with the analytical structure presented before. Figure N°6(a) – Integration of Strategic Planning, Budgeting and Project Appraisal Criteria 1 (*) Strategic Planning Criteria 2 Public Investment Criteria 3 Framework Budgeting Criteria 4 Criteria 5 Criteria 6 Project Appraisal Criteria 7 and Execution Criteria 8 Source: CRI Int. (2015). Proposal for Strengthening the Public Investment Management System in Vietnam. (*) Note: Criteria should be defined according to the specific context of Iraq. 10 Figure N°6 (b) – Integration of Strategic Planning, Budgeting and Project Appraisal Investment Cost and Schedule Recurrent Cost and Schedule Sources of Funding Project Funding and Schedule for Section each Source Project Title Land availability Project Identification Supplementary Scope and Objectives Section Infrastructure availability Budgeting Key Performance Indicators Cost of Program Strategic Planning Program Cost and Title of Investment Fiscal Effectiveness Project Revenues Program Section Tax Collections on Program Identification Project Inputs and Scope and Objectives Section Outputs Share of the Project Targets (KPIs) Cost in Program Cost Results Section Net Fiscal Qualitative Effectiveness Conformity of the Project with the Program Section Quantitative Construction period Preliminary Analysis of Environmental Impacts Operation period Preliminary Analysis of Social Impacts Socio-Economic Effectiveness Appraisal and Execution Section Quantitative Demand Indicators Macroeconomic Data Stakeholders and their List of sub projects reputation Project Management Scheme Proposed management Roles and responsibilities scheme Hierarchy (reporting structure) Source: CRI Int. (2015). Proposal for Strengthening the Public Investment Management System in Vietnam. 2.3 Objectives of prioritization methodology In the context outlined below, in light of the mission to enhance the quality of investment decision-making, this Technical Note presents an evaluation criterion for public investment project selection and evaluation, to ensure relevance, efficiency, effectiveness and sustainability of public policy in developing countries. Furthermore, in simplified terms, it develops different alternatives to conduct the project selection process, taking into consideration the difficulties of establishing a complete efficiency approach framework for 11 project preparation and project evaluation in the short term. This tool is based on international best practices. 3. ESTABLISHING PROJECT PROFILES A complete model for project prioritization shall be guided for the following principles: 3.1 Project lifecycle In this context, the project lifecycle is the process by which an idea is transformed into a concrete solution through the analysis of alternatives and the choice of the most profitable alternative from the economic point of view. Every project has certain phases in its development and implementation. These phases are very useful in planning a project, as they provide a framework for budgeting, resource allocation, the schedule of project milestones in implementation and the establishment of a monitoring system. The purpose is to provide a basis for organizing the project in such a way as to establish resource requirements, and to set up the management system that will finally guide project activities. Although the exact division of a project’s life into its different phases is somewhat sector specific, the project lifecycle phases may be broadly placed in different steps. Figure N°7 shows the project development lifecycle. Figure N°7 – The Project Development Cycle (Planning Phase and Project Cycle) Monitoring & Planning Ex Ante Evaluation Ex Post Evaluation Follow Up Planning and • Idea and profile • Operation and Identification Investment Phase execution Phase • Pre feasibility • Feasibility • Public Investment • Detailed design Needs • Implementation Operation and Pre Investment • National Ex Post Phase Development Plan Evaluation Phase Approval Source: based on international best practices. The last figure is completely consistent with Figure N°1, and incorporates the planning phase. As the project moves through its lifecycle, the focus of managerial activities shifts from planning to operating and controlling activities. It should be emphasized that these phases only represent one natural order among several in which projects may be planned and carried out. Also, none of these phases becomes truly final until the project approaches its termination stage. The Planning and Identification Project Phase is the first phase of the cycle and is concerned with the identification of potential public-sector projects. The purpose of having such a phase is to establish the basic desirability of a project and to identify high-priority projects that fall within the responsibility of the public sector. Projects are a valuable tool for directing investments into the priority sectors of an economy. 12 3.2 Planning and identification phase The process of project identification is complex. Projects are brought forward one at a time and are generally identified with their sponsors rather than as part of an economic strategy. Over the years, however, many countries have developed their own planning capability and the process of project identification has become more systematic. In most of the countries, this process is a reference to the National Development Plan or National Development Strategy. In these documents, the emphasis is put on “investment efficacy” or on spending on the right public assets. The strategic fit of projects therefore ensures the strategic alignment of investment projects with national and sector strategies. In a strategic planning exercise, the emphasis is on “investment efficacy” or on spending on the right public assets. Spending should promote the achievement of strategic priorities, and resources should be allocated only to those areas that are best aligned with the government’s objectives. In this sense, the strategic planning exercises, performed at different levels, are top- down processes that produce key deliverables, a buy-in and a given consensus on the NDP, sector development plans and others documents. The strategic fit of projects therefore ensures the strategic alignment of investment projects with national and sector strategies. The specific objectives of each investment project should consequently be designed in such a way that they support the overall national development agenda. At this stage it is necessary to consider the demands of different interest groups, community, public, private and other agencies. It also includes the diagnosis of the current situation, in order to verify the existence of a problem, need or opportunity, through the collection of information and data. In order to safeguard the strategic fit of projects, all public investment initiatives in a country such as Iraq shall be designed according to the log-frame approach (LFA), so that the strategic objectives of implementing each project are identified early on (i.e. the efficacy question) and are also measured applying suitable key performance indicators (KPIs). The strategic planning exercises and economic planning are both necessary and complementary, and they should not be disconnected from one another. If these two do not overlap and match, there will be severe inconsistencies in public policy priorities and corresponding investment decisions for the future of the nation. One way of matching the two planning exercises is for all investment projects to submit their log-frame matrix (LFM) at the very beginning of the project cycle; that is, at the project profile stage. 3.3 Pre-investment phase This analysis introduces the project into the pre-investment phase. As a result of this stage, the sponsoring agency should be able to: • Discard infeasible alternatives. • Select a feasible alternative and possibly advance to the next stage. • Wait or postpone applying the solution to the problem while the authority makes a decision. 13 4. USING PROJECT APPRAISAL TO STRENGTHEN THE SELECTION CRITERIA Evaluation tools are essential for making decisions related to the selection of projects to ensure the highest return. Comparing the total costs (investment and operation) of a project with its benefits allows a decision to be made over whether the project will make a true contribution to the wealth of the country. Thus, project evaluation allows: 1) The identification of those criteria for investment policies that maximize social welfare. 2) The termination of "bad projects" and the promotion of those that are "good". 3) The definition of whether the public or private sector should implement the project. 4) An estimation of the fiscal impact of the project. 5) Agreements on desirable cost recovery to be established. 6) An assessment of their impact on the environment and on regional development and poverty, among others. The utilitarian approach and applied welfare economics provide an analysis framework to estimate the goodness of public policies in terms of social welfare and thus to answer the above questions. To estimate the contribution of projects, it is necessary to identify, measure and assess their costs and benefits. The identification of costs and benefits entails qualitatively determining the positive and negative impacts generated by the project. The assessment of benefits and costs entails transforming financial cash flows into economic resource flows, using economic prices (efficiency or shadow prices) of goods and services produced and resources used. Of course, there will be some costs and benefits that can be identified but are unlikely to be quantified and valued. However, it shall be the duty of the evaluator to conduct the process rigorously, to identify all the effects and impacts of projects and to reasonably quantify and value as many as possible of them. In the context of applied welfare economics, using the Efficiency Approach (EA) and the tools related to CBA and CEA is the preferred approach to prioritizing and sorting multiple public expenditure proposals. However, many developing countries do not have an ongoing system for appraising the economic efficiency of the proposed investment projects. The situation is further complicated by the fact that in many cases investment proposals arise from a condition of limited information regarding economic costs and benefits. The EA is the only path that can lead to sustainable long-term economic growth; also, it creates a solid platform for the successful introduction of public–private partnerships (PPP) to a country, and assists in the identification of those projects that can be undertaken by the private sector. However, the application of CBA and CEA to insure economic efficiency (or “value for money”) takes time, resources and effort to establish and disseminate among stakeholders. It is important to note that core evaluation criteria should be applied to all public investment programs and projects in their evaluation during implementation, at completion or during operation, regardless of the size of their capital investment or classifications. However, sub- criteria and indicators may be selected as appropriate so long as they are plausible and measurable. 14 4.1 What are we appraising? Ex-ante project evaluation is a prerequisite to making sound investment decisions. To determine whether the benefits are higher than the costs, is recommended that CBA or CEA be used. Benefits and costs can be quantified and measured by assigning adequate measures and units to benefits, after which, ideally, they are given a monetary value. Project appraisal activities may be outsourced, depending upon the capacity resident in any given line ministry or public independent unit. In any case, these institutions must provide for project planning and studies within their current budget baselines, including, as necessary, funding for outsourced capital project appraisals. This analysis introduces the project into the pre-investment phase. As a result of this stage, the analyst should be able to: • Discard infeasible alternatives. • Select a feasible alternative and possibly advance to the next stage. • Wait or postpone applying a solution to the problem while the authority makes a decision. The project format is a kind of analytical tool in its own right that facilitates the task of planning for economic growth and development in the country or region. The main advantage of casting investment decisions in a project format is that it enables the planner or the analyst to establish a framework for analyzing information in a systematic way. Rigorous project identification and selection systems act as a screening mechanism to prevent inappropriate and inefficient projects from getting into the project cycle, gaining political support and momentum that make them difficult to stop at later stages. An integrated project management system allows the investment decision process to be slowed down by introducing gradualism via a mandatory project lifecycle. This concept is clearly represented in the PIM diagrams Pre-investment, Investment and Operation Phase of Projects. 4.2 Appropriate evaluation criteria and indicators This proposal is based on an integrated project selection analysis that is useful for performance measurement in terms of the development performance of public investment programs and projects, through the linking between project planning and budgetary allocation. In this context, the most used criteria for development evaluation are: relevance, efficiency, effectiveness and sustainability. Relevance. Refers to the assessment of programs and projects to measure design and time of evaluation in relation to the consistency of project and program objectives against the beneficiaries’ requirements, country needs, and global priorities. In other words, it is a measure of the consistency of project objectives, associated with the needs of project beneficiaries and the relevance of the project to the policies and priorities of the sector and its alignment with the National Development Plan or National Development Strategy. Short-term assumptions performance. Once a project has been implemented (i.e. its construction phase is finished), the results are revised and cost deviations are analyzed on the assumption that the benefits are achieved (the focus is on project management indicators such as schedule, time of construction, overall construction costs, quality and technical specifications). Changes in the expected economic criteria are explained according to higher investment costs, timing, size, etc. This short-term ex-post evaluation is focused solely on project costs, schedule and checking the assumptions made during the project pre-investment stage. 15 Efficiency is concerned with how in economic terms the project or program resources are converted into results (outcomes, impacts). It assesses process efficiency (including timeliness), investment efficiency (returns on investment or cost effectiveness) and social and environmental efficiency, among others. In other words, this indicator measures the resource use associated with the generation of products or services to be provided by the project, their execution timing and investment costs. It is ultimately a measure of the efficiency of the entire project, relating the achievement of its aims and objectives with the use of resources linked to this achievement. Effectiveness assesses the extent of the likely or observed achievement of intended objectives/outcomes or results as specified in the project/program documents. In other words, it is a measure of the achievement of the aims and objectives of the project, associated with the operation and use of the results. Sustainability examines all risks to the continuity of the program and project outputs and outcomes that could have long-term development impacts. It assesses sustainability in terms of finance, management capacity, environment and society/community. Therefore, it is associated with the maintenance of the project's capacity to provide goods and services as defined in its original conceptualization and according to the initial standard that was set; also, it is associated with the technical and management capacities of the implementing agency to properly maintain the operation of the project in terms of the goods and services defined in the initial design (measured in terms of service standard). It is very important to focus the analysis on financial sustainability so as to ensure the existence of enough resources (required budgetary support) to continue generating the flow of outputs and outcomes from the programs and projects. These five core criteria are consistent with the evaluation criteria and standards of the Development Assistance Committee for the Organization for Economic Co-operation and Development (DAC – OECD). In second place, the indicators criteria linked with any singular criterion should be defined, as shown in Table N°1. Table N°1 – Core Evaluation Criteria and Indicators Core Criteria Sub-Criteria / Indicators Relevance, design  Project rationale and justification. and quality  Consistency with national-, sector- and governorate-level development goals/strategies/territorial/sector master plans.  Urgency and priority (e.g. disaster relief, national security…).  Compliance with relevant technical standards and qualifications.  Funding plan.  Realism about implementation arrangements.  Alignment of project design and estimated total budget as stated in the project approval decision.  Plans for developing the necessary capacity for project management and operation.  Use of a report form (RF), database and reporting systems for monitoring progress/achievement. 16 Core Criteria Sub-Criteria / Indicators Short-term Depending on the type of information they provide, indicators are distinguished assumptions among: performance  User satisfaction and quality: measure compliance with specifications and evaluate attributes against standards, external references or the satisfaction of beneficiaries (users). For example, reliability of hydro-meteorological forecasts, percentage of patients who received attention and were satisfied with the care received, percentage of participants who demonstrate an understanding of techniques.  Economics: Reflect ability to attract and mobilize financial resources. For example, percentage of budget executed per year, percentage of credit recovery, etc.  Indicators of cost: Related costs (budgeted / actual). For example, percentage deviation of the cost of equipment.  Indicators of time (schedule): Measure the level of compliance with schedule/timing. For example, number of days of delay in given activity, time of execution planned/real.  Indicators of quantity: Measure the level of compliance with the planned amount of goods or services to be generated by the program/project. For example, percentage of actual trainees compared to the percentage that was intended. Efficiency in the use  Associate costs or resources with products. For example, average annual cost of of resources hydro-meteorological data, total cost per km of road maintained, number of patients seen per physician.  Process efficiency: extent of mobilization of funding sources compared to the plan as per project approval decision, investment to output ratio, cash-flow scheme against total funding estimation (against those stated at approval decision), elapsed time/time overrun, disbursement rate against cash-flow projection, funds use rate, cost overrun (actual/approved project cost).  Investment efficiency: adjustment of project scale/scope compared to the approved design (if expected benefits are relevant to master plans and/or other kinds of plans, likely or observed rate of returns on total investment, production and supply of goods and services, in financial as well as economic terms or cost effectiveness [where benefits are not quantifiable, e.g. comparison of unit cost within the sector and governorate or cost of alternatives]).  Social and environment efficiency: likely or observed effects in relation to gender, environment, resettlement, employment, and household income. Effectiveness in  Measure the level of achievement of objectives. For example, percentage achieving outcomes changes in the number of users who consult hydrological data by year and type of information requested, percentage of road network that sustains their designed service level (or is close to it).  Extent of achievement of the investment objective/s or intended intermediate outcomes (e.g. cost reduction, disease control, improved child health). Sustainability – risks,  Nature and extent of risks, resilience, and their mitigation measures. identification and mitigation measures  Provision for financing of operation and maintenance, O&M (revenue or state budget)?  Extent of capacity development (staff training, equipment, spare parts, etc.).  Likelihood of continuity of outputs and outcomes? Source: World Bank (2014). TA 7725–VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011–2015 (Vietnam). 17 4.3 Evaluation criteria formulation As was mentioned before, public investment programs and projects need to be addressed and evaluated before being chosen for investment decision-making. The first step must always be the definition problem (which needs to be resolved) and the subsequent project proposal solution, in accordance with potential beneficiaries and stakeholders. The second step is the options analysis through a preliminary review, the purpose of which is to confirm the consistency of the project proposal solution with the National Development Plan (Iraq), sector and sub-sector strategies and the program that includes the project. After passing the consistency test, pre-feasibility studies are being carried out, using the EA. This means the module approach: demand, technical, human resources, legal and environmental analysis, as well as financial, economic, stakeholders and risk analysis. Detailed feasibility analysis (the same as pre-feasibility analysis but with more detail and depth) can only be implemented after the pre-feasibility analyses if the investment decision is to be fully considered. Figure N°8 below shows integration between the project selection criteria and the project cycle. 18 Figure N°8 – Project Selection Criteria and the Project Cycle National Development Plan or National Development Strategy Design quality Implementation Overall project Relevance, design and and readiness performance implementation and quality operation performance ST Quality performance ST Economics performance ST assumptions ST Costs performance performance ST Schedule performance ST Quantity performance Efficiency in the use of Continued viability Continuity in economic resources Viability viability Baseline and Satisfactory Continuity in achieving Effectiveness in Target outputs achievement of intended outputs and achieving outcomes and outcomes outputs/outcomes outcomes Sustainability – risks Risk Measures to Risks mitigated identification and identification ensure sustain. of Likelihood of achieving mitigation measures and mitigation outputs/outcomes long-term impacts Project profiles Initial project registration Pre-Feasibility Feasibility Investment End of Decision End of project evaluation 1 2 3 Public Investment Management System (PIMS) National Operation and Ex-Post Planning Phase Investment Phase Investment Phase| Evaluation Phase Source: International best practices 19 4.4 Establishing a rating scale After the definition of indicators, it is necessary to select scores related to each criterion. This score should be based on evidence and justification, and an evaluation result band will be assigned based on this. Following the example included in World Bank (2014). TA 7725–VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011–2015 (Vietnam), it may be assigned one of four different scores: Highly Satisfactory Score (HSS) if the evaluation score is higher than 80%; Satisfactory Score (SS) if the evaluation score is higher than 60% and lower than or equal to 80%; Partly Satisfactory Score (PSS) if the evaluation score is between not less than 50% and not more than 60%, and Unsatisfactory Score (USS) if the evaluation score is less than 50%. See Table N°2. Table N°2 – Score Definition Partly or Less Highly Satisfactory Satisfactory Unsatisfactory Satisfactory The overall average The overall average The overall average The overall average rating score is greater rating score is greater rating score is not less rating score is less than than (>) 80% for than (>) 60% and less than (≥) 50% and not (<) 50% achievement or achievement or than or equal to (≤) 80% more than (≤) 60% compliance, as compliance, as achievement or achievement or appropriate. appropriate. compliance, as compliance, as appropriate. appropriate. Source: World Bank (2014). TA 7725-VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011–2015 (Vietnam). The scores are given by evaluators based on how projects met those criteria, providing evidence and justification (and means of verification). Appraised or evaluated projects are then ranked by their average score. 4.5 Establishing a project weighting scheme After the score definition, it is necessary to define a project weighting scheme. Following the example included in World Bank (2014). TA 7725–VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011–2015 (Vietnam), the scoring scheme is provided, shown here in Table N°3. The maximum score is assigned to each core criteria with varying weights. In this case, the highest weight (40%) is assigned to the criterion of efficiency in resource use for all types of evaluation to emphasize its importance in public investment decision-making. Relevance, Effectiveness and Sustainability are assigned variable weights depending on their importance in relation to the stage in the project cycle and to types of evaluation. 20 Table N°3 – Scoring Scheme and Distribution of Weights for Different Types of Evaluation Ex-ante Midterm Terminal Ex-post Core Criteria Appraisal Evaluation Evaluation Evaluation Relevance 20 25 10 10 Short-term assumptions 20 0 0 0 performance Efficiency 30 40 40 40 Effectiveness 10 15 25 25 Sustainability 20 20 25 25 Total 100 100 100 100 Source: World Bank (2014). TA 7725–VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011–2015 (Vietnam). Finally, Table N°4 provides an illustration as an example of each type of project investment evaluation. Table N°4 – Evaluation Focus Core Criterion/ Types of Ex-ante Mid-term Terminal Ex-post Evaluation Design Quality Change in Implementation Overall project market, design performance. implementation or scope. and operation performance. Implementation Any change in arrangements implementation arrangements Relevance and performance so far. Readiness: actual as Progress in Extent of Continued well as planned compliance with compliance with compliance with compliance with various laws and various laws and various laws and relevant laws, approval terms approval terms approval terms regulations and and conditions. and conditions. and conditions. technical standards. Quality: evaluate attributes against standards, external references or satisfaction of the Short-term beneficiaries (users). assumptions Economics: Reflect performance ability to attract and mobilize financial resources. Cost: Related costs (budgeted/actual). 21 Indicators of time (schedule): Measure the level of compliance with schedule/timing. Indicators of quantity: Measures the level of compliance with the planned amount of goods or services to be generated by the project/program. Indicators of quality: Measures compliance with the specifications. For example, percentage of participants who demonstrate an understanding of the techniques. Time & cost Time overrun; Overall Continuity in efficiency; efficiency and financial and/or cost overrun; financial and/or economic Efficiency financial and/or financial and/or economic viability. economic viability. economic viability at viability. completion. Baseline and target Progress toward Satisfactory Continuity in outputs and achieving achievement of achieving outcomes. intended outputs outputs and intended outputs Effectiveness and outcomes. outcomes and outcomes. compared to the target/baseline. Risk identification Progress in Measures Risks to and proposed addressing risks undertaken to sustainability mitigation measures. to sustainability: ensure mitigated sustainability of (social, social, project outputs environment, environment, and outcomes by human human resource Sustainability mitigating the resources, O&M development, identified risks. finance). and financing of O&M cost. Adequate Likelihood of technical skills. achieving long- term Financing of development O&M. impact. Source: World Bank (2014). TA 7725–VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011–2015 (Vietnam). 22 5. PROJECT SELECTION Following WB (2013) report1, this section shows an example of tool application to ex-ante project evaluation. The purpose of appraisal is to enable decision-makers to decide whether the activity represents an appropriate use of resources. Table N°5 shows a framework of application of ex-ante evaluation criteria, sub-criteria, indicators and scoring scheme. 1 Source: World Bank (2013). MPI Workshop: Session 1 Methodology for Selection and Prioritizing Critical Infrastructure Investments. 23 5.1 Check list for ex-ante evaluation Table N°5 – Check List for the Application of Ex-ante Evaluation (Criteria, Sub-criteria, Indicators and Scoring) Project details: to be completed. Key Criteria & Remark Sub-criteria Key Indicators Scoring Basis/Justification Score (Further Justification for Evaluation for the Score) Relevance, design, strategic consistency & readiness (maximum 30 points) Statement of project objectives Objectives and outcomes are clearly defined, linked and expected development to provincial, sector and national socioeconomic .../2 outcomes and outputs in relation development plans, and targets realistically set. to the needs of the locality/sector. 1.1. Assess the rationale A report form stating inputs, Outcomes and outputs are clearly defined and and justification for the outputs, outcomes, and impacts, measurable, and data sources and responsibility for .../2 project/program (6 and monitoring plan. monitoring identified. points). To what extent the design of the project complies Compliance with relevant laws with major laws, regulations and technical .../2 and regulations. standards (e.g. investment, budget, procurement, and environment). 1.2. Assess the realism of Quality of the approach and content of the Project implementation the implementation plan implementation plan (including logistics, budget .../4 arrangements/readiness. (4 points). and staffing). 1.3. Assess the Stakeholder and beneficiary Coverage and quality of the consultations with .../3 appropriateness of choice. consultations. stakeholders and beneficiaries. of location, land Justification for the location. .../1 24 acquisition, and technical Location, land acquisition and option (9 points). Coverage and appropriateness of the LAR plan for resettlement (LAR) plan, where .../2 the project. relevant. Choice of technology, basic How realistic are the technical justifications given engineering design, and analysis in the project document and the justification for the ../3 of technical feasibility. choice of technology? How realistic is the estimation of the total Total investment requirement and investment requirement and financing plan (sources .../3 1.4. Assess how realistic financing plan. of funding)? and feasible the financing plan is. Normal mark for state capital funding .../2 Funding sources (own, state, Extra mark for governorate’s off-budget resources (7 points). borrowing as relevant). and PPP planned (2 as maximum on top of normal .../2 mark). 1.5. Any other governorate or sector-specific sub- To be defined case by case. To be defined case by case. .../4 criteria that may be added (4 points). Average Score for relevance. .../30 Short-term assumptions performance (maximum 20 points) The range of variation of forecast was not higher Reliability of forecasts. .../2 2.1. Assess the user than 20%. satisfaction and quality Beneficiaries who received indicator (2 points). More than 80% of beneficiaries were satisfied with attention and were satisfied with .../2 the goods and services provided by the project. the goods and services received. Percentage of budget executed per 2.2. Assess the economic The percentage of the budget executed per year was year; percentage of credit .../4 indicator (4 points). above 80%. recovery. 25 2.3. Assess the costs The percentage of deviation from the total cost was Percentage deviation of total cost. .../4 indicator (4 points). below 80%. 2.4. Assess the timing Percentage deviation of total The number of days of delay in given activity was .../4 indicator (4 points). timing. below 25%. Percentage of beneficiaries that 2.5. Assess the quantity receive the goods and services, The percentage of total beneficiaries that received .../4 indicator (4 points). compared to the percentage that the goods and services per year was above 80%. was intended. Average score for short-term assumptions performance. Efficiency in resource use (maximum 30 points) Schedule for project implementation, including 3.1. Assess the plan to Project implementation schedule. .../3 construction, where appropriate. ensure process efficiency (timelines) (5 points). Procurement plan and fund Whether approach, timelines and milestones for .../2 utilization schedule. procurement and fund utilization are realistic. Financial viability based on estimated returns on Financial internal rate of return investment (the higher the EIRR the better, the (to be viable it should be higher minimum should be not less than the cost of than the weighted average cost of 3.2. Assess financial borrowing). capital). .../6 viability (6 points). Or, where it is not possible to estimate EIRR, Or, project unit cost comparisons comparison of unit cost with similar projects in the for non-revenue generating governorate or nearby locations and construction projects. standards. Identification and quantification How realistically has the economic cost and benefit .../4 of economic costs and benefits. been identified and estimated? 3.3. Assess economic viability (8 points). Economic viability based on the estimates of Estimation of economic internal EIRR, higher then better but minimum should not .../4 rate of return (IRR) (higher than be less than the opportunity cost of capital. 26 the economic opportunity cost of Or, where it is not possible to measure EIRR (e.g. capital). in small-scale projects, social sector projects), cost Or, cost effectiveness (compared effectiveness analysis may be used or unit cost with cost of alternatives and unit comparison within the governorate or within the costs in the sector) in the case of sector, or cost of alternative may be used. projects/programs where economic benefits are not quantifiable). How realistic are the plans for ensuring social efficiency and safeguards? Total jobs likely to be created. Of which, jobs created for women and girls, where applicable (gender: % of women/girls among the Positive impact on employment total beneficiaries of the program/project as creation and gender equity – jobs .../4 determined by a social impact assessment). for girls and women. Other benefits to women and girls (e.g. health, education, anti-trafficking, where relevant). 3.4. Assess plan for social Or, where the above is not relevant due to the efficiency (10 points). specific nature of the project, assessment may be based on the estimate for total jobs creation. Resettlement of project-affected Whether all the number of affected households are households and likely going to be resettled and their welfare (income, .../2 improvement in their welfare health, education) improved. (income, health, education). Improvement in household Increase in household income in the project- .../2 income. influenced area(s). National defence and social Realistic plan for ensuring national defence and …/2 security, as appropriate. social security. 3.5 Environmental Plan for addressing environmental How realistic is the plan to: (a) Address positive or …/4 efficiency (4 points). issues (if relevant). negative effect in local environment as identified in 27 environmental impact analysis? (b) Reduce % CO2 emissions (where applicable) (c) Comply with green standards and others (where applicable)? 3.6. Any other governorate- or sector- To be defined case by case. To be defined case by case. .../3 specific sub-criteria that may be added (3 points). Average score for efficiency .../30 Effectiveness (achieving outcomes) (maximum 10 points) Targets for achieving project How realistic are targets and timelines for 4.1. Assess how outputs and outcomes (consistent .../4 achieving project outputs and outcomes? realistically targets and with that stated in the RF). indicators for outputs and Baseline target (beginning) and outcomes are set (8 points). How well are the baselines and targets for outputs end-lines (final) for outputs and .../4 and outcomes defined? outcomes. 4.2. Any other governorate- or sector- To be defined case by case. To be defined case by case. .../2 specific sub-criteria that may be added (2 points). Average score for Effectiveness .../10 Sustainability (maximum 20 points) 5.1. Assess the plan for the Assess nature and extent of risks to continuity of sustainability of outputs Risk identification. outputs and outcomes, and their mitigation .../10 and outcomes of the measures. 28 proposed project (16 How realistic is the financing plan for O&M? What points). Financing of O&M. are the main sources of funding O&M cost? .../3 How plausible are the plans for staff training, Capacity development plan. .../3 equipment, spare parts, as applicable to the project? 4.2. Any other governorate-specific sub- To be defined case by case. To be defined case by case. .../4 criteria that may be added (4 points). Average score for sustainability .../20 Source: World Bank (2014). TA 7725–VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011–2015. (*) Note on scores: the notation “.../x” indicates the assigned value at the sub-indicator of the total amount of “x” points. For example, indicator 1.1. “Assess the Rationale and justification for the project/ program” has a total of 6 points with 3 sub-indicators of at most 2 points each; then, the evaluator can assign 0, 1 or 2 points to sub-indicator “Statement of project objectives and expected development outcomes and outputs in relation to the need of the locality/sector”. In this example, the notation 1/2 means that the evaluator assigned 1 point out of 2 for this sub-indicator. 29 5.2 Scoring for ex-ante evaluation After the definition on criteria, sub-criteria, indicators and scoring, it is necessary to define the level aggregation of scores, as is shown in Table N°6. Table N°6 – Project-level Aggregation of the Ex-ante Evaluation Score Project details: to be completed. Ex-Ante Evaluation Full Score Average Score Score % Evaluation Criteria Result Band (A) Value (B)* [(B/A)*100] (HS, S, PS , US) (a) (b) (c) (d) (e) Relevance Short-term assumptions performance Efficiency Effectiveness Sustainability Project Average Project Result Band Source: World Bank (2014). TA 7725–VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011–2015 (Vietnam). 5.3 Ranking of selected projects After the project-level evaluation results aggregation, projects should be ranked in their specific level (for example, sector at provincial level) according to their respective score and result band, as is shown in Table N°7. Table N°7 – Ranking of Projects in their Specific Sector by their Percentage Average Score (and Corresponding Results Band at Governorate Level) Total Result Name and Total Band Name and Address Project State Average No Address of Project of Project Code Funding Score (HS, S, Investor Cost (USD) (USD) PS, US) Transport Sector: 1 2 3 Education Sector: 4 5 6 Health Sector: 7 8 9 30 Agriculture/Irrigation: 10 11 12 13 Water Supply: 15 16 Etc. Source: World Bank (2014). TA 7725–VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011–2015 (Vietnam). Proposed projects for state funding should be ranked by their evaluation score and result band, and the decision taken for their selection in accordance with the availability of budgetary resources. The higher the evaluation score, the better the anticipated or actual project performance (relevance, short-term assumption performance, efficiency, effectiveness and sustainability). 6. CONSOLIDATED APPLICATION ON PROJECT SELECTION 6.1 Score indicators Following the World Bank (2013). MPI Workshop: Session 1 Methodology for Selection and Prioritizing Critical Infrastructure Investments report, the developed matrix can be consolidated in two main indicators; sometimes this type of consolidation provides a more intuitive understanding of the tool, helping to introduce the concept into the public system. Table N°8 shows the aggregation of scores presented in Table N°6 under two main indicators with respective average weights. Table N°8 – Aggregation of Score Indicators Aggregated Indicator Evaluation Criteria Sub-indicator Weight Indicator Score Relevance indicator Relevance 1 = 20/20 50 Performance indicator Short-term 0,25 = 20/80 assumptions performance Efficiency 0,375 = 30/80 50 Effectiveness 0,125 = 10/80 Sustainability 0,25 = 20/80 Project Average 100 Source: MENA PIM Technical Practice, based on “World Bank (2013). MPI Workshop: Session 1 Methodology for Selecting and Prioritizing Critical Infrastructure Investments” and “World Bank (2014). TA 7725–VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011–2015 (Vietnam)”. 6.2 Tabulated results In this example, political authorities assume an equal weight for both relevance and performance indicators. However, this combination depends on the policy-maker criterion. Given the criteria weights and the value attributed to the project, Table N°9 presents tabulated results for a hypothetical example. 31 Table N°9 – Tabulated Results (Hypothetical Example) Aggregated Indicator Sub-indicator Evaluation Criteria Indicator Score Project A Score Weighted Relevance indicator Relevance 4x1 Relevance indicator is 2 = 0,50 x 4 Performance indicator Short-term 5 x 0,25 assumptions performance Performance Efficiency 4 x 0,375 indicator is 2,0625 = 0,50 x 4,125 Effectiveness 3 x 0,125 Sustainability 4 x 0,25 Source: MENA PIM Technical Practice, based on “World Bank (2013). MPI Workshop: Session 1 Methodology for Selecting and Prioritizing Critical Infrastructure Investments” and “World Bank (2014). TA 7725–VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011– 2015 (Vietnam)”. This methodology can be applied to different sectors, which will require their own specific criteria and benchmarks. Some criteria may be similar across different sectors, and different strategies can be applied to establish the weights to the criteria; in general, some judgment and discussion may be required to fine-tune them. This standardized framework then facilitates the analysis and comparison of project proposals. 6.3 Prioritization and selection After that, and following on from the report entitled “World Bank (2013). MPI Workshop: Session 1 Methodology for Selection and Prioritizing Critical Infrastructure Investments”, two possible methods for prioritization and selection of specific sectors can be applied: 1) Graphical representation through a populated matrix; and 2) A quantification of social–environmental and economic scores. However, neither of these methods addresses interdependencies or inter- relationships among projects. Method 1 is shown in Figures N°9, 10, 11 and 12 below. 32 Figure N°9 – Populated Matrix 5 4,5 4 The results of the evaluation done previously in 3,5 the table can be plotted to further illustrate Relevance indicator their positioning in terms of meeting the above 3 requirements. 2,5 2 Project A: (2.0, 2.0625) 1,5 1 0,5 0 1 1,5 2 2,5 3 3,5 4 4,5 5 Performance indicator Source: MENA PIM Technical Practice, based on “World Bank (2013). MPI Workshop: Session 1 Methodology for Selecting and Prioritizing Critical Infrastructure Investments” and “World Bank (2014). TA 7725–VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011– 2015 (Vietnam)”. The relative positioning of each project can be represented and thus compared against its relevance and performance indicators. See Figure N°10. Figure N°10 – Populated Matrix 5 4,5 4 Relevance indicator 3,5 3 2,5 2 1,5 1 1 1,5 2 2,5 3 3,5 4 4,5 5 Performance indicator Source: MENA PIM Technical Practice, based on “World Bank (2013). MPI Workshop: Session 1 Methodology for Selecting and Prioritizing Critical Infrastructure Investments” and “World Bank (2014). TA 7725–VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011– 2015 (Vietnam)”. 33 Figure N°11 – Populated Matrix 5 High relevance High relevance & indicator 4,5 performance indicators 4 Relevance indicator 3,5 3 2,5 Low relevance & 2 performance indicators High performance indcicator 1,5 1 1 1,5 2 2,5 3 3,5 4 4,5 5 Performance indicator Source: MENA PIM Technical Practice, based on “World Bank (2013). MPI Workshop: Session 1 Methodology for Selecting and Prioritizing Critical Infrastructure Investments” and “World Bank (2014). TA 7725–VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011– 2015 (Vietnam)”. Focus on the projects that have high relevance and performance impact (indicators). Further study can be done on how to finance shortlisted projects. See Figure N°12. Figure N°12 – Populated Matrix 5 4,5 4 Relevance indicator 3,5 3 2,5 2 1,5 1 1 1,5 2 2,5 3 3,5 4 4,5 5 Performance indicator Source: MENA PIM Technical Practice, based on “World Bank (2013). MPI Workshop: Session 1 Methodology for Selecting and Prioritizing Critical Infrastructure Investments” and “World Bank (2014) TA 7725–VIE: 34 Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011– 2015 (Vietnam)”. Another method of comparison could be done by ranking the results of the evaluation of all project proposals. In this case, different weights could be set to favor relevance or performance dimensions. In the example, relevance and performance dimensions have equal weight; in such cases, 5:5 ratio is used. Table N°10 shows the results. Table N°10 – Tabulated Results (Hypothetical Example) Aggregated indicator Sub-indicator Evaluation criteria Indicator score Project A score weighted Relevance indicator Relevance 4x1 Relevance indicator is 2 = 0,50 x 4 Performance indicator Short-term 5 x 0,25 assumptions performance Performance Efficiency 4 x 0,375 indicator is 2,0625 = 0,50 x 4,125 Effectiveness 3 x 0,125 Sustainability 4 x 0,25 Aggregated Project A Weighted Average indicator is 4,0625 Source: MENA PIM Technical Practice, based on “World Bank (2013). MPI Workshop: Session 1 Methodology for Selecting and Prioritizing Critical Infrastructure Investments” and “World Bank (2014). TA 7725–VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011– 2015 (Vietnam)”. 7. FINAL COMMENTS It is important to begin the process of improving the PIM system (PIMS) in Iraq as soon as possible. The new and growing needs of local populations put pressure on the public project portfolio and consequently on budget. The recommended solution is to start improving the processes of project preparation and evaluation immediately. The efficiency approach is the preferred framework to drive this process. However, it takes time and resources. Therefore, it is recommended that a gradual process be started, developing and progressively incorporating different tools into the PIMS, until in the coming years a complete system of preparation, evaluation and selection of projects is achieved. This paper presents some ideas and alternatives for starting the discussion related to project selection. Project selection and monitoring is a continuing function that uses systematic collection of data on specified indicators to provide management and the main stakeholders of an ongoing development intervention (i.e. project or program) with indications of the extent of progress in achieving objectives and of progress in the use of allocated funds. 35 8. REFERENCES  Belli, P., Anderson, J., Barnum, H., Dixon, J., Tan, J. (2001). Economic Analysis of Investment Operations: Analytical Tools and Practical Applications. The World Bank, Washington D.C.  CRI Int. (2015). Proposal for Strengthening the Public Investment Management System in Vietnam.  Harberger, A. (1996). The Economic Evaluation of Projects, Papers from a Curriculum Development Workshop. Economic Development Institute of the World Bank, The World Bank, Washington, D. C.  Jenkins, G., Harberger, A., Kuo, Ch. (2013). Cost–benefit Analysis for Investment Decisions, the Integrated Analysis of Investment Projects. Queen’s University, Kingston, Canada.  World Bank (2013). MPI Workshop: Session 1 Methodology for Selecting and Prioritizing Critical Infrastructure Investments.  World Bank (2014). TA 7725–VIE: Support for the Preparation and Implementation of the Results-Based Socioeconomic Development Plan, 2011–2015. Evaluation Criteria for Public Investment Projects/Programs. Technical Paper, V.5. TA Project Office, Hanoi. 36 9. ANNEX N°1 – Practical Worksheet V 1.0. User Manual 2016 Simplified Methodology for Public Investment Project Selection and Prioritization in Kurdistan Region The present section develop a short user manual for the “Prioritization Practical Worksheet V 1.0”, that is intended to work along with the “2015 SIMPLIFIED MANUAL FOR PUBLIC INVESTMENT PROJECT SELECTION AND PRIORITIZATION” methodology. This tool have been developed using Microsoft Excel 2010, and works with Microsoft Office 98 suit or newer version. This tool works with Visual Basic programming tools inside, so the user have to first allow Microsoft Excel to work with Macros. The first time that the user opens the workbook, a warning message will pop up from the application. The user has to click “Enable Content” in order to correctly use this tool. The steps to successfully perform a project prioritization are the followings: 1. Start a new Prioritization 2. Fill Table 1, Project Descriptors. 3. Fill Table 2, Project Scoring. 4. Press “Save Results” when the information of a project is done. 5. Press “Clear information” to add a new project. 6. Fill Table 1 and Table 2 Information for each new project. 7. Click “Finish” when done. The detail of every step is described as follow. Step One: Start a new Prioritization. When the “Prioritization Practical Worksheet V 1.0”, Excel Workbook is opened for the first time, the “Cover Page” worksheet will appear. 37 In order to start a new project prioritization, the user has to click in the “Start new Prioritization” button. In order to confirm, the application will ask the user: “Do you want to start a new Analysis?”, and if the user click “Yes”, the step is done. This button will clean all previous information that was charged in the workbook. Finally, the “New Project Scoring” worksheet will be selected. Step Two: Fill Table 1, Project Descriptors. In “New Project Scoring” worksheet, please fill the information of Table 1. Not all information has to be filled, but at least Project Sector and Project Code are mandatory. Project Sector has to be selected from a dropdown list in the respective cell. Step 3: Project Scoring, saving and clearing. This is the main step in this methodology. The user has to assign a score to every Key Criteria and Sub-criteria, and every Key Indicator. The score have to be an integer between zero and the max score indicated next to the score. For more information about the indicator, please refer to the main body of the methodology. 38 For each Key Indicator, the user should fill the “Remark” box, adding the justification of the assigned score. When all information is filled, and every score have been assigned, the user must press in order to save all the information in a new worksheet. All the information generated will be saved in the worksheet that will generate automatically, a sheet named “Project _Project Code”. For instance, if the saved project has the assigned code 300102030, the new worksheet will be called “Project 300102030”. When project information is ready and saved, the user can go back to the “New Project Scoring Worksheet” and press in order to clear all the previous information, and start adding new projects information. As an example, we will show some projects that a user might have filled for prioritization in Agriculture and Transport Sector. 1. Project Descriptors Project Name T-A Project Code 123 Project Sector Transport Sector Institution Transport Geographic Location North Total Project Cost (USD) $ 300.000,0 Total State Funding (USD) $ 250.000,0 1. Project Descriptors Project Name T-B Project Code 124 Project Sector Transport Sector Institution Transport Geographic Location North Total Project Cost (USD) $ 250.000,0 Total State Funding (USD) $ 200.000,0 39 1. Project Descriptors Project Name T-C Project Code 125 Project Sector Transport Sector Institution Transport Geographic Location North Total Project Cost (USD) $ 440.000,0 Total State Funding (USD) $ 440.000,0 1. Project Descriptors Project Name A-2 Project Code 223 Project Sector Agriculture and Irrigation Sector Institution Agriculture Geographic Location North Total Project Cost (USD) $ 120.000,0 Total State Funding (USD) $ 120.000,0 1. Project Descriptors Project Name A-1 Project Code 222 Project Sector Agriculture and Irrigation Sector Institution Agriculture Geographic Location North Total Project Cost (USD) $ 440.000,0 Total State Funding (USD) $ 440.000,0 1. Project Descriptors Project Name A-3 Project Code 225 Project Sector Agriculture and Irrigation Sector Institution Agriculture Geographic Location North Total Project Cost (USD) $ 120.000,0 Total State Funding (USD) $ 120.000,0 1. Project Descriptors Project Name A-4 Project Code 226 Project Sector Agriculture and Irrigation Sector Institution Agriculture Geographic Location North Total Project Cost (USD) $ 254.000,0 Total State Funding (USD) $ 220.000,0 When the user is done adding projects, he or must to press in order to deploy the final results. The final product will be a list of all the projects, ordered by Sector and SCORE. The score is calculated in the basis of the individual score for each Key Indicator, and the proportional weight of each Key Criteria Group- 40 There will also display a bubble chart showing the selected projects, representing both Performance and Relevance Indicator. The size of the bubble represents the Total Project Cost, and the label will show the Project Code. 70% 225 Performance Indicator 60% $ 120,000.00 226 50% 223 $ 254,000.00 123 $ 120,000.00 40% 222 $ 300,000.00 30% $ 440,000.00 124 125 20% $ 250,000.00 $ 440,000.00 10% 0% 0% 10% 20% 30% 40% 50% 60% Relevance Indicator 41