ICRR 12777 Report Number : ICRR12777 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 09/21/2007 PROJ ID : P037049 Appraisal Actual Project Name : Ar Pub.Inv.Strengthg US$M ): Project Costs (US$M): 25.00 n.a Country : Argentina Loan/ US$M ): Loan /Credit (US$M): 16.00 10.30 Sector Board : PS Cofinancing (US$M ): US$M): 9.00 n.a. Sector (s): Central government administration (100%) Theme (s): Public expenditure financial management and procurement (67% - P) Decentralization (33% - S) L/C Number : L3958 Board Approval Date : 11/21/1995 Partners involved : Closing Date : 06/30/2001 12/31/2006 Evaluator : Panel Reviewer : Group Manager : Group : Pierre M. De Raet Steven Benjamin Jaime Jaramillo IEGCR Webb 2. Project Objectives and Components: a. Objectives: The original objective was to enhance the effectiveness and efficiency of the Borrower's public expenditure management by: (a) improving its investment process; (b) enhancing its project analysis capabilities; and (c) building up its project and program evaluation capacity (Loan Agreement). In June 2000, the Board approved adding a second objective and two new components, without, however, changing the first objective or modifying the other components, that, in the meantime had been amended in July 1999. Adding a second objective was in response to the request of the de la Rua Government, which took office in December 1999, to complement its state modernization program by funding, under the project, technical assistance (TA) and training to design and implement: (i) a results-oriented management system; and (ii) a system to develop, publish, and monitor adherence to agency service standards . The amendment did not become effective, however, until April 20, 2001. The second objective read as follows: ..."and, more generally, to support the establishment within the Borrower's public administration of a results -oriented management approach and of a monitorable emphasis on providing service to the public" (Loan Agreement). b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 06/08/2000 c. Components (or Key Conditions in the case of DPLs, as appropriate): Original project description . There were 5 components : US$ 1.2 million; actual n .a.) Part A : Sectoral Investment Decision Making (estimated costs : US$1 1. Carrying out of a series of high -level seminars and workshops aimed at familiarizing decision -makers with best practices in the areas of public investment and public expenditure management and the concepts of the Public Investment Law (Ley del Sistema Nacional de Inversiones P úblicas); Part B : Improvement of Project Analysis and Evaluation Capacity of Agencies US$ 7.6 million; (estimated costs : US$7 actual n .a.) 1. Conducting capacity-strengthening courses on investment decision -making, project management, procurement and project cycle implementation; 2. Carrying out of training programs, including for local trainers, on principles and techniques of project analysis and management, and development of a local training methodology that can be applied in all sectors of the economy; 3. On-the-job training to strengthen institutional capacity for the preparation of strategic medium term expenditure and investment frameworks; and 4. Provision of TA on application of project analysis techniques and of impact evaluation techniques . Part C : Studies in support of Sectoral Investment Strategies (estimated costs US$ 8.6 million; actual n .a.) 1. Carrying out of studies to provide the analytical basis for the evaluation of environmental costs and benefits of public-policy decisions and investment projects; 2. Carrying out of studies on natural resources aimed at producing environmentally sound investment strategies; and 3. Carrying out of studies on cross -regional infrastructure aimed at providing basic information for the preparation of related public investment policies and strategies . Part D : Preparation of the National Public Investment Plan (estimated costs US$ 2.7 million; actual n .a.) Provision of TA in the preparation and implementation of the national public investment system . Part E : Project administration (estimated costs US$ 2.5 million; actual n .a.) Carrying out of project coordination and administrative activities . Estimated price contingencies amounted to US$ 2.3 million . July 1999 amendments to the components . On July 2, 1999, the Board approved amendments to the original components without, however, modifying the original objective of the project . This was prompted by a decision of GOA to : (i) place greater stress on assistance in program and project evaluation, with substantially more attention to results -oriented criteria; (ii) de-emphasize assistance for development of public investment strategies and reinforcing the national public investment system; (iii) reduce support for sector studies in favor of greater concentration on public expenditures; and (iv) reduce the number of seminars and broad-scale training. GOA also decided to reduce the scale of the activities as a result of budget constraints, bringing total costs down from US$ 25 million to US$16.5 million; it also requested a loan cancellation of US$5.5 million. Parts A to D were modified as follows: Part A : Policy and Legislative Framework 1. Carrying out of a series of seminars and other training activities on topics aimed at familiarizing decision -makers and other public sector officials with best practices with respect to the evaluation of public investment and public expenditure and at contributing to the strengthened implementation of the Public Investment Law; 2. In line with Part A 1: (a) identification and preparation of possible modifications to the Public Investment Law and its regulatory framework to improve provisions regarding results -oriented policies and practices; and (b) development of other proposals for establishing policies emphasizing results -oriented principles and practices in public sector management; 3. Training decision-makers and technical staff on the proper application of the legislative and regulatory framework governing public investment; and 4. Carrying out informational activities, for the benefit of relevant public sector staff, on international experience with public sector performance management . Part B : Improvement of Project Analysis and Evaluation Capacity of Agencies 1. Carrying out of capacity-strengthening courses for federal, provincial and municipal public investment agencies and related government entities on investment decision -making, project preparation and management, and program and project monitoring and evaluation; and 2. Provision of TA on the application of project analysis techniques and monitoring and evaluation techniques . Part C : Studies in Public Expenditure 1. Carrying out of studies aimed at providing the analytical base for improved expenditure planning in the preparation of sector strategies and project development, and for expenditure reviews, with particular emphasis on social areas; 2. Carrying out of studies on the effects of national privatization measures, actions designed to improve the performance of critical social programs and other activities related to significant national expenditure issues; and 3. Carrying out of studies on natural resources and infrastructure for obtaining basic information for the preparation of related public investment strategies and projects . SERP) Part D : Strengthening of Secretariat of Economic and Regional Plannin g (SERP) Provision of TA to three directorates of SERP in guiding and bolstering the program and project analytical and evaluation functions of public sector expenditure agencies : the National Directorate for Public Investment, the National Directorate for Social Expenditure Control and the National Directorate for Projects with International Organizations. June 2000 restructuring The two new components were added : Part E : Results -Oriented Management of the Borrower ’s Public Administration 1. Provision of TA and training to staff to design, establish and apply a results -oriented management approach based on: (a) strategic planning; (b) use and monitoring of results agreements entered into by federal public sector entities as a tool for defining and measuring expected public agency performance; (c) application of a system of sanctions and incentives to motivate compliance with such agreements; and (d) adaptation of the procurement and civil service rules to conform to such an approach; and 2. Carrying out of seminars and technical briefings as a means to disseminate the aims and achievements of such a results-oriented management approach among specialists in the field, communication media, academic institutions, interested non-governmental organizations and, through such intermediaries, society at large . Part F : Public Service Standards for the Borrower ’s Public Administration Provision of TA and training to staff to design, establish and apply a system of formal commitment to compliance with monitorable standards for the provision of governmental services to the public . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The Loan Agreement became effective in April 1997, some 17 months after approval because of divergent views within GOA as to implementation responsibilities . The project had a lifespan of 11 years, with seven extensions of the closing date. US$5.5 million and US$0.2 million were cancelled in April 1999 and in December 2004, respectively, at the request of the Borrower . Total disbursement amounted to US$ 10.3 million. The ICR (Annex 1) does not provide adequate information on the contribution from the Borrower or on final project costs. 3. Relevance of Objectives & Design: Relevance of objectives . The nature of the stated objectives (original and revised) was relevant to the structural reforms undertaken by the country since 1989 in multiple areas, including in the budget process and public expenditure. One of the main axes of the 1995 CAS was to support the consolidation of these reforms, inter alia, to improve competitiveness and rebuild the institutional base of the economy, which had considerably deteriorated during the 1980s. Along these lines, the project aimed at improving returns to public investment . The nature of the objectives is also consistent with the 2006-08 CAS which, under improved governance, stresses the need to strengthen efficiency and transparency in public expenditure management . Relevance of design . The project was conceived as a stand -alone TA operation directed at the entire public investment system at the national level, as opposed to other ongoing projects with a TA component directed at a particular sector. Thus, at appraisal, it covered the key central ministries, line ministries, and other government investment agencies. The design of the operation evolved as a result of the July 1999 and June 2000 amendments. With the July 1999 amendment, the project's coverage was extended to the provincial and municipal levels, but the focus of the operation remained the public investment system, however complex the operation had become as a result. In contrast, the June 2000 restructuring considerably distracted from pursuing the first objective . As it turned out, this effort was futile as the new administration was short -lived, prompting efforts to refocus the operation again on the public investment system, but through other means than originally envisaged . IEG concurs with the ICR's view that, as a result of political vagaries, activities were repeatedly modified and revised according to the priorities of the successive administrations; in other words, the focus and content of the project constantly evolved. Nonetheless, IEG is of the view that, from the very beginning, the project was over -ambitious and overly-complex, attempting to cover too many different things with a myriad of activities, at the expense of clarity, simplicity, focus, and consistency . Its design partly defeated its very objectives . As the ICR notes, the Program Document (PD) did not have a sound Results Framework linking objectives and design to outcomes. The indicators reflected outputs - even activities, like the number of participants in a seminar - rather than outcomes. According to the ICR, indicators were added for the two new components introduced in June 2000, but there is no evidence of this in the amendment to the Loan Agreement . 4. Achievement of Objectives (Efficacy): Objective 1: to enhance the effectiveness and efficiency of public expenditure management . Modest . In the absence of indicators allowing to measure direct outcomes of this objective, such as a finding of better economic returns on public investments, the ICR lists the following main outputs : (i) establishment of a National Public Investment System (NPIS), including principles, norms, procedures, and related requirements to support the system; (ii) development and implementation of a revised and more performing database of public investment projects (referred to as BAPIN-II) (as a result, the number of users and of projects in the BAPIN database increased from May 2003 to October 2006); (iii) regular production since 2003 of the three-year National Public Investment Plan (NPIP); and (iv) a wide-ranging capacity building program through the delivery of seminars, workshops, training, etc . Many provinces have now adopted the NPIS and are linked to BAPIN, while developing their own compatible database . Despite the substantial outputs delivered under the project, IEG is not in a position to make a judgment on the achievement of objectives in terms of increased effectiveness and efficiency in public expenditure management, thus justifying a rating of Modest. Objective 2 (added at the June 2000 restructuring): to support the establishment of a result -oriented management approach and of a monitorable emphasis on providing service to the public . Modest . The same remark applies as in the preceding paragraph regarding outputs vs . outcomes. A results-oriented management approach was established by implementing two financial management systems : (i) a management and budget control system for executing agencies funded by International Financial Institutions (IFI), such as the Bank and IDB, allowing for handling multi-annual budget and multiple currencies; and (ii) an harmonized integrated financial information system for national agencies, replacing the different applications of the national system that existed before . The sub-component aimed at establishing "a monitorable emphasis on providing service to the public " was not implemented. 5. Efficiency (not applicable to DPLs): N.A. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Overall outcome is rated Moderately Unsatisfactory on the basis of the lack of evidence of outcomes that may be linked to the project. However, IEG notes that the outputs delivered should constitute a solid basis to perfect the different systems in place so as to allow the monitoring of economic performance of public investments in terms of economic returns. a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The legal and institutional framework in place, a more stable political situation, and the outputs delivered under the project minimize the risks to the future achievement of positive outcomes . A key factor in reaching the latter will be a successful implementation of the successor project . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Quality at entry was inadequate, therefore rated moderately unsatisfactory : (i) the project was much too complex, attempting to reach too many actors (not only government officials, but other agents, such as journalists, private sector, etc.); (ii) the Bank overestimated the commitment to and ownership of the project not only at the political but also at the technical level (cfr. the conflict early on between different ministries regarding implementation responsibilities); and (iii) an overemphasis on seminars, workshops, and training, as noted by the ICR . The Quality of Supervision Assessment by QAG in 2001 rated both readiness for implementation and borrower commitment as marginal. Supervision is rated moderately unsatisfactory . IEG concurs with the ICR - and Bank files support this finding - that Bank supervision was poor until the early 2000. Supervision was made particularly difficult by the constantly changing political and economic environment, including the changes in priorities of different administrations . As a result, the project had a lifespan of 11 years requiring seven extensions . at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Unsatisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: The volatile political situation and the changes in administrations negatively affected GOA's commitment to the central objective of the project, i .e., putting in place a well performing public investment system so as to increase the efficiency of investments to the benefit of the economy as a whole . They also impacted negatively on project implementation. The lack of consistency in approach on the part of GOA is illustrated, inter alia, by the uneven support for the operation and the appointment of numerous project directors, as noted by the ICR . a. Government Performance :Moderately Unsatisfactory b. Implementing Agency Performance :Moderately Unsatisfactory c. Overall Borrower Performance :Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: As already noted, the list of indicators related to outputs - or even activities - and not to outcomes, therefore preventing the monitoring of progress in achieving the project objectives and precluding an assessment of achievements. a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Nothing special. 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately See Sections 4 and 6. Satisfactory Unsatisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately See Section 8. Satisfactory Unsatisfactory Borrower Performance : Moderately Moderately See Section 9. Satisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: 1. Although in this specific case the project contributed to maintain the Bank /Borrower dialogue during a highly volatile political situation, there are cases when it might be appropriate to cancel a project when too many operational difficulties and/or too long delays arise. 2.M&E Indicators should be linked to the objectives and defined as allowing measuring progress in outcomes in qualitative and quantitative terms . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is barely satisfactory . Although it has many insightful comments on project design, implementation, etc ., the ICR has many shortcomings in presentation : (i) it lacks in conciseness (too many details; duplication; much too long - the guidelines recommend 10 to 15 pages); (ii) Section F of the Data Sheet (Indicators) lacks in clarity; (iii) the dates of the amendments are not correct and the treatment of the latter is often confusing; (iv) Annex 1 is confusing and does not provide cost data or financing by GOA; also, it reports total disbursement as US$ 10.5 million instead of US$10.3 million. a.Quality of ICR Rating : Satisfactory