ICRR 13073 Report Number : ICRR13073 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 02/16/2010 PROJ ID : P049012 Appraisal Actual Project Name : Coastal US$M ): Project Costs (US$M): 18.76 23.60 Contamination Prevention & Marine Management Project (gef) Country : Argentina Loan /Credit (US$M): Loan/ US$M ): 8.35 7.79 Sector Board : ENV US$M): Cofinancing (US$M ): Sector (s): Sub-national government administration (40%) Central government administration (30%) Ports waterways and shipping (26%) General agriculture fishing and forestry sector (4%) Theme (s): Biodiversity (23% - P) Environmental policies and institutions (22% - P) Water resource management (22% - P) Pollution management and environmental health (22% - P) Participation and civic engagement (11% - S) L/C Number : Board Approval Date : 04/17/2001 Partners involved : GEF Closing Date : 12/31/2006 06/30/2008 Evaluator : Panel Reviewer : Group Manager : Group : Richard C. Worden Fernando Manibog IEGSE ICR Reviews IEGSE 2. Project Objectives and Components: a. Objectives: Global Environmental Objective: The Project's global environmental objective is to support long-term protection of international waters and the conservation and sustainable use of marine resources. This objective will be achieved by financing incremental activities aimed at improving Argentina's capacity to protect marine biodiversity and safeguard Patagonia's marine ecosystem. Project Development Objective: The PAD states that: "The objective of the proposed GEF Project is to strengthen Argentina's efforts to reduce pollution of the Patagonia marine environment and improve sustainable management of marine biodiversity by: (i) Improving oil spill prevention and response capacity and preventing ship-based pollution; (ii) Improving the knowledge base about the Patagonia marine environment and its biodiversity; and (iii) Building capacity and promoting regional knowledge sharing for sustainable management of marine resources. According to the GEF Trust Fund Grant Agreement: "The objective of the Project was to support the Recipient’s efforts to reduce pollution affecting the Patagonia Marine Environment and to protect its biodiversity through: (i) the improvement of oil spill prevention and response capacity; (ii) the improvement of the knowledge base about the Patagonia Marine Environment and its biodiversity; and (iii) the promotion of sustainable management of marine biodiversity." There is no substantive difference between the project objective as stated by the PAD and the GEF Trust Fund Grant Agreement. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): Component 1: Maritime Pollution Prevention (US$ 3.5 M total cost estimated at appraisal and US $ 2.1 M estimated GEF grant at appraisal; US$ 3.8 M total actual cost and US $2.0 M actual disbursed GEF funds.) a) Improve preparedness and response to oil spills and prevent ship -based pollution by: improving database organization and analysis of contingency plans, conducting extensive training for effective oil spills response, Modeling oil spill trajectories, and improving enforcement of the International Convention for the Prevention of Pollution from Ships (MARPOL) regulations on ship-waste discharges. b) Reduce navigational risks by introducing a marine electronic infrastructure program by : enhancing the vessel tracking system (SICOM), and preparing hydrographic maps of critical zones and improving the electronic charts system. Component 2: Marine Biodiversity Protection (US$ 12.5 M total cost estimated at appraisal and US$ 4.3 M estimated GEF grant at appraisal; US$ 12.0 M total actual cost and US$ 3.8 M actual disbursed GEF funds.) a) Improve knowledge base and identify ecologically sensitive areas by : targeting programs to better understand dynamics of ocean production and environmental degradation of key areas of the Patagonia ecosystem, preparing a trans-boundary analysis (TBA) and sensitivity atlas to improve the knowledge base of the Patagonia marine ecosystem and complete identification of ecologically sensitive area, and inter -calibrating key marine laboratories’ diagnostic services. b) Develop marine protection tools, such as : creating priority-setting tools to determine areas of highest marine biodiversity, and to analyze the regulatory and technical aspects of pilot marine reserves, and evaluating incidental by-catch of birds and mammals, and develop a corrective action program based on the severity of impacts to different species. c) Promote capacity building and knowledge sharing on marine biodiversity protection through : a Matching Grant Program with local NGOs and universities . Component 3: Capacity Building, Monitoring & Evaluation, and Project Management (US$ 2.8 M total cost estimated at appraisal and US$ 1.9 M estimated GEF grant at appraisal; US$ 3.0 M total actual cost and US$ 2.0 M actual disbursed GEF funds.) a) Local Capacity Building and Dissemination, b) Monitoring and Evaluation Program, and c) Project Management: the project supported the creation of a Project Execution Unit (PEU) within the Secretaría del Ambiente y Desarrollo Sustentable (SAyDS), and selected UNDP as the executing agency . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: This was a self-standing GEF project. Total estimated project cost at appraisal and GEF grants at appraisal were US $18.76 M and US$ 8.35 M, respectively. When summed across project components, the final total project cost was US$ 18.8 M, and actual disbursed GEF grants were US$ 7.8 M, falling by US$ 558,000 below the appraisal estimate. However, when project costs are added up according to the various sources of funding (i.e., GEF, borrower, and "local in-kind sources"), rather than by project component, the total project cos reported in the ICR is US $ 23.6 M, not US$ 18.8 M. This difference of US$ 4.8 M is due to the significant increase of US$ 7.1 M in "local cash and in-kind co-financing provided by 47 competitive sub-projects" (going from US$ 6.1 M to US$ 13.2 M) minus the drop in the Borrower's (Government of Argentina or GoA) contribution of US$ 1.76 M and GEF disbursements by US$ 0.56 M at the end of the project due to SDR exchange rate appreciation . The project got off to a very slow start and was not completed until 1½ years after the planned closing date (6/30/2008 vs. 12/31/2006). 3. Relevance of Objectives & Design: Relevance of Objectives : The project documents (PAD and ICR) correctly state that the project supported the Country Assistance Strategy (CAS) goal of promoting improved governance and institutional capacity, especially at the provincial or sub-national level. A review of the CAS in effect at the time of project design (2001 - 2004) emphasized taking a "comprehensive approach to effective management programs in forestry, mining, water, and fisheries ... to address emerging natural resource management issues, most notably, fisheries, protection of sensitive areas in Patagonia, etc." The more recent CAS (2006 - 2008) and CPS (2010-2012) documents, applicable at the time of evaluation, reveal that additional investments in infrastructure, including in water supply, sanitation, and flood prevention, and in abating growing resource degradation due to pollution, overfishing, and rapid unplanned tourism development have increased in terms of their relevance to country and Bank priorities . The relevance of the project´s objectives to the Bank’s CAS/CPS is rated as substantial . Relevance of Design : The focus and emphasis placed on "outputs" vs. quantified or qualified statements about expected results or "outcomes" made the design less relevant than if a clear connection between the stated objectives and time-bound targets for achieving them had been established . This was a major flaw in the design of the project, which was never addressed adequately . The project lacked targets related to the objectives . No targets were included to indicate the extent to which the government's efforts were to be strengthened . The project identified five “main sector issues,� regarding the sustainable management of the Patagonian coast and marine environment . They were the following: 1) unsustainable exploitation of marine resources, 2) land-based contamination, 3) oil spills and ship-based (bilge and ballast) pollution, 4) insufficient knowledge of marine eco -regions, and 5) weak institutional capacity. The first two issues were not addressed by the project (PAD) at appraisal; the first because another WB-financed LIL project (Sustainable Fisheries Management Project ) was under preparation. However, that project was later dropped due to “political considerations� by the Borrower. The second issue of concern (land-based contamination) was also not addressed by this project due to the presence of other, unspecified investments being made in wastewater treatment plants and solid waste landfills in coastal communities allegedly made by other lenders at the time of project design and implementation . However, no evidence was presented to substantiate that assertion. The emerging importance of improving management of fisheries and commercial fishing industry practices, as well as providing adequate water supply and sanitation services to growing urban areas, are highlighted in the most recent CAS and CPS. Those documents stress the critical need for new and improved infrastructure in water supply and sanitation as well as flood prevention and protection, as part of the new proposed US $ 840 million APL-1 loan, to address land-based contamination sources . Since this project did not address all five "main sector issues" but only three of them, the other two that were to be addressed by other projects but fell through, the project's results framework became less relevant over time . On the positive side, the three project components were well matched to the development challenges posed by the last three main sector issues identified . The project design also squared with lessons learned from previous experiences regarding the importance of gaining local ownership, which appeared to have been done well through the sub-grants program and by convening the Consultative Commission, as well as developing the implementation capacities of the executing agencies (i.e., the Argentine Coast Guard (PNA) and Naval Hydrographic Service (SHN)). Overall, the relevance of design is rated as modest .. 4. Achievement of Objectives (Efficacy): The overall efficacy rating for the project is substantial . Objective (i): to strengthen Argentina's efforts to reduce pollution of the Patagonia marine environment . Efficacy is rated as Substantial . The project strengthened Argentina ’s efforts, if not its resolve, to reduce pollution of the Patagonia marine environment. A wide array of activities and products were delivered to improve preparedness and response to oil spills and prevent ship -based pollution in the region, enhance marine mapping capabilities to identify ecologically sensitive areas, calibrate marine laboratories ’ diagnostic measurements, develop marine protection tools, and promote capacity building and knowledge sharing about marine biodiversity protection with local universities and environmental NGOs . However, there is little indication of the extent to which these project activities and products contributed to actually strengthening Argentina's capacity to reduce pollution in the Patagonian LME . Admittedly, demonstrating real outcomes is difficult to do when, for example, the hoped -for outcomes are the prevention or minimization of serious ecological damages from accidental tanker oil spills or maritime collisions that were avoided as a result of better ship -tracking and bathometric maps, or minimized through improved oil spill response. Anecdotal evidence was provided in the case of the coordinated oil spill response at Caleta Cordova, but an attempt to better quantify the benefits or outcomes achieved should have been made . Overall, the efficacy for this objective is rated substantial . ii): to improve the sustainable management of Patagonia ’s marine biodiversity . Efficacy is rated as Objective (ii): Modest . The project posited great credence in improving the knowledge base and promoting regional knowledge sharing as the means to improve sustainable management of Patagonia ’s marine biodiversity. However, the political willingness to take actions based upon this improved knowledge cannot be taken for granted . In this case, the Argentine government, due to resistance from the commercial fishing industry, did not make a serious attempt to build the political support required to implement the needed regulatory measures . Hence, it is difficult to imagine how improving the knowledge base or information sharing mechanisms, by themselves, could have ultimately led to the desired outcome. Also, more evidence should have been presented to demonstrate the outcomes from the Bank actions taken with regard to knowledge creation and sharing . Both PDOs were weighted equally and thus the overall efficacy rating is a borderline case between substantial and modest. However, there was some anecdotal evidence of strengthened capabilities by Argentine implementers resulting in a more efficient and coordinated response to two off -shore oil spill incidents, and the uptake of project outputs and capacity utilization rates by the implementing agencies (SHN and PNA) and by other project beneficiaries was very high. This is evidenced by the continuation of many project activities and the continued use of most of the analytical tools and information developed by the project . In addition, it is inherently difficult to prove achievement of a desired outcome based on the absence of an incident . Finally, the project contributed to achieving the GEO as was stated in the Trust Fund Grant Agreement . Therefore, for all these reasons, an overall efficacy rating of substantial is given for this project with the caveat that clearer statement of objectives and their linkage to measurable outcomes would have better focused project staff on achieving outcomes . This project made a positive contribution overall to the GEO . 5. Efficiency (not applicable to DPLs): An ERR was not calculated at appraisal or in the ICR . Therefore, other measures of efficiency were used to ascertain whether the outputs and outcomes were commensurate with the inputs . The project was rated as having achieved only modest efficiency due mainly to the government's lack of political will to act on necessary regulatory measures to achieve sustainable use of marine resources, thus casting doubt on whether the costs incurred to improve and share knowledge would eventually lead to the desired benefits . Moreover, serious delays in project roll-out during the first three years of project implementation, resulting in the need to extend the project by 18 months. This delay in implementation was caused by three factors : 1) extended negotiations with the Borrower over the politically sensitive issue of creating a fisheries quota management system, which was eventually dropped from the project, 2) the decision to introduce the UNDP as a disinterested third -party financial intermediary between the Bank and the Borrower to ease implementation, but which turned out to be unexpectedly cumbersome, and 3) the decision to convert from Special Drawings Rights (SDR) to American dollars after the effective date, which introduced uncertainty over payment for goods and services that slowed down project implementation. There were also modifications made to component and sub -component level indicators during the mid-term review (MTR) to more accurately reflect project outputs and intermediate outcomes, which resulted in an improvement in the overall design of the project, but did not change the focus on activities and outputs instead of outcomes. All of these issues caused a loss of efficiency in terms of time and the opportunity cost of lending capacity not fully utilized. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project was rated substantial for relevance of objectives, modest for relevance of design, substantial for efficacy, and modest for efficiency . The overall outcome rating for the project is moderately satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The risks to development outcomes are far from being clearly resolved or known at this time . They revolve around budgetary and staffing decisions involving former Project Implementation Unit staff within the Environmental and Sustainable Development Secretariat (SAyDS) of the Argentine government, and continued utilization of the knowledge gained and the new tools developed as a result of the project by both implementing Argentine governmental agencies and non -governmental organizations alike over the expected remaining useful life of project outcomes. In the former instance, continued funding of project staff in SAyDS has not been sustained according to the ICR; in the latter instance, it appears that both SHN and PNA have committed budgetary resources and developed work plans to maintain the equipment received and continue the activities started under the project . The longer-term sustainability of the PDOs will hinge upon pressure from civil society groups and the public prompting more government commitment and actions to take full advantage of the information and knowledge gained to tackle the thornier issues of commercial fleet over -fishing and needed investments in land -based pollution sources in order to sustainably manage the Patagonian LME ´s bioresources and reduce pollution levels . There is some promising indications regarding progress toward these more politically and economically difficult issues that appear in the latest CPS for 2010 - 2012. This will be the real test of whether they will put the knowledge and training gained from this project into action by tackling the remaining challenges they face to protect and sustainably manage the Patagonian coastal and marine resources . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: at-Entry: Assessment of Quality -at-Entry : Moderately Satisfactory In terms of the project concept, objectives, and approach, t he evaluation concurs with the ICR ’s rationale, institutional analysis, technical and environmental aspects, and assertion of strong stakeholder support for and involvement in the project, but finds moderate shortcomings with the Bank’s performance in terms of financial management and project readiness assessment, the design of the project's objectives, and lack of indicators for outcome or results (vs. outputs). Assessment of Quality of Supervision : Moderately Satisfactory In terms of identifying and assessing implementation and development impact problems in a timely and accurate fashion, and then devising appropriate and effective solutions, the project was rated as having been generally satisfactory with some shortcomings, especially in terms of project start -up delays due to problems with the financial management system and the insertion of the UNDP as a financial intermediary . With respect to the adequacy of Bank supervision inputs and processes, the risk mitigation measures designed to address project financial administration did not work out as smoothly as hoped . Although it was thoroughly discussed and debated during the MTR, the PDO remained overly ambiguous and no targets for expected outcomes were included in the revised output indicators or logical framework . The modification of output indicators clearly improved the ability to monitor and evaluate the outputs of the project better, but not to assess its effectiveness in achieving any measurable changes in the Government's ability to reduce marine pollution or protect biodiversity . It is also unclear why the recommended changes in output indicators as part of monitoring and reporting processes did not occur after the MTR . This indicates a lapse in maintaining adequate Bank supervision of the project. On the other hand, the project took a highly participatory and client -oriented approach, resulting in pragmatic and useful hardware and software procurement decisions . In addition, the matching grants program with local counterparts greatly increased the “ownership� of the project by local counterparts and leveraged project resources with “in-kind� contributions. at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Government Performance : The borrower (SAyDS) generally complied with the grant agreement, coordinated with other donors and partners, and ensured the participation of major stakeholders during project implementation. However, the government did a less satisfactory job in terms of creating the enabling environment of supporting sectoral policies and institutional structures, ensuring its readiness to implement the project initially or respond to the changes recommended during the MTR to the M & E system adequately, or carrying out its fiduciary obligations under the project . There were problems with the borrower assuming ownership and responsibility for achieving the PDOs as evidenced by inadequate staffing for the PIU initially, followed by high staff turn-over of key staff throughout the project, and then a collapse of project staffing leading up to the project’s closure. Thus, there were shortcomings in the government ’s overall performance, resulting in a rating of moderately satisfactory . Implementing Agencies ’ Performance : There were three implementing agencies (PIU, SHN, and PNA). While there were several issues associated with financial management and procurement which caused delays to the project, the PIU appears to have eventually done a good job of facilitating the participation of major stakeholders and the Consultative Commission, as well as involving interested parties (e.g., local universities and environmental NGOs) in civil society through the small grants program, thereby greatly leveraging “in-kind� contributions to the project. SHN and PNA appear to have done an outstanding job of preparing themselves to implement the project, actively integrating the training and equipment they received from the project into their work plans and continuing to use the training, equipment, and facilities to improve performance, information management, and inter-agency coordination beyond the close of the project . They assumed their responsibilities as members of the Consultative Commission seriously, and performed well under live conditions in two actual maritime ship-based oil spill incidents off the Coast of Argentina at Caleta Cordova and Rio de la Plata . Overall, implementing agencies' performance is rated as satisfactory . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: M&E Design: The Project’s M&E design was initially unclear, confusing the project ’s two PDO with the three means by which it hoped to achieve them (that is, the three qualifying definitions regarding the scope of the PDOs ) around which project components were then designed . Intermediate output and outcome indicators were often confused with one another, particularly when defining enhanced management capacity in terms of real outcomes or results. This caused the project to have considerable difficulty in developing project indicators that were both meaningful and measurable. This challenge was ultimately addressed during the MTR, but not completely overcome. Therefore, the M&E design is rated as modest . M&E Implementation: The indicators defined at appraisal were not reflected systematically in the ISR record during the initial years of the project due in large part to project implementation delays . In the fourth year of the project, a MTR was undertaken to examine the Results Framework . As mentioned above, a number of outcome indicators were changed in order to monitor and evaluate the project outputs more accurately . However, while the new output indicators were reflected and tracked in subsequent ISRs, these were not fully consistent with the indicators agreed to by the MTR, and there is no indication given as to why this occurred . Finally, two consultancies were carried out during March 2007 and June 2008 to further improve the indicators originally defined for the project, this last consultancy occurring in the last month of the project . Therefore, M&E implementation is rated as modest . Utilization: It was unclear from the ICR how the M&E data was utilized in project implementation . Outputs seem to have been comprehensively monitored and recorded, but no targets or dates by which outcomes would be achieved were mentioned in project documents . This made it difficult to rate M&E utilization . Therefore, in the absence of clearer information, the rating is downgraded to modest . a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): The project triggered two safeguard policies : Environmental Assessment (OP4.01) and Projects in Disputed Areas (OP 7.60). The Project was classified as a Category “C� project because it did not cause any significant direct or induce indirect negative or adverse impacts on the environment . The Disputed Areas safeguard policy was also triggered because the project originally intended to cover fisheries management issues in the area of the Malvinas/Falklands Islands, but a fisheries component was not included in the final project design . Therefore, the Safeguard policies were effectively a “non-issue.� In terms of fiduciary, financial management, and procurement compliance, an external audit of the project and two ex post reviews of contracts were done annually, finding that the Borrower and the UNDP has met the conditions of the loan and the applicable regulations in general . Regarding financial management, the ex post reviews concluded that the procurement process had not been adequately documented . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately The project was rated substantial for Satisfactory relevance of objectives, modest for relevance of design, substantial for efficacy, and modest for efficiency . Risk to Development Negligible to Low Moderate The long-term RDO is far from clear at Outcome : this time, depending on political decisions regarding remaining challenges to the Patagonian coastal and marine eco-region. Funding to maintain project staff in SAyDS has not been sustained, but the implementing agencies have continued activities started under project. There are some promising indications in the latest CPS of progress in reducing marine/near coastal pollution and protecting biodiversity going forward. Bank Performance : Satisfactory Moderately Quality at entry and quality of Satisfactory supervision were moderately satisfactory (as discussed in section 8). The ICR rates quality at entry as satisfactory and quality of supervision moderately satisfactory--according to the harmonized criteria this should have resulted in a moderately satisfactory rating. Borrower Performance : Satisfactory Moderately Government Performance is rated Satisfactory moderately satisfactory and Implementing Agency Performance satisfactory (as discussed in section 9). The ICR rates Government Performance as satisfactory and Implementing Agency performance as moderately satisfactory--according to the harmonized criteria this should have resulted in a moderately satisfactory rating. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Public -private partnerships can add unexpected value and benefits to projects, and should be systematically sought out, analyzed, and taken full advantage of during project preparation, design, and implementation . Involvement by local NGOs, universities, and research institutions in the matching funds grant program was one of the most successful elements of the project, reflected by the important contributions they made to the project ’s eventual outcomes. Thus, the lesson learned here is the importance of considering and then attempting to align project activities with those of local civil society groups so as to leverage project resources and expand the project's ultimate impact/outcomes. Knowledge generation is a necessary, but insufficient without a means to leverage it to build public and political support to drive policy initiatives and reform . It does not appear that the borrower was fully committed to the PDO/GEO of this GEF project, as evidenced by the lack of and high turn -over of project staff in SAyDS, and then by the lack of funding to continue staffing to fully realize the development objectives of the project . Thus, building local knowledge and information-sharing networks is not enough in itself, nor an end in itself, but rather a means to a greater end when it is put to use . It is important to define realistic and measurable parameters of expected results or benefits to be achieved as a result of project interventions . In this project there was considerable confusion around the two PDOs and the three means selected to achieve them in the ICR . This created a ripple effect throughout the ICR in terms of failing to distinguish between outputs and outcomes or results . The almost total lack of targets for objectives and the dates by when they would be achieved (e.g., X % reduction in pollution loadings by year Y ) made it very difficult to determine how to assign weights to the relative importance of various outputs' contributions toward achieving project objectives or desired outcomes . For example, how much reduction in marine pollution was expected to be achieved from ship-based sources relative to land -based sources, or what would be define as a “success� in terms of strengthening management capacity for biodiversity protection? 14. Assessment Recommended? Yes No Why? To verify the ratings. 15. Comments on Quality of ICR: The ICR was clearly and logically written especially given the large turnover of TTLs over the life of the project and is satisfactory overall, but could have been stronger in specific areas . The ICR should have been more outcome and results-oriented. To the extent relevant data were lacking, it should have discussed M&E gaps more systematically and, where possible, gathered more data, at least qualitative data . In addition, due to the “in-kind contributions� of local NGOs, universities, and research institutions under the small grants program, it was very difficult to add up the cost numbers between the appraisal estimates and the actual expenditures . Clearer presentations of financial data should have been made. The ICR does not correctly apply the formula for overall Bank and Borrower Performance which should both have been rated moderately satisfactory given the ICR's sub -ratings. a.Quality of ICR Rating : Satisfactory