Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sust Finan of East Cari Marine Resources (P103470) Report Number : ICRR0020840 1. Project Data Project ID Project Name P103470 Sust Finan of East Cari Marine Resources Country Practice Area(Lead) OECS Countries Environment & Natural Resources L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-10220 31-Dec-2016 18,872,000.00 Bank Approval Date Closing Date (Actual) 04-Aug-2011 31-Dec-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 8,750,000.00 8,750,000.00 Revised Commitment 8,743,691.10 8,743,691.10 Actual 8,743,691.10 8,743,691.10 Prepared by Reviewed by ICR Review Coordinator Group Katharina Ferl John R. Eriksson Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives The grant agreement of September 6, 2011 (p. 6) and the Project Appraisal Document (PAD) (p. 4) state that the objective of the project was “to enhance the long-term sustainability of protected area networks in the participating countries by i) establishing sustainable financing mechanisms, ii) strengthening of the marine protected area networks, and iii) deploying a regional monitoring and information system for the protected area networks.” The objective of the project is the same as the Global Environmental Objective. Page 1 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sust Finan of East Cari Marine Resources (P103470) b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The project included four components: Component 1: Establishment of Sustainable Financing Mechanisms (appraisal estimate US$15.77 million of which US$7.6 million GEF financing, actual US$20.70, 131% of appraisal estimate): This component was to finance the facilitation of the establishment of the Caribbean Biodiversity Fund (CBF) and carry out its initial capitalization. The aim was to generate sufficient income to finance sustainable management activities in the protected areas of participating countries through their respective National Protected Area Trust Funds (NPATF). Furthermore, this component was to finance the facilitation of establishing respective NPATFs in each of the participating countries, as well as the designing and implementing of a capitalization strategy to generate additional financing to that provided by the Caribbean Diversity Fund. Component 2: Strengthening and phased expansion of marine protected area networks (appraisal estimate US$0.80 million of which US$0.25 million GEF financing, actual US$0.21 million, 25.9% of appraisal estimate. Contributions by other partners did not materialize and GEF was the only source of financing): This component was to finance the expansion of the systems of the marine protected areas through the designation of new ones and the establishment of demonstration sites in participating countries to showcase best practices in the management of marine protected areas. Component 3: Deployment of a regional monitoring and information system (appraisal estimate US$0.57 million of which US$0.34 million GEF financing, actual US$0.18 million, 32% of appraisal estimate. Contributions by other partners did not materialize and GEF was the only source of financing): This component was to finance the facilitation of eco-regional and management effectiveness monitoring including regular observation of data collection on biophysical and social economic indicators within the protected areas network, and assessment of management effectiveness. Furthermore, this component was to finance the establishment of an electronic database for an eco-regional environmental information system and the facilitation of the dissemination of results derived from monitoring systems. Component 4: Project management and coordination (appraisal estimate US$1.72 million of which US$0.56 million GEF financing, actual US$1.11 million, 64.8% of appraisal estimate): This component was to finance the coordination and supervision of the managerial, financial, and technical coordination and M&E activities of the project at the regional and participating country levels. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The project was estimated to cost US$18.18 million, and actual cost was US$22.21 million, 117.7% of the appraisal estimate. Financing: The project was to be financed through a US$8.75 million grant by the GEF, a US$4.50 million grant by the German Development Bank (KFW), and a US$4.70 million grant by The Nature Page 2 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sust Finan of East Cari Marine Resources (P103470) Conservancy (TNC) Borrower Contribution: The borrower (TNC) was to contribute US$0.93 million. Actual contribution was US$3.35 million. Dates: The project was restructured on March 27, 2014 to: i) cancel some activities under component 2 (due to insufficient project funds to accomplish objective of sub-component, lack of support by OECS governments, and requirement of large fundings by OECS countries) and reallocate funds to other activities within the component; ii) reflect changes in donor financing to the Caribbean Biodiversity Fund endowment and subsequent minor changes to the grant agreement; iii) adapt the Results Framework to reflect the restructured activities and modify language of some indicators to increase implementation flexibility. 3. Relevance of Objectives & Design a. Relevance of Objectives High: The objectives of the project were highly relevant given the economic and environmental importance of marine ecosystems in the Eastern Caribbean and the countries included in the project, Antigua and Barbuda, Grenada, Saint Kitts and Nevis, Saint Lucia, and Saint Vincent and the Grenadines. According to a study included in the PAD (p. 2), between US$350 million and US$870 million could be lost between 2015 and 2050 per year due to declining fish stocks, reduced tourism, loss of shoreline protection, and coral reef degradation throughout the region. The objectives of the project were in line with the countries’ national strategies. Many of the countries had, at their highest level, (Prime Minister level) signed an agreement to expand protection of their near shore line marine area as part of their national strategies. The agreements are part of the Caribbean Challenge Initiative, which aims to protect and sustainably manage 20% of the Caribbean’s marine and coastal ecosystems by 2020. Also, the objective of the project was in line with the Bank’s Regional Partnership Strategy (2010-2014) which aimed to strengthen resilience to climate change. In addition, the Bank’s current Regional Strategy (FY2015-2019) emphasizes the importance to increase the region’s resilience to extreme weather events associated with climate change. Furthermore, the project supported several priority focal areas for the fourth Global Environment Facility (GEF) including biodiversity conservation, international waters, and adaptation to climate change. Rating High b. Relevance of Design Modest: The planned activities were logically and plausibly linked to the achievement of the project’s objectives. Activities to establish sustainable financing mechanisms included the establishment and Page 3 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sust Finan of East Cari Marine Resources (P103470) capitalization of the CBF National Protected Area Trust Funds (NPTAFS). Activities to strength the marine protected area networks included the establishment of demonstration sites to showcase best practices and the provision of trainings in different areas such as mapping. Activities to support deploying a regional monitoring and information system for the protected area (PA) networks included the establishment of an electronic database for an eco-regional environmental information system the dissemination of results derived from monitoring systems. However, the project design was overly ambitious and planned to establish, with a limited allocation of funds in a limited timeframe, five different NPATFs in five different countries, requiring a legally and political complex process. Even though the logic of the Results Framework was adequate, targets of some intermediate outcome indicators were unrealistic. Rating Modest 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Establishing sustainable financing mechanisms: Rationale Outputs: • The NPATFs did not provide financing as set out as the original or restructured target since the CBF had not transferred funds to them due to delays in the setting up and operationalization of NPATFs. This target was not achieved. • Sustainable financing mechanisms to provide resources for long-term management and use of PAs in each participating country were not established. Therefore, this target was not achieved. • Sustainable financing plans were completed in three countries. These plans outlined the amount of funds the country planned to spend on conservation, the amount of funds they received, the types of activities supported, and the financial gap between what was received and what was planned, thereby achieving the target. • Four countries adopted a financing mechanism to generate new and additional resources to at least match the investment income from the endowment fund for the country, thereby achieving the target. Outcomes: • The Caribbean Biodiversity Fund (CBF) was legally established and was operational with a board, secretariat, an asset manager and work program. Furthermore, the Fund built an endowment of US$32 Page 4 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sust Finan of East Cari Marine Resources (P103470) million, achieving the target. • The CBF generated an investment income of US$2.4 million by the end of the project, largely surpassing the target of US$0.25 million. Taking all the different outputs produced and outcomes achieved under this objective, the overall rating is Substantial. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Strengthening of the marine protected area networks: Rationale Outputs: • Demonstration sites to generate useful marine protected area (MPA) management information and lessons were established in Grenada and in Antigua and Barbuda, achieving the target. • Findings have been published and disseminated. Protected area (PA) managers have attended seminars and workshops in the region, achieving the target. A total number of 375 staff within the Ministries of Finance and Fisheries were trained through face to face training and webinars. Outcomes: • The increased knowledge which PA managers and PA related staff which they strengthened through training activities had a positive impact on drone activities such as drones are now being used to monitor illegal construction on floodplains. • Based on evidence provided through this project, the government of Antigua and Barbuda developed a national policy on drone usage for PA management. • Survey results show that due to training on water improvement planning in Grenada, stakeholders were more willing to engage and support the MPA and its protection. Also, The Water Quality Improvement Plan was adopted and is being used by the Ministry of Fisheries for Woburn Clark Courts Bay. Rating Substantial PHREVDELTBL Page 5 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sust Finan of East Cari Marine Resources (P103470) PHEFFICACYTBL Objective 3 Objective Deploying a regional monitoring and information system for the protected area (PA) networks: Rationale Outputs: • The region-wide, open access, web-based monitoring system for effective protected area management, CaribNode, became operational, and the outputs were being used by national and regional PA managers and other stakeholders, achieving the target. • Relevant biodiversity, ecological, economic and social indicators for effective management of PA networks in the participating countries were identified and started to be monitored, achieving the target. Outcomes: • Over 2000 users accessed CaribNode during the launch between May and June 2015, allowing access to web-generated reports such as the Caribbean coral reef report cards. Rating Substantial PHREVDELTBL PHREVISEDTBL 5. Efficiency The PAD does not include a traditional Economic analysis due to the difficulty of assigning an economic value to the prevention of degradation of various environmental services provided by protected areas, and how the change in the level of these services affects wellbeing directly and indirectly. The PAD (p. 69) estimated that donors would provide the Caribbean Biodiversity Fund (CBF) with financing of US$2.94 million per country by year 4 of project implementation, and assumed a 7 percent return (net of the CBF management fees), generating US$0.25 million in year 5. The NPATFs were estimated to obtain approximately US$0.17 million per year from the CBF and were to generate an equal match, resulting in a total of US$0.33 million for protected areas. However, the ICR (p. 18) states that by September 2016, the CBF had already achieved returns in the amount of US$2.4 million due to an additional endowment made by KfW resulting in an increase in interest income from the predicted US$0.25 million to US$2.5 million. When the project closed, the CBF obtained US$4.27 million instead of US$2.94 million per country, representing a 31 percent increase in available funds per country by year four. Page 6 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sust Finan of East Cari Marine Resources (P103470) However, implementation delays and the cancellation of activities related to the expansion of marine protected areas due to lack of funds indicated some inefficiency in the use of resources. Given the lack of an Economic and Financial Analyses (internal rates of return) as well as implementation delays, overall Efficiency is rated Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of the project objectives Is High given the economic and environmental importance of marine ecosystems in the Eastern Caribbean. The relevance of design is Modest given the overly ambitious design and unrealistic targets included in the Results Framework. Achievement of the objective to establish sustainable financing mechanisms was Substantial. Achievement of the objective to strengthen the marine protected area networks was Substantial. Achievement of the objective to deploy a regional monitoring and information system network was Substantial. Efficiency was rated Modest. Taking everything together, the outcome rating is Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating The CBF is well-capitalized and fully operational. Furthermore, additional funds will be provided by the KfW. Also, the five NPATFs have been legally established and will soon obtain financing from the CBF. Two countries have already signed their Vertical Agreement and have received their funds in June 2017. Two additional countries will receive funds by the end of 2017. However, there are some risks to the sustainability of the achieved outcomes. In a couple of countries, the NPATFs might encounter the challenge of not being able Page 7 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sust Finan of East Cari Marine Resources (P103470) to raise the required matching resources which is a key prerequisite. Also, several of the countries chose debt- for-nature swaps to raise funds requiring timely, complex legal procedures and political commitment. Taking everything together, the risk to development outcome rating is Substantial. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry The project was built on the Bank’s and GEF”s experience in financing protected areas, especially with the establishment of conservation endowment trust funds. The project design took lessons learned from other donor-administered GEF projects into account. The Bank identified several substantial risk factors such as countries inability to generate matching co- financing to access CBF resources, the challenge of multiple trust funds with limited initial capital generating inadequate investment income to provide resources for a large number of protected areas, and slow creation of NPATFs resulting in implementation delays. An implementation and sustainability risk was identified due to TNC’s lack of a local office in the Eastern Caribbean. Also, the project identified TNC’s and NPATF’s capacity to implement the Bank’s safeguards as risk. Furthermore, insufficient country ownership and political commitment and the countries’ failure to harmonize their policy and programs were also identified as risks. The risk of low procurement capacity at the National Protected Area Trust Funds was identified as high. All these risks materialized against which mitigation efforts were insufficient, resulting in substantial implementation delays. Furthermore, targets for performance indicators included in the Results Framework turned out to be overly ambitious (see section 10a for more details). Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The Bank team conducted bi-annual supervision missions and held monthly audio calls with the participating countries and TNC. The Bank provided assistance to overcome implementation challenges and brought on board additional Bank experts in conservation trust funds and water pollution. The Bank team identified shortcomings in the project design and restructured the project in in order to address these shortcomings in March 2014. The project closed at its planned closing date, 12-31-2016. Quality of Supervision Rating Satisfactory Page 8 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sust Finan of East Cari Marine Resources (P103470) Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance The project included the governments of five different countries (Antigua and Barbuda, Grenada, Saint Kitts and Nevis, Saint Lucia, and Saint Vincent and the Grenadines). The commitment of most governments was relatively strong during project preparation. However, during implementation some governments showed reluctance to support the establishment of new Marine Protection Areas (MPAs) due to the financial commitment of managing the new protected areas once they were legally created. Furthermore, the governments also experienced challenges in establishing NPATFs due to the requirement of matching resources in order to be able to receive donations from the CBF. Governments participated in monthly conference calls with the Bank to discuss implementation and any other issues. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance The National Conservatory (TNC) was the grant recipient and implementing agency at the same time. Since several OECS countries were concerned about a possible conflict of interest due to TNC’s dual role, TNC adapted its implementation approach and messaging, satisfying all participating countries. TNC was committed to the project’s objective and closely cooperated with the OECS Secretariat and the Bank team. According to the ICR (p. 24) TNC performed the project’s financial management, safeguards and M&E functions well. Also, TNC contributed US$3.35 million to the project to support the capitalization of the CBF. However, at the beginning of implementation, the project experienced delays due to procurement issues (see section 11b for more details). Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The objective of the project was clearly specified and well reflected in the Results Framework. The original Results Framework included five PDO and 14 intermediate outcome indicators. The selected indicators were measurable in terms of numbers, timing, and location. Page 9 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sust Finan of East Cari Marine Resources (P103470) The Nature Conservancy (TNC) hired for each country a national project coordinators (totaling five coordinators) to work with the National Implementing Entity to track project progress. However, for example, PDO indicator 1 tried to measure several categories of information at the same time: “The Caribbean Biodiversity Fund (CBF) is operational with a Board, a Secretariat, an Asset Manager, a work program and has an endowment of US$15 million for participating countries to generate income for protected areas (PAs) management.” Also, some PDO and intermediate outcome indicators measured the establishment and capitalization of the regional and national financing mechanisms and the creation of new Marine Protected Areas twice. Furthermore, targets for some performance indicators were overly ambitious and unrealistic such as the CBF to be created within three months and for NPATFs to provide a minimum of US$1.5 million a year for all five countries. b. M&E Implementation TNC had regular interactions with the national project coordinators through monthly calls and semi-annual meetings. TNC contracted out baseline and impact studies and results were shared with the Bank. Also, quarterly progress reports were submitted to the Bank including performance of each indicator and the status of each country in regards to sustainable revenue-generating mechanisms for the NPATF. Activities under components 2 set up monitoring mechanisms to help communities carrying out assessments of the watershed and water quality. Studies conducted under component 3 collected data to build a web-based ecosystem monitoring system. During the March 2014 restructuring, the Results Framework was modified to reflect the restructured activities and adapt the language of some indicators. c. M&E Utilization M&E data was used to inform decision making. The study conducted at the Woburn Clarke’s Court Bay Marine Protected Area in Grenada identified major pollutants to the Marine Protected Area identifying the main source of polluter, informing the key polluter of results, and providing the government with scientific data to follow up pollution abatement strategies. Also, the web-based regional monitoring system includes data from national and regional entities and allows for creating new natural resource management tools in the region. M&E Quality Rating Substantial 11. Other Issues a. Safeguards Page 10 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sust Finan of East Cari Marine Resources (P103470) The project was classified as category B and triggered safeguard policies OP/BP 4.01 (Environmental Assessment), OP/BP 4.04 (Natural Habits), OP/BP 4.26 (Forests), OP/BP 4.11 (Physical Cultural Resources), and OP/BP 4.12 (Involuntary Resettlement). An Environmental Management Framework (EMF) was developed. During implementation an environmental assessment was conducted for only two pilot demonstration projects. The project did not conduct any civil works and there were no significant impacts on the environment or physical cultural property and no need for involuntary resettlement. The ICR (p.10) states that safeguard compliance was Satisfactory. b. Fiduciary Compliance Financial Management A grant specialist within TNC was responsible for all financial management activities of the project. Quarterly interim financial reports were submitted on a timely basis. The project was audited by external auditors and the opinions were unqualified. The financial transactions and reports by the CBF were reviewed by the Bank’s financial specialist and found Satisfactory. Procurement The project’s procurement activities were conducted by a certified contract specialist within TNC who was experienced in managing and implementing Bank and other donor-financed projects. The Bank’s procurement guidelines were followed throughout implementation. During the initial years of implementation the project experienced delays due to differences between Bank procurement requirements and U.S. law, which TNC, as a U.S. NGO, had to follow. It took nine months until differences could be solved. The Bank carried out a post- review in September 2014 and compliance with the Bank’s procurement rules was deemed Satisfactory. c. Unintended impacts (Positive or Negative) --- d. Other 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Satisfactory Satisfactory Risk to Development Raising the required matching Modest Substantial Outcome resources will be a challenge Page 11 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sust Finan of East Cari Marine Resources (P103470) but is a key prerequisite. Also, several of the countries chose debt-for-nature swaps to raise funds requiring timely, complex legal procedures and political commitment. Moderately Moderately Bank Performance --- Satisfactory Satisfactory Moderately Moderately Borrower Performance --- Satisfactory Satisfactory Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The following lessons, with some adaptation of language, have been selected from those presented in the ICR (p. 25-26): Countries’ ability to match donor funds for each year for the next 20 years or more needs to be assessed carefully during the project design phase. Especially countries facing economic difficulties will find it challenging to enact tax laws or work with different industries to come up with the required financing. Mapping out detailed steps for activities that involve the government or other institutions, can be useful to identify actions which are necessary to obtain the desired result. Often internal procedures that are required are often not obvious to the Bank team at the design stage, making appropriate time and budget allocations challenging. Involving the Ministry of Finance in a multi-country environment project is critical since that ministry has the capacity to assess the financial aspects of the project and its potential results. In this project, countries that involved the Ministry of Finance demonstrated stronger performance 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR provides a good overview of project preparation and implementation and is internally consistent. Also, the ICR provides useful lessons learned. However, the ICR does not include a traditional Economic or Financial analysis (cost-benefit or internal rate of return).. Furthermore, the ICR is relatively output driven and Page 12 of 13 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sust Finan of East Cari Marine Resources (P103470) does not explain the results chain well. Overall the ICR is rated Substantial. a. Quality of ICR Rating Substantial Page 13 of 13