ICRR 12618 Report Number : ICRR12618 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 10/15/2007 PROJ ID : P042573 Appraisal Actual Project Name : GEF Central Asia US$M ): Project Costs (US$M): 13.65 10.59 Biodiversity Gef Project Country : Central Asia Loan/ US$M): Loan /Credit (US$M): 10.15 10.13 Sector Board : ENV Cofinancing (US$M): US$M ): 1.5 1.5 Sector (s): Forestry (47%) Central government administration (22%) General education sector (12%) Other industry (10%) Other social services (9%) Theme (s): Biodiversity (23% - P) Participation and civic engagement (22% - P) Environmental policies and institutions (22% - P) Rural non-farm income generation (22% - P) Law reform (11% - S) L/C Number : Board Approval Date : 06/22/1999 Partners involved : EC Closing Date : 06/30/2004 06/30/2006 Evaluator : Panel Reviewer : Group Manager : Group : Lauren Kelly Ridley Nelson Alain A. Barbu IEGSG 2. Project Objectives and Components: a. Objectives: The Primary Objectives of the Project are to: (1) Support the protection of vulnerable and unique biological communities within the West Tien Shan and to (2) assist the three countries to strengthen and coordinate national polices, regulations and institutional arrangements for biodiversity protection . Associated objectives are: (a) Strengthen and expand the zapovednik (strict nature reserves) network in the West Tien Shan to conserve unique plant and animal communities, including wild relatives of domesticated species; (b) Identify alternative and sustainable income -generating activities for local communities and other stakeholders to reduce pressure on the zapovedniks and their biological resources; (c) Strengthen local and national capacity through education and training; (d) Raise public awareness of biodiversity values and increase participation in biodiversity conservation; and (e) Establish regional (transnational) coordination and cooperation mechanisms for biodiversity conservation activities to strengthen zapovednik management and wildlife protection and prevent the fragmentation of habitat corridors . The Project's Global Objective is to ensure the conservation of the globally important biodiversity within the West Tien Shan. Specific objectives associated with the Global Objective are: (a) Conserve biodiversity through the implementation of an ecosystem-based management approach that involves the strengthening of zapovednik management systems and the integration of a coordinated management concept across regional, national and local programs; (b) Improve knowledge of the distribution and status of rare, endangered and endemic species through targeted surveys to better focus conservation measures; (c) Enhance biodiversity conservation within mountain ecosystems by developing cross-sectoral multi-use management systems to preserve critical ecosystems; (d) Promote the protection of ecosystems, natural habitats, landscapes and the in -situ maintenance of viable populations of species by developing sustainable land-use which integrates conservation management between zapovedniks and adjacent forest production units (leshoz) and farming communities; and (e) Increase the awareness of biodiversity conservation and endangered species by the development of training programs and dissemination of information . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): (A) Legal and Financial Reform (US$0 US$0.55 million estimated at appraisal; $US 0.11 million actual ): assist in improving the policy and legal framework for biodiversity conservation in all three countries and in developing and implementing a strategy for budgetary reform needed to enhance project sustainability; (B) Strengthening the Protected Area Network of the West Tien Shan (US$5 US$ 5.7 million estimated at appraisal; $6.65 million actual ): Finalize management plans for four protected areas (Aksu Djabagly Zapovednik, Chatkal Zapovednik/Ugam-Chatkal National Park, Sary Chelek Zapovednik and the Besh Aral Zapovednik ) and their implementation, including technical assistance, training, communications and other equipment, and infrastructure . Conduct rapid biological assessments at sites to rationalize protected area boundaries, identify appropriate land/resource use and zoning needs, and improve habitat management and species protection . (C) Sustainable Uses of Biodiversity (US$1 US$ 1.0 million estimated at appraisal; US$ .86 million actual ): Finance training activities and a small grants program that will finance model or pilot projects (sustainable grazing, forest management) or sustainable use of biodiversity activities, to be developed and implemented by local communities, NGOs and individuals living in villages around the project protected areas . The program will also provide seed money for activities that have the potential to develop alternative livelihoods which reduce pressure on the parks and their biological resources. The small grants program was designed to also finance demand -driven activities to develop rural tourism, such as technical advice and training for providing accommodation in rural homes, guiding and site interpretation; cottage industries for handicrafts, nursery development, traditional resource use practices, renewable energy projects (photovoltaic systems) and appropriate animal husbandry programs . (D) Strengthening Local and National Capacity Through Education and Training (US$1 US$ 1.5 million estimated at appraisal; US$ 0.81 million actual ): implement a professional development, education and training program to build the necessary capacity for : (a) protected areas management; (b) conservation field skills; (c) public awareness; (d) project management skills; (e) business management; and (f) training of trainers and maintaining a trainer network . The project will support opportunities for regional training, work -study visits, and exchange of regional expertise and support implementation of project components . (E) Public Participation in Biodiversity Conservation (US$2US$ 2.8 million estimated at appraisal; US$ 0.00 actual ) finances the participation of local communities in habitat restoration and reforestation in areas adjoining the protected areas with priority given to areas that function as wildlife corridors, or are otherwise important for biodiversity conservation. In order to reduce pressure on local forests for fuelwood, the project will support a study on alternative energy sources and potential financing for its implementation . This component was designed to be complemented by activities financed under the small grants program . (F) Project Management (US$2 US$2.1 million estimated at appraisal; US$ 2.6 million actual ): The project will be managed by the Regional PIU (RPIU) in Kyrgystan and National PIUs (NPIUs), under the direction of responsible line Ministries. It should be noted that the establishment of national and regional project implementation units was a condition of project effectiveness . As recommended by the Mid-Term Review, Components C and E were "merged" however all planned finance for Component E was transferred to Component B, while actual financing for Component C - the Small Grants Program -- was less than planned. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The project was extended by two years, due (according to the ICR) to the difficulty associated with starting a new operation in three countries (with cumbersome and different implementation procedures ) and reduction in counterpart funding -- from an initial US$2.0 million to US$0.9m from the Kyrgyz Republic compounded by the decision of Kazakhstan and Uzbekistan to reduce their contributions to "maintain a balance" rather than "stepping in to help." (ICR p.5) 3. Relevance of Objectives & Design: Relevance of Objectives : Although highly relevant from a global environmental perspective and in line with the GEF Operational Strategy, the objectives of this project are only moderately relevant from the Bank's country program perspective. First, the multiple and overlapping layers from two sets of objectives labeled in the ICR "original" and "global" is extremely confusing and makes evaluation against objectives difficult . For example, the first "associated" objective under "original" is a different formulation of the first under "Global." Second, the PAD refers to the specific CAS objectives which are triggered by this project (poverty reduction through integrated rural natural resource management etc .). A review of the three countries' CASs in place at the time of project design, reveals that biodiversity conservation is not specifically targeted in the CASs as an environmental priority. Rather, in Kazakhstan, the 1997 CAS emphasizes the need for an improved legal and regulatory framework for environmental management, with reference mainly to oil, power, gas, mining and infrastructure . Although the Central Biodiversity project is mentioned as a 'smaller regional activity', the CAS emphasizes that "the Bank's principal direct contribution to environmental improvement in Kazakhstan to date has been through, and will remain the regional Aral Sea Basin Program (ASBP). The 1998 Kyrgyz Republic CAS also refers to the fact that the "Government's strategy of improving the legal and regulatory framework, including the introduction of project level environmental assessments; enhancing institutional capacity to enforce environmental standards; addressing past environmental liabilities and developing national biodiversity and water management strategies, lies within the framework of the Aral Sea Basin Program. Uzbekistan's CAS does not mention biodiversity conservation; it exclusively targets the environmental damage resulting from the mismanagement of natural resources in the Aral Sea Basin. Project design is only moderately relevant since the design was less relevant at completion then appraisal . Public participation and the introduction of sustainable alternative livelihoods (specific activities in the PAD include fuelwood plantations and tourist facilities ) were key design aspect of this conservation project which was dropped, per the MTR, as funds were reallocated to purchase more equipment and develop infrastructure as part of Component B. As stated in the ICR, some project objectives and indicators were "over-ambitious." This review concurs. Statements such as 'the Small Grants Program will "lessen" rural poverty in the communities adjacent to the Zapovedniks' were indeed "over-ambitious" in light of the design of the project (and in light of the lack of baselines provided as proxies for rural poverty measurements and the low level of investment initially planned and subsequently lowered for the SGP). The ICR does not provide an adequate rationale for World Bank implementation of this GEF project -- it should have carried forward (from the PAD) more thematic explanation of how the project would conserve biodiversity by addressing the under -investment in agriculture, the collapse of the forestry sector, etc . -- to take pressure off of the protected areas . The only rationale provided is the global environmental objective of conserving a "biodiversity hotspot". 4. Achievement of Objectives (Efficacy): The project had two primary objectives . The first, to support the protection of vulnerable and unique biological communities within the West Tien Shan was substantially achieved . The project successfully created, strengthened, and extended networks of protected areas . This entailed the establishment of four new nature reserves, a national park, and extension of two nature reserves, thus increasing the potential for conservation in Western Tien Shan . The territory protected by nature reserves and parks was expanded by 183 percent. The total territory of Western Tien Shan protected as natural reserves increased from 4 percent in 2000 to near 8 percent in 2006, and the total territory under national parks in the region has increased from 11 percent in 2000 to 13.75 percent in 2006. As a result of project activities, there has been a stable increase and restoration of the population of several species such as ibex, argali, wild boar, marmot, and golden eagles (as measured by frequency of observation, not number of animals ), as well as several species of flora . The project strengthened protected area management . An assessment of the effectiveness of nature reserve management carried out by the IUCN Commission for Northern Eurasia shows that the ratings for the four nature reserves created by the project increased by 25 percent between 2003 and 2005. The project successfully used an ecosystems approach to conservation and developed a bioregional plan for each protected area that contains baseline data developed using a common methodology and scientifically justified recommendations for long -term biodiversity conservation . The second objective, t o assist the three countries to strengthen and coordinate national polices, regulations and institutional arrangements for biodiversity protection , was only negligibly achieved . The project was built on the premise that all three countries were already actively developing national environmental legislation, separately . Each had ratified the CBD and with GEF financing, had completed their national biodiversity strategies . As suggested by the title, the project was designed to "consolidate" the initial legislation through the creation of a Transnational Supervisory Committee and consultant services . While the project succeeded in updating the legal framework for the establishment of a tripartite agreement on the Western Tien Shan Biosphere, the Ministries of Foreign Affairs of the three countries have not yet agreed on the text of the agreement . Given that the core of the project concept was transboundary coordination, this failure is weighed heavily in assessing overall outcome . 5. Efficiency (not applicable to DPLs): Neither an ERR/FRR was conducted for the project as a whole or for the subprojects, however, based on the limited evidence, it appears that the efficiency of this project was low, both from the point of view of the sub -granting arrangements and the costs of project management . A case should have been made, in the ICR, concerning the costs required to prepare and implement a regional GEF project, since coordination and transaction costs are usually higher for a regional program but may be necessary to sustain ownership . According to the ICR, a detailed procurement and financial management review was conducted which reported higher transaction costs in Uzbekistan than in the other two countries - although no specific rationale or lessons are offered to explain how the country comparison was made . An analysis of the cost-effectiveness (the operating costs) of a small grants program, financed at a level of US$ 600,000 which spread its awards across 592 projects would have been warranted in this case . The ICR provided little explanation as to whether there were varying levels of efficiencies between the 412 grants approved for the Kyrgyz Republic, of an average size of $US 500, and the 89 and 92 grants approved in Kazakhstan and Uzbekistan respectively with an average grants size of US$ 2500 and US$3000. Next, an analysis of the transaction costs incurred in convening both the national steering committee meetings and the transnational steering committee meetings against the decision -making capacity and timely outputs of these inter-ministerial groups would have been beneficial in understanding the efficiency of a regional program with a "consensus building" agenda. The project spent 25 percent of its budget on project management . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project's outcome is rated unsatisfactory due to significant shortcomings in the operation ’s achievement of its objectives, its efficiency, and its relevance to the countries' development strategies and the possibility that Bank safeguards were not adhered to due to the assignment of an incorrect EA category a. Outcome Rating : Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: As indicated in the ICR, the "risks that project achievements will not be sustained are "substantial" (p. 13). Risk to development outcome is heightened by the lack of public participation in the development of alternative livelihood schemes and the lack of uptake of the tripartite agreement to more effectively coordinate management plans of the transnational PA scheme. The ICR indicates that the governments of the three countries have expressed an interest for a follow-on operation - which if approved, would help reduce risk by providing additional resources for ongoing project activities, such as nature based tourism, as well as developing new approaches for reforestation and carbon sequestration and trade activities (although only the Kyrgyz Republic is mentioned in this regard in the ICR ). a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: It is not clear that there was sufficient due diligence performed at entry . As noted in the ICR and in the subsequent section on Safeguards, the project team miscategorized the project as an EA category "C" rather than classifying it properly as a "B" considering the activities proposed under the small grants program . Moreover, the Bank proceeded with the project without reviewing the social assessments that had been carried out prior to expansion of the protected areas (retroactive reviews of the reports were conducted ). It is not clear, since the MTR was not made available to the IEG team when it was requested, how Component E was dropped since the ICR indicates that the Bank should have merged C and E (not dropped E). a. Ensuring Quality -at-at -Entry :Moderately Satisfactory b. Quality of Supervision :Moderately Unsatisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: Borrower commitment was mixed. Country governments reduced overall level of counterpart funding and, of particular relevance to outcome, were unable to reach consensus on the Tripartite Biosphere Agreement . The project was generally well managed at the agency level, although M&E is rated modest . However, interviews with Bank staff conducted for IEG's Review of Regional Programs found there were disagreements between the staff of the regional PIU and the national PIUs . In retrospect, one task team leader suggested that allocating responsibilities for specific regional activities to individual national PIUs may have been more effective and efficient, and avoided the creation of a suprastructure over the level of the national PIUs . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: M&E Design As identified in the ICR, the "design of the monitoring and evaluation system lacked specific targets and was somehow overly complex, leading to limited utilization during project implementation " (p.6). The GEF PAD, drafted in 1999, includes a Results Framework in Annex I that sets the Project Objectives against Performance Indicators, Monitoring and Evaluation tools, and critical risks associated with the achievement of the project objectives . Although Annex 1 provides specific performance indicators, such as : 'reduced rate of decline of important species, habitat, and communities within and outside the PA' and 'Increased and expanded conservation activities', it provides no baselines against which to measure progress . The ICR does not utilize this original framework in a logical way . It is also curious how baselines are suddenly introduced in the ICR Results Matrix when they were absent from the PAD . M&E Implementation An assessment of the Small Grants Program was completed in August 2005 by an independent consulting firm in the Kyrgyz Republic. This ICR Review requested the evaluation from the project task manager . IEG reviewed the external evaluation, and found it lacking in several aspects . There is no review of the grant application and selection process (how were proposals solicited, approved or rejected, fund level awarded, implementation capacity assessed, importance of the design of the proposed monitoring system ); The assessment was conducted as the project was closing and hence does not include any baseline data (livelihoods or welfare, environmental ); the questions are highly subjective and attribution is questionable . The assessment was candid in pointing out that in some reserves, illegal hunting and deforestation continue (or that stricture enforcement may have curbed this activity over the incentives provided by the SGP ), that enhanced environmental awareness did not always lead to changes in nature management (cattle grazing and haymaking ), The Bank facilitated the design and conduct of an independent assessment of Protected Areas management performance according to the IUCN scorecard method . This assessment, conducted before and during project implementation allowed for enhanced monitoring and reporting of PA management capacity . M&E Utilization There is no discussion in the ICR how M&E was utilized to affect project performance over the implementation period. Supervision conducted for the mid -term review adjusted component financing, however it is unclear why funds were reallocated to Component B, since the Mid -term review recommended merging Component E with C (the Small Grants Program). a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): As identified in the ICR, the project was misclassified as Env Category "C" instead of a more appropriate "B" classification since it included civil works inside of protected areas, expansion and creation of new protected areas, and the construction of small dams under the Small Grants Program . According to the ICR, this issue was identified in the Mid-term Review triggering a "few sample retroactive Environmental Impact Assessments ." However the ICR does not provide any information on the type and extent of negative impacts that resulted from the miscategorization . A few sample EIAs do not raise this project aspect to a 'satisfactory' rating when the project financed at least 580 subprojects. 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Unsatisfactory The difference is for four main reasons : 1. While the project was relevant from a global environmental perspective, biodiversity conservation is not referred to in the country CASs. 2. Only one of the two objectives was fully achieved and this was the objective signalled by the "Transboundary Biodiversity " element of the project title as central to the whole project concept. 3. The project dropped a critical component relevant to overall sustainability after MTR restructuring. 4.There is little or no evidence of project or subproject efficiency in the ICR. Risk to Development Significant Significant Outcome : Bank Performance : Satisfactory Moderately Both the EA and Social safeguards Unsatisfactory were ill-applied during entry and there was inadequate justification for removing the public participation component from the project, arguably the main constraint to enhancing biodiversity conservation . Borrower Performance : Satisfactory Moderately Lack of will on the part of the Satisfactory government(s) to sign the tripartite agreement stymied achievement of the project's main objective. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: As concluded by IEG's Review of Regional Programs , regional programs often reflect an appropriate rationale for adopting a multicountry approach, such as an ecosystem wide approach to conservation of a hotspot, yet as in this case of the Central Asia Biodiversity project, they are often weakly linked to the country's development objectives, at least as articulated in the Bank's country assistance strategy . A Regional Program, as concluded b y IEG's Review of Regional Programs, is more effective when there is a clear delineation and coordination of the roles of national and regional institutions . In the case of the Central Asia Biodiversity project, implementation arrangements were split between regional strategic coordination and national level implementation. The National and Transboundary Steering Committees provided the strategic guidance and coordination, while the national PIUs implemented national activities at national level . The success of regional activities planned through a regional project often depends on the adequate contribution of the national structures. In this case, frequent delay of one of the sides delayed execution of all activities . To avoid such problems it is necessary to strengthen the role of a regional supervisory body, such as the Transboundary Steering Committee. Regional projects are usually challenging in terms of overall management to sustain ownership and commitment from the individual countries. This requires more time to achieve a consensus when this is essential, as in the case of the tripartite agreement to establish the Transboundary Biosphere . Even with the necessary coordinating mechanisms in place, projects should be realistic about the efforts required to harmonize legislation across countries with disparate land ownership and resource rights regimes . 14. Assessment Recommended? Yes No Why? IEG may want to conduct a "thematic review", through a series of cluster audits, of the performance of environmental small grants programs designed as components of larger projects . Although IEG could select a portfolio of large, Bank/GEF blended projects, inclusion of some GEF stand alone full size Bank implemented projects could serve as a useful comparison . 15. Comments on Quality of ICR: The ICR is rated Satisfactory, although its main shortcoming is lack of reporting on the environmental outcomes achieved through the Small Grants Program . The ICR reports that the Small Grants Program was designed to "change behavior" toward environmental protection and biodiversity conservation . Yet the ICR does not provide any evidence of behavioral change, rather it reports mainly on (1) number of trees planted and (2) jobs created (handicrafts, carpets, clothing, fruit processing ). Nevertheless, the ICR is candid about the project's shortcomings . The ICR is results-based, although reporting is sporadic (discussed in M&E section). The ICR should have more effectively linked its analysis to the PAD, particularly in relation to the Project Context - which contained a more nuanced description of the development challenges associated with conserving the Western Tien Shan . a.Quality of ICR Rating : Satisfactory