OPS3: Progressing Toward Environmental Results Third Overall Performance Study of the Global Environment Facility EXECUTIVE VERSION OPS3: Progressing Toward Environmental Results Third Overall Performance Study of the Global Environment Facility EXECUTIVE VERSION* June 2005 * The full text of the study can be found on the CD on the inside back cover. © 2005 Office of Monitoring and Evaluation of the Global Environment Facility 1818 H Street, NW Washington, DC 20433 Internet: www.thegef.org E-mail: gefteam@thegef.org All rights reserved. The findings, interpretations, and conclusions expressed herein are those of the author(s) and do not necessarily reflect the views of the GEF Council or the governments they represent. The GEF Office of Monitoring and Evaluation does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of the GEF concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The GEF encourages dissemination of its work and will normally grant permission promptly. ISBN: 1-8841-2248-5 Global Environment Facility Director of the Office of Monitoring and Evaluation: Robert D. van den Berg Task Manager, Office of Monitoring and Evaluation: Claudio R. Volonte OPS3 Team: ICF Consulting and partners 1725 I Street, NW, Suite 1000 Washington, DC 20006 www.icfconsulting.com A FREE PUBLICATION Contents Foreword v Preface ix OPS3 Teams xi Purpose and Scope 1 Main Findings 3 Focal Area Results 3 Strategic Programming for Results--Focal Area Level 4 Strategic Programming for Results--Country Level 6 Responsiveness to Conventions 7 Information Management within the GEF Network 9 Network Responsibilities and Administration 10 Small Grants Programme 13 Major Recommendations 15 Programming for Results--Focal Area Level 15 Programming for Results--Country Level 17 Responsiveness to Conventions 17 Information Management within the GEF Network 18 Network Responsibilities and Administration 18 Small Grants Programme 19 iii Approach and Methodology 21 Evaluation Challenges and Strategies 21 Elements of the OPS3 Approach 23 Focal Area Analysis 25 Portfolio Analysis (TOR 1E) 25 Biodiversity (TORs 1A and 1B) 25 Climate Change (TORs 1A and 1B) 30 International Waters (TORs 1A and 1B) 32 Ozone Layer Depletion (TORs 1A and 1B) 34 Land Degradation (TOR 1C) 36 POPs (TOR 1C) 37 Responsiveness of the GEF to Conventions (TOR 4C) 38 Achieving and Sustaining Global Environmental Benefits 41 Findings (TORs 1D, 2A, 2B, and 2C) 42 Challenges and Strategic Trade-Offs 43 Recommendations 43 The GEF as a Catalyst 45 Catalytic Impacts: Extent and Factors (TORs 3A and 3B) 45 National Priorities of Recipient Countries (TOR 4E) 49 Varying Capacities of SIDS, LDCs, and CEITs (TOR 4F) 50 The Effects of the GEF Structures on Performance:The GEF Network 53 Findings (TORs 4A, 4D, and 4G) 53 Challenges and Strategic Trade-Offs 57 Recommendations 58 GEF Procedures 61 GEF Project Cycle (TOR 5A) 61 The Information Management Challenge for the GEF (TOR 5B) 64 References 67 Notes 69 Annex 1: Clarification of OPS3 Terms of Reference 71 Annex 2: Complete Terms of Reference 73 OPS3 Acronyms and Abbreviations 80 CD-ROM of full report inside back cover iv OPS3: ProgressingToward Environmental Results Foreword T he Global Environment Facility is replenished by Any impression that the GEF on its own would be able to donors every four years. Ideally, any replenishment solve global environmental problems needs to be qualified should be based on the achievements so far and the immediately. The world community currently spends approx- problems that need to be addressed in the coming years. The imately US$0.5 billion a year on solving these issues through fourth replenishment, which will be negotiated and agreed in the GEF. The problems are immense. Any solution would the second half of 2005, will be informed on the achievements need the strong involvement of many other actors. The of the GEF through the present "Overall Performance Study," amount of Green House Gases emissions continues to which is the third of its kind. It provides an overview of the re- increase. Extinction of animal and plant species continues. sults in dealing with global environmental problems and it also Pollution and waste treatment pose enormous challenges. looks at how the GEF functions as a network and partnership Access to safe water is not ensured and even endangered for of institutions and organizations. Given the fact that the GEF is many people. Land degradation is a huge problem in many the main financial mechanism for several global environmen- countries across the world. The only global environmental tal conventions, the report amounts to a review of what gov- problem that is almost solved is that of the elimination of ernments are doing to improve the global environment. It also ozone depleting substances. For all of these problems, the provides an indication of the status of some of the most im- GEF contribution needs to be seen in its proper perspective portant global environmental issues. as a catalyst or innovator rather than as the direct purveyor of international public goods. v My personal assessment of this study is that it provides a The tendering process was handled by the Operations solid basis for discussion and decisions on the fourth replen- Evaluation Department of the World Bank in July and August ishment of the GEF. The questions of the Terms of Reference of 2004 in a timely and professional manner. The tender was of the study have been addressed. It provides an authoritative won by ICF Consulting and its international partners. In Au- overview of the current state of knowledge in the GEF about gust and early September 2004 contract negotiations took its results. Furthermore, it gives a challenging picture of the place. Furthermore, in early September I started in my posi- GEF as a network of organizations and institutions. The report tion as Director of Monitoring and Evaluation. My arrival is consistent with the methodology presented in Inception meant that the monitoring and evaluation unit of the GEF and Interim Reports. The study draws on data gathering and Secretariat changed into the independent GEF Office of Moni- analysis based on literature review, evaluative evidence in the toring and Evaluation. As required by the Terms of Reference GEF (mainly from studies of the GEF Office of Monitoring and of OPS3, as Director of Monitoring and Evaluation I provided Evaluation), and extensive stakeholder consultations and oversight of the process, ensuring that the terms of reference country visits. were being followed. Furthermore, a High Level Advisory The current state of knowledge about results in the GEF is Panel was established as part of the technical backstopping, well presented, as well as shortcomings concerning these re- reporting directly to me and providing written comments on sults. Furthermore, the strategic choices that the GEF is facing all deliverables. in reaching (and maintaining) these results are identified. The The GEF Council in its session in November 2004 requested difficulties of sustaining results is highlighted and the catalytic me to work with the study team to ensure consistency and role of the GEF receives due attention. Last but not least the high quality in the field analyses to be undertaken by the study contains many recommendations and suggestions for team. To this end, further discussions were held with the increasing the results orientation of the GEF in the fourth re- study team on the composition of field teams and the prepa- plenishment period. ration of field visits. I participated fully in one field visit and re- It should be noted that OPS3 did not do an independent gional meeting to witness the team in operation. This led to a empirical assessment of the environmental results that were satisfactory conclusion on the preparedness and openness of achieved by the GEF. This was not possible given the time limi- the team concerned. tations of the study. To explain why this is the case, let me turn The primary way in which the time limitations were ad- back to the origin of OPS3. The GEF Council attached great im- dressed by the study team was through fielding a large team portance to the independence of the OPS3 team from GEF of mostly senior experts. This approach is sound in itself, but management, and devoted extra time and energy to ensure led to some unanticipated difficulties when it turned out that that this would be the case. In the first half of 2004, when no proper sequencing of efforts could take place. Ideally, the preparations for the Third Overall Performance Study started, desk review of evaluative evidence would have finished be- the monitoring and evaluation unit of the GEF Secretariat was fore the field visits and interactions with stakeholders took not yet fully independent. As a result, Council decided to take place. A more systematic agenda for checking the reality be- the final drafting of the terms of reference for the study in its hind literature and evaluative findings in field visits could have own hands. This took longer than initially expected, which been developed if there had been sufficient time. In reality, meant that the tendering process for the study started rela- the desk review and the field visits and consultations had to tively late in June 2004. The study started in September 2004, run in parallel. It seems to me that these difficulties raised con- which meant that the actual time available was reduced sig- cerns on first the quality of the field work and second on the nificantly, since it still had to be finalized in April 2005 in order (lack of) emergence of findings in early stages of the analysis. to feed into the replenishment process. This meant that given These concerns were raised by several Council members in the scope and range of the questions in the terms of refer- November 2004 (on the quality of field visits) and in February ence, no empirical data gathering on environmental results 2005 (on the lack of emerging findings on results of the GEF) was possible. and by the High Level Advisory Panel at several occasions. vi OPS3: ProgressingToward Environmental Results The field visits were logistically difficult to arrange. Often technically sound and prepared with high professional stan- dates and agendas had to be changed, sometimes at the last dards. The entire GEF Office of Monitoring and Evaluation was moment. In November, the study team promised to involve it- involved in the exercise and provided essential inputs through self at an adequate level (senior and mid-level participation) the preparation of studies of the three main GEF focal areas and that counterparts from developing countries would par- which constituted the basis for OPS3's assessments on results. ticipate. This was realized for most, but not all, of the work- The High Level Advisory Panel, chaired by Nancy McPherson, shops and field visits. The workload was increased when some provided critical comments that pushed the study team to im- workshops were added on the request of the Council (notably prove the quality of the analysis and final report. Cuba and Fiji), as well as an informal exchange with Council Council members also should be acknowledged for their members in Paris in February 2005. The regional workshops active participation from the preparation and approval of the were generally well attended. Terms of Reference to extensive comments on several of the Although these constraints and added milestones and products produced by the study team. Staff of the GEF Secre- meetings limited the time available even further, the process tariat, GEF Implementing Agencies and global convention has been managed by the team in a truly exemplary manner. Secretariats as well as many of the STAP members provided Many evaluation managers would have buckled under the many valuable hours of their time throughout the process. I pressure and have asked for a delayed delivery of the final would also like to acknowledge the very active and open con- product. The study team, under the leadership of Mark tribution made by GEF Focal Points and representatives of the Wagner, has not done this and has excelled in keeping the many NGOs from around the globe that participated in the whole process within the time limits set for it. Given the scale extensive consultation process conducted by the study team, and the scope, this is to be applauded. probably the most extensive one so far in the history of the During the study, the team did take the advice of the High GEF. Last, but not least, I would like to thank the national and Level Panel on board in various ways. Furthermore, the inter- local governments as well as the GEF project teams that actions of the team with the GEF Council and with GEF Coun- opened their doors to share their experiences during the visits cil members have helped to focus the study on the issues that conducted by the OPS3 teams. Although it is impossible to ac- are important for the replenishment process. curately portrait the extensive tapestry of GEF activities in a In many respects the Third Overall Performance Study was report like this one, the projects that were visited helped the a global effort and therefore there are many people from study team to recognize the richness and uniqueness of GEF around the globe that should be acknowledged in making it experience. possible. The ICF Consulting OPS3 team, led by Mark Wagner, and the GEF Office of Monitoring and Evaluation OPS3 sup- Rob D. van den Berg port team, led by Claudio Volonte, should be recognized first Director of Monitoring and Evaluation of all. Both of these teams ensured that the final report was Foreword vii Preface T he team for the Third Overall Performance Study of One of the key challenges for OPS3 was collecting and as- the GEF (OPS3), which took place between Septem- sessing results. While results are critical to project success and ber 2004 and June 2005, was charged with evaluat- aggregated results are essential to the evaluation process, this ing the 1) results of GEF activities, 2) sustainability of results information is not always available in the GEF system. This dif- at the country level, 3) GEF as a catalytic institution, 4) GEF ficulty is attributable to a range of factors, but most impor- policies, institutional structure and partnerships, and 5) GEF tantly to limited baseline data, inconsistency on what will be implementation processes. From the onset, our team viewed measured and how, a vast array of projects with differing the evaluation as an opportunity to evaluate the progress of goals, and nascent centralized data collection systems. All of GEF activities, and also to set the stage for future evaluations. these issues are overlaid by very high expectations for achiev- This perspective was essential to avoid the development of ing global environmental benefits. a static evaluation (a "snapshot") for a dynamic and evolving While there clearly has been progress in the GEF system institution such as the GEF. As such, OPS3 attempted to place and while all stakeholders are more informed, and processes all analyses, findings, and recommendations in the context of are better off, than they were 4 years ago when OPS2 took the future. In particular, we asked the question, "What infor- place, further attention is needed in certain core areas. In mation will OPS4 and future evaluations need to conduct particular, if the GEF is to be robust, there must be continued analysis, and how will having this information ensure the dialogue on baseline setting and, specifically, how to define success of the GEF?" baselines in the face of a moving target; for example, as addi- tional species are catalogued or abandoned stockpiles of POPs are uncovered. ix Additionally, measuring results relative to these often shift- and Evaluation, especially the newly appointed director ing baselines can be difficult, and while improvements are be- Robert van den Berg and his staff, notably Claudio Volonte, ing made in data collection, verification, and analysis, it is criti- was integral to the success of this study and made our work cal that simple procedures for gauging results be agreed more targeted than would have otherwise been possible. upon. Simple measures of results will be more practical for Their help in establishing a High Level Advisory Panel and in tracking progress than the use of complex, resource intensive orchestrating panel interactions also was instrumental. Fur- measurement schemes. To house this ever growing universe ther, the contributions of the High Level Advisory Panel itself of data, transparent, centralized data systems, accessible to all improved the quality of the evaluation. parties will be necessary to enable future evaluations. Across all of the groups with whom we interacted, there Collaboration is critical to success. Improving outcomes was a great commitment to the GEF and its mandate, a great and results will depend on furthering the emerging acknowl- enthusiasm for the work being undertaken, and eagerness to edgement of the GEF as a network--collaboration between demonstrate success. We hope that the recommendations the parts of various institutions that focus on the GEF. By real- put forward by OPS3 will be helpful in moving the GEF's agen- izing the advantages of this network arrangement, the effec- da forward in achieving global environmental benefits in a tiveness of the GEF can be improved, but it will take compro- sustainable way. mise and a willingness to work toward the utility of all--to Finally, I would like to personally acknowledge my col- sacrifice self interest for the overall good of the system. leagues at ICF Consulting and our regional partners for their During the field study portion of our evaluation, the OPS3 creativity, thoughtfulness, and dedication to this evaluation. team spoke to more than 600 GEF stakeholders from country governments, Implementing and Executing Agencies, NGOs, Mark C. Wagner GEF project managers, and representatives from the private OPS3 Team Leader sector and civil society, in addition to representatives of the and Senior Vice President GEF Council, the GEF Secretariat, and the Scientific and Techni- ICF Consulting cal Advisory Panel. The leadership of the Office of Monitoring x OPS3: ProgressingToward Environmental Results Third Overall Performance Study Teams Core OPS3 Team ICF Consulting Partners Mark Wagner, Team Leader ICF-EKO (Russian Federation) Christopher Durney Olga Varlamova Will Gibson Abyd Karmali Walter Palmer Polly Quick OPS3 Support Team ICF Consulting Partners Paula Aczel Africon (South Africa) Joana Chiavari Joseph Asamoah Chiara D'Amore Thomas van Viegen Craig Ebert Centre for Environmental Education (India) David Hathaway R. Gopichandran Alan Knight ICF-EKO (Russian Federation) Johanna Kollar Svetlana Golubeva Daniel Lieberman Mexican Institute of Water Technology (Mexico) Pamela Mathis Alberto Guitron Jeremy Scharfenberg Marian Martin Van Pelt Jessica Warren GEF Monitoring and Evaluation Office OPS3 Team Robert van den Berg, Director Claudio Volonte, Senior Monitoring and Evaluation Specialist Aaron Zazueta, Senior Monitoring and Evaluation Specialist Siv Tokle, Senior Monitoring and evaluation Specialist Juan Jose Portillo, Project Officer Joshua Brann, Junior Professional Associate OPS3 High Level Advisory Panel Professor Zhaoying Chen, Director, China's National Centre for Science and Technology Evaluation Dr. Lawrence Haddad, Director, Institute of Development Studies, University of Sussex, United Kingdom Dr. Alcira Kreimer, independent consultant Dr. Uma Lele, Sr. Adviser, Operations Evaluation Department, World Bank Ms. Nancy MacPherson, Senior Adviser, Performance Assessment, IUCN--The World Conservation Union xi Purpose and Scope T he purpose of the Third Overall Performance Study format for decision makers. The executive version focuses on (OPS3), commissioned by the Global Environment the main findings and major recommendations coming from Facility (GEF) Council, is "to assess the extent to OPS3 because, from the beginning of the OPS3 process, this which GEF has achieved, or is on its way toward achieving, its has been the subject of greatest interest to the GEF Council main objectives, as laid down in the GEF Instrument (GEF and the host of other interested parties. The OPS3 team heard 1994) and subsequent decisions by the GEF Council and the on many occasions that it was essential to articulate clearly Assembly, including key documents such as the Operational what the results of the GEF have been and to clearly point out Strategy and the Policy Recommendations agreed as part of not only strengths and weaknesses of the GEF, but also to the Third Replenishment of the GEF Trust Fund" (GEF/C.23/4 show the way forward by identifying actionable recommen- 2004). The scope of the OPS3 study is defined by the "Terms dations that can be readily implemented, and with continued of Reference for the Third Overall Performance Study of the attention, lead to improvements in GEF processes, systems, GEF (Annex 1)," approved by the GEF Council on May 21, and mechanisms for generating results. 2004, which covers five main themes: (1) results of GEF activi- The organization of the executive version parallels the ties; (2) sustainability of results at the country level; (3) the GEF full report with the exception that main findings and major as a catalytic institution; (4) GEF policies, institutional struc- recommendations are placed first. ture, and partnerships; and (5) GEF implementation processes. This executive version of the OPS3 report presents the voluminous material assembled by the study into a digestible 1 Main Findings O PS3 has identified findings in seven main areas: (1) has had a notable impact on slowing or reducing the loss of results in each of the focal areas, (2) strategic pro- biodiversity. Unfortunately, global trends in biodiversity loss gramming for results at the focal area level, (3) continue to be downward. The GEF has produced significant strategic programming for results at the country level, (4) re- outcomes in biodiversity conservation through protected ar- sponsiveness to conventions, (5) information management eas. Indeed, the GEF has been credited by many with helping and knowledge sharing, (6) network responsibilities and coor- to achieve the global goal of 10 percent of the world's land dination, and (7) the Small Grants Programme (SGP). area under protection. By the end of fiscal 2004, the GEF had supported protected area investments that constitute almost 17 percent of the total land area protected globally (Interna- Focal Area Results tional Union for the Conservation of Nature [IUCN] 2003). The GEF has also contributed to improving the enabling environ- The GEF has achieved significant results, particularly at the ments in which biodiversity conservation and sustainable use outcome level, in the focal areas of Biodiversity, Climate occur. The GEF has far exceeded the mid-term targets set in Change, International Waters, and Ozone Depletion, and is the Third Replenishment Agreement for the Biodiversity focal well placed to deliver important results in the newer focal ar- area (GEF/R.3/38. 2000). However, outcomes related to access eas of Land Degradation and Persistent Organic Pollutants and benefit sharing arising from the use of genetic resources, (POPs). the third objective of the Convention on Biological Diversity The OPS3 team believes that the GEF Biodiversity Pro- (CBD), have been less robust (though less guidance has been gram, as likely the world's largest government-funded mech- issued to the GEF on this issue). anism for biodiversity conservation in developing countries, 3 In the Climate Change focal area, although the GEF's role In the Land Degradation and POPs focal areas, the OPS3 is relatively minor in slowing worldwide climate change, the team finds that there are signs of health; in particular, these GEF portfolio has satisfactorily performed (given its limited focal area programs seem poised to learn from the experi- resources), exceeding its interim greenhouse gas (GHG) emis- ences of the other, more mature focal areas, although it is pre- sion reduction targets set by the Third Replenishment Agree- mature to assess the likelihood of results generation. In the ment in an increasingly cost-effective manner. Additionally, POPs focal area, there has been significant progress in imple- the GEF has played an important catalytic role in developing menting convention guidance through the funding of nation- and transforming the markets for energy and mobility in de- al implementation plans (NIPs) in more than 100 countries, veloping countries, particularly through its energy efficiency and it is likely that the relatively straightforward approach to portfolio. Market transformation results in the renewable en- chemicals management will allow for a clear results chain, ergy cluster have been more varied, although some good re- particularly if the proper steps are taken up front to identify sults have been identified. human health and environmental baselines. The GEF's International Waters (IW) Program has achieved some stress reduction impacts, particularly in the Black Sea­Danube and Lake Victoria. Because only a few Interna- Strategic Programming for Results-- tional Waters projects have entered a Strategic Action Pro- Focal Area Level gramme (SAP) implementation phase, however, it is too early to report on impacts in terms of environmental improvement. OPS2 recommended that the GEF shift from an approvals fo- In general, the outcomes of the International Waters Program cus to a results and quality orientation. In general, the OPS3 have been robust and are expected to result in stress reduc- team has observed good steps in this direction, and signifi- tion and environmental improvement impacts over time. The cant results have been achieved, but much remains to be International Waters Program has exceeded its mid-term per- done to focus on and manage results. In particular, clarifying formance target set by the Third Replenishment Agreement. and improving the coherence of strategic direction in each of The program has supported the negotiation and implementa- the focal areas is an important step toward more effective tion of a number of global and regional conventions; has been programming for results at the focal area level, as well as to- an effective agent for policy, legal, and institutional reforms; ward developing and tracking meaningful indicators for re- and has served as an example of the benefits of systematic sults. identification and incorporation of lessons learned through the International Waters Learning Exchange and Resource Improving Coherence of Strategic Guidance Network (IW:LEARN). Better cooperation at the regional level, more coherence in strategic partnerships, and stronger on- Strategic guidance in the GEF has been mixed--abundant in the-ground management and supervision are needed, howev- some areas but notably absent from others. For example, in er, to improve results. 2003, additional strategic direction was issued in the form of In the Ozone Depletion focal area, the GEF has essentially Strategic Priorities for each focal area as part of a general achieved its main objective--to eliminate the consumption Strategic Business Planning Framework. Although these (that is, production, exports, and imports) and emissions of Strategic Priorities have been helpful for some focal areas, ozone-depleting substances (ODSs) in countries with they constitute additive strategic guidance and an additional economies in transition (CEITs), with more than 99 percent of review screen during project approval. Indeed, the Strategic the agreed phaseout having been accomplished. Moving for- Priorities have resulted in a broadening, rather than a refining, ward, the GEF Secretariat (GEFSEC) should coordinate with of the overall strategic focus of the focal area programs. In ad- the Multilateral Fund of the Montreal Protocol (MLF) Secretari- dition to direction issued by the GEF, guidance from some at regarding the status of hydrochlorofluorocarbon (HCFC) conventions, in particular the CBD, has proliferated without and methyl bromide phaseouts. any prioritization. Thus, to some extent, rather than better aligning the goals of the GEF, this proliferation of guidance appears to have defined a sufficiently vast area that GEF enti- ties may find whatever direction they seek in it. In other areas, 4 OPS3: ProgressingToward Environmental Results such as for calculating incremental costs, guidance has not been sufficient (for instance, in the POPs focal area), and stakeholders find the issued guidance difficult to understand and implement. As a result, in many cases, only specialized consultants brought in specifically to develop the project de- sign documents are able to perform the arcane calculations. This simultaneous proliferation and lack of guidance has, in part, resulted in focal area programs that do not have strate- gic focus and coherence. This lack of strategic focus and co- herence in the more established focal areas, including Biodi- versity, Climate Change, and International Waters, is discussed in more detail below. Biodiversity. The development of the GEF-3 (FY2004­06) Strategic Priorities (and those proposed for GEF-4 [FY2007­10]) has brought increased strategic direction to the GEF Biodiversity Program; and the development of impact and coverage indicators and targets, as well as the tools to measure them, should improve management of the portfolio. leading to action. Additionally, the strategic objectives pro- Nevertheless, the OPS3 team agrees with the Biodiversity Pro- posed for GEF-4 have been reformulated using the model for gram Study 2004 (BPS2004) that the Biodiversity Program still market development presented by the CCPS2004, and they needs to refine, clarify, and strengthen the overall strategy have been fit into the established OP framework, providing and vision of the program. Furthermore, OPS3 finds that the more clarity. OPS3 finds, however, that the Climate Change development of Strategic Priorities has led to additive guid- Program would benefit from a clarification of its role with re- ance and has broadened, not streamlined, the overall strate- spect to carbon finance initiatives, and by providing more dis- gic focus of the GEF Biodiversity Program. Consequently, not tinct guidance on the role of adaptation in its portfolio. only is the interplay between Operational Programs (OPs) and Strategic Priorities not sufficiently clear to the operational- to International Waters. The strategic programming challenges country-level participants, but projects that address a wide for the International Waters (IW) focal area differs from those range of biodiversity outcomes can be funded through the faced by Biodiversity and Climate Change. The GEF Interna- GEF, making aggregation of results difficult. tional Waters Program has achieved significant success at the foundational or capacity-building level. To date, the Interna- Climate Change. The OPS2 recommendation for the Climate tional Waters focal area has primarily been a mechanism for Change Program--that the GEF would benefit from a more fo- catalyzing action by gathering information, conducting analy- cused program in climate change--does not appear to have ses of transboundary concerns, building capacity to work been fully achieved during GEF-3. The Climate Change Pro- jointly, identifying needed reforms and investments in action gram Study 2004 (CCPS2004) found that "the linkages be- programs, and leveraging funds to implement the programs. tween GEF's overall mission or goals, its strategic priorities, The new challenge for the GEF International Waters Program, OPs, project clusters, and performance measurement indica- which the International Waters Strategic Priorities have identi- tors are no longer conceptually clear, nor are they entirely con- fied, is to push beyond the shorter-term goals of OPs 8 and 9 sistent." OPS3 also found a lack of clarity regarding the links (water body­based and integrated land and water multiple between GEF strategic directions reported at several stake- focal area OPs, respectively), to longer-term financial mobi- holder levels, including Implementing Agencies (IAs). Howev- lization and realization of demonstration projects necessary er, recent progress on the part of the GEFSEC in response to is- under OP10 (contaminant-based OP). sues raised in the CCPS2004 has shown that dialogue is Main Findings 5 Tracking Indicators the CCPS2004 pointed out, although projects can be in line with national priorities, the current system for project devel- Finally, an important part of clarifying the strategic direction opment and approval has led to inconsistent focus within in the focal areas is developing meaningful and user-friendly some countries, such as India and Mexico, where the GEF is indicators for results at the output, outcome, and impact lev- not addressing the major climate change needs of the coun- els that can be aggregated to report on the results of the focal try. The OPS3 team also heard reports from GEFSEC and GEF area programs overall. Critical questions concerning what to Office of Monitoring and Evaluation (OME) representatives, measure, how to measure, and how to scale up project results country focal points, and nongovernmental organizations to the program level are still not resolved. The recent develop- (NGOs) of somewhat duplicative projects in some coun- ment of targets and indicators in the focal area strategic ob- tries--an issue that could be resolved through managed jectives for GEF-4, as well as the tools to measure them, will country portfolios. By contrast, success has generally been likely improve the management of the focal area portfolios, achieved in China, where the World Bank and the United Na- but the existing indicators do not allow for easy aggregation tions Development Programme (UNDP) collaborated early to of benefits at the program level, particularly in Biodiversity. develop the overall climate change portfolio. The CCPS2004 This reality presents a serious challenge to the evaluator in- set forth, and OPS3 agrees, that countries with significant tent on amassing the results of the GEF. The Ozone Depletion GEF portfolios would benefit from a simple, but integrated, focal area--a model for results in the GEF system--stands as a country program, and that countries with smaller portfolios success primarily because of the systems for agreeing on and may not need a full-blown program but would benefit from measuring results that were established under the Montreal explicitly articulated priorities. In addition to promoting Protocol and recorded by the Ozone Secretariat. Developing country ownership and country drivenness, country pro- appropriate monitoring and evaluation (M&E) tools in all focal grams could also optimize GEF resources by enabling better areas is an urgent task. synergies and multiple benefits through programming of a strategic portfolio for each country, rather than approving projects in a more piecemeal manner. Strategic Programming for Results-- In short, OPS3 finds that the type of programmatic ap- Country Level proach needed is one that (1) targets cross-focal area syner- gies, (2) prioritizes country projects, (3) explicitly considers In addition to strategic coherence within focal area programs, global environmental benefits, and (4) sharpens the focus on which can be thought of as the vertical portfolio strategy, the sustainability and catalytic effects. GEF also needs strategic coherence at the national, or hori- Looking for synergies across focal areas, such as benefits zontal, level. These two dimensions of strategic direction in- and capacity sharing, is essential for maximizing results and teract with each other and form a natural feedback loop, such leads to increased cost effectiveness--and it can be facilitated that, ideally, national priorities are developed with an eye to within existing structures through a country program ap- GEF strategy in each focal area, and GEF strategies are devel- proach. Recognizing this, the GEF-4 Programming Document oped taking into consideration the activities recipient coun- proposed that "the GEF move towards more integrated ap- tries really need and want. proaches to the national resource management challenges In fact, GEF projects are often developed in a more ad hoc that span the global environmental agreements. Pursuing in- and sometimes opportunistic manner, rather than systemati- tegration across focal areas, at the various levels--basin, land- cally to contribute to an overall country strategy. Conse- scape, ecosystem, country, and region--will allow the GEF to quently, because coherent portfolios are not always devel- fulfill its role as catalyst and facilitator of global environmental oped for countries, results may not always be maximized or sustainability" (GEF/R.4/7 2005). achieved in the most cost-effective manner. For example, as 6 OPS3: ProgressingToward Environmental Results Stakeholders at all levels consulted during OPS3 suggested OPS3 finds that recipient countries would benefit from joint, that a more country-oriented programmatic approach to coordinated GEF country programs that bring many actors funds disbursement would improve strategic alignment. Ac- (and the outcomes of other initiatives) to the table in a collab- tivities such as the National Capacity Self-Assessment (NCSA) orative, egalitarian exercise. and the National Consultative Dialogue Initiative help coun- Finally, recipient countries would also benefit from a tries identify and develop national environmental priorities, sharper focus on sustainability and catalytic effects among but these priorities, GEF priorities, and the projects actually GEF entities. The multidimensional and dynamic nature of developed for countries are not always aligned. How national sustainability is not systematically addressed in GEF projects, priorities are linked to the projects submitted by many coun- as is apparent in project documentation prepared during the tries is sometimes unclear and may be partially based on op- design, implementation, and evaluation phases. Likewise, the portunistic access to available funds (instead of national prior- mechanisms for sharing information and systematically pro- ities). Indeed, in the event that a Resource Allocation moting the replication of successful innovations, demonstra- Framework (RAF)1 is approved, the GEF will likely have to allo- tions, and approaches are conducted on a relatively ad hoc cate resources among countries in a systematic manner. In basis within the GEF network. Moreover, no systematic re- this context, developing and managing national strategic porting on indicators for catalytic effects is in place across all portfolios for results would maximize results with the re- GEF focal areas, although a tracking tool for measuring main- sources allocated to each country. Under any RAF approach, streaming in the Biodiversity focal area has recently been es- however, it will be necessary not only to program at the coun- tablished. A tighter framework for conceptualizing, measur- try level, but also to prioritize projects for the country at the ing, and tracking the sustainability and catalytic effects of portfolio level. A process for choosing among projects based GEF projects would allow the GEF to better understand the on certain characteristics (for example, innovativeness, replic- extent of its success and areas of weakness at the portfolio ability, cost effectiveness)--which may vary significantly de- and country levels. This in turn could help prioritize resource pending on the country, focal area, or project size--has, to allocations within an RAF (if approved), as well as within date, not been explicitly included in the RAF. In particular, al- countries themselves. though the proposed RAF indicators look at governance and environmental performance at the country level, there is no discrimination between projects. Clearly, however, there are Responsiveness to Conventions factors that affect performance and attractiveness at the proj- ect level. For example, protected area projects in the Biodiver- In general, OPS3 finds that the GEF has been responsive to sity focal area are common and can be developed based on a guidance from the CBD, the United Nations Framework Con- history of approvals for similar projects. In fact, it may be easi- vention on Climate Change (UNFCCC), the Montreal Protocol, er to move such a project through the pipeline than to create the United Nations Convention to Combat Desertification a more innovative, but potentially riskier, project that may in (UNCCD), and the Stockholm Convention. the long run generate greater benefits. These trade-offs · Biodiversity and CBD. In the Biodiversity focal area, OPS3 should be considered and reflected in criteria for choosing finds, as did OPS2, the Second CBD Review of the GEF, and projects. BPS2004, that the GEF has been generally responsive to Although the IAs have their own systems for programming Conference of the Parties (COP) guidance. The GEF has activities in countries (for example, the World Bank has its funded activities in almost all of the areas of guidance pro- Country Assistance Program, and UNDP has its Country Pro- vided by the COP. In particular, as BPS2004 points out, the gramme Action Plan and also identifies regional priorities in GEF has been particularly responsive to guidance on forest business plans), these programs do not necessarily explicitly consider global benefits in the manner that the GEF does. Main Findings 7 ecosystems and capacity building in biosafety. The GEF tion priority of the Convention, the GEF has begun to re- faces some challenges, however, in addressing COP guid- spond by approving an adaptation Strategic Priority for ance. In particular, OPS3 finds that the GEF has not ade- GEF-3 and proposing one for GEF-4, although the GEF still quately addressed the convention priority on Access and has much to sort out in terms of its funding of adaptation Benefit Sharing (ABS), although this is partly due to the activities. current lack of clarity on ABS in the context of the CBD. · Ozone Depletion and the Montreal Protocol. In the Ozone · Climate Change and UNFCCC. OPS3 also finds--as did Depletion focal area, the GEF has essentially achieved its OPS2, the 2002 COP 8 review of the GEF, and CCPS2004-- role in the main objective of the Montreal Protocol--to that the GEF has effectively performed its role as financial eliminate the consumption and emissions of ODS--and mechanism of the UNFCCC and has been responsive to its has been responsive to strategic guidance from the Meet- mandate as defined by the Convention and guidance and ing of the Parties to the Montreal Protocol (MOP). priorities as given by the COP. GEF funding of projects has · Land Degradation and UNCCD. The GEF has generally ad- been in direct response to the priorities outlined by the dressed the UNCCD global priorities with two exceptions: COP. Moreover, communication and coordination be- (1) The UNCCD set a priority for combating desertification tween the UNFCCC and the GEFSEC has improved over the in Africa, whereas the GEF Land Degradation focal area past few years. The GEF has been particularly responsive in strives for geographic balance; (2) the UNCCD focuses on quickly mobilizing and implementing special trust funds, combating desertification, whereas GEF projects tackle all as requested by the COP. The GEF has been responsive in causes of land degradation, including those that occur in supporting countries' first rounds of national communica- humid areas. tions, (NCs), and the second round provides an opportuni- ty to identify country priorities. With respect to the adapta- 8 OPS3: ProgressingToward Environmental Results · POPs and the Stockholm Convention. In POPs, all global such as IW:LEARN and UNDP knowledge management servic- priorities mentioned in the Stockholm Convention are ad- es, OPS3 was not able to identify any systematic, comprehen- dressed in the GEF strategy, with the exception of the po- sive, GEF-wide approach to ensuring that lessons learned are tential need to identify and regulate the production of new captured and disseminated properly throughout the network. chemicals with POPs characteristics. There are also differ- This conclusion was supported by a broad consensus at every ences in the emphases placed on priorities in the Conven- level of the GEF partnership. Recent focal area program stud- tion versus those articulated in Persistent Organic Pollu- ies also highlighted the inadequacy of current processes for tants (OP14) and the POPs Business Plan: the GEF places capturing lessons learned. Given that there has not been an greater emphasis on capacity building and institutional adequate systematic process for capturing lessons learned strengthening, the need for innovative and cost-effective over time, there is a real risk that substantial lessons learned, technologies for the disposal of POPs, and the aim of pro- capacity, and institutional knowledge among individuals will moting synergies by integrating POPs management prac- be lost if they are not captured and recorded. Although posi- tices with other focal areas. tive discussion on how to enhance KM in the GEF is under way, more remains to be done. Mechanisms for communication between the GEFSEC and MIS in the GEF have also been lacking since its inception. the Convention Secretariats exist, and dialogue takes place Each of the GEF entities maintains its own database, but cur- regularly, but it is not always easy to engage on certain issues. rently no comprehensive and integrated MIS captures infor- For instance, although guidance from the COPs is not always mation systematically and makes that information regularly sufficiently prioritized, the Convention Secretariats are hesi- available to GEF partners, which makes accurate monitoring tant to interpret guidance issued from the Convention COPs. of GEF activities at the portfolio level very difficult. The Project Through consultations with GEFSEC representatives, the OPS3 Tracking and Management Information System (PMIS) main- team has also learned that there is some awkwardness re- tained by the GEFSEC is inadequate to meet the management garding what has been construed as "guidance back to the and monitoring needs of the GEF. The lack of transparency conventions." There are often circumstances, however, threatens the GEF's partnership with recipient countries by wherein the GEF entities, through implementation experi- not empowering them to stay actively involved in tracking ence, have relevant perspectives on what is working, what their projects through the project cycle. The current inability could be improved or clarified, and what might benefit from a of the GEF to monitor its portfolio at a macro level inhibits fresh approach. Indeed, more frank and timely exchange of strategic vision, though the GEFSEC is slated to undertake ideas between the GEFSEC and the conventions could be monitoring at the portfolio level. The lack of MIS also greatly helpful in furthering the agenda and success of the conven- inhibits the ability of the evaluator to report on results in the tions within the context of the GEF. focal area programs and in other areas, such as actual cofi- nancing. The GEF's ability to demonstrate success in the Ozone Depletion focal area, where the GEF can rely on the Information Management within the Ozone Secretariat's systems for tallying results, underscores GEF Network the need for more robust data systems. A comprehensive, reli- able, and harmonized MIS could allow OPS4 to confidently re- GEF systems for information management, which encompass port on the results of the GEF and the GEF's progress in meet- knowledge management (KM), management information sys- ing its operational principles. tems (MIS), and infrastructure are inadequate. Although OPS3 identified some components of a system for learning lessons, Main Findings 9 Network Responsibilities and Administration FY1995­98 (GEF-1), the apparent ability of the GEF coordina- tion mechanism to absorb an increase in coordination and OPS3 finds that the GEF, based on its composition, structure, communications channels resulting from the addition of two and division of roles and responsibilities, is a network organi- focal areas and seven Executing Agencies (EAs) with expand- zation, which is different from a stand-alone hierarchical or- ed opportunities could suggest either a maturing economy of ganization.2 A network is an emerging form of organization in scale or a positive efficiency outcome. Although the GEFSEC which independent or at least semi-autonomous entities has absorbed these expansions to some degree, its effective- work together to achieve a common result. OPS3 finds that ness and ability to implement a comprehensive, GEF-wide this network structure is the appropriate institutional form to coordination strategy, rather than individual coordination enable the GEF to meet its mandate and operations. Indeed, efforts, will be compromised as the GEF continues to expand. the literature strongly supports the assertion that organiza- Without additional support in the form of staff and resources, tions that undertake complex and geographically dispersed it is unclear whether the growing responsibilities of the net- challenges, are composed of multiple independent entities work administrative office can be accommodated. With addi- that have some claim on the mission, and require flexibility tional resources, and as the network administrative office and responsiveness most effectively operate as a network. function matures, however, the GEFSEC will be better able to provide key central coordination services that will help to fully integrate GEF partners, such as NGOs, EAs, the Scientific and Network Administrative Office Technical Advisory Panel (STAP), and the private sector. The literature supports OPS3's contention that complex net- works such as the GEF require a network administrative office Competition versus Collaboration to administer, guide, and coordinate network activities. The GEFSEC has worked consistently to manage the increasingly OPS3 found that roles and responsibilities were not always complex GEF network and to serve the network administra- clear for IAs and EAs, especially with regard to collaboration tive office function for the GEF. The activities undertaken and competition. IAs are aware of their stated comparative include implementing GEF Council and Assembly decisions; advantage, but there were a number of projects for which it preparing criteria, standards, priorities, and business plans; was not possible for OPS3 to discern from the characteristics and coordinating various activities and partners, including of the project why a particular IA was the implementer of Inter-Agency Task Forces, Executive Coordinator meetings, record. The OPS3 team also heard testimony from the IAs maintaining the PMIS, coordinating the Council member and themselves and other stakeholders that competition for proj- focal point support programs and National Dialogue Initia- ects and resources was forcing IAs to look ever wider for proj- tives (NDIs), undertaking dialogue with the Convention Secre- ects and investigate new lines of business to support their tariats, and performing general communication and outreach sustained growth, even when those projects crossed over into activities. These tasks are in addition to the day-to-day activi- the comparative advantage of one of the other IAs. This ten- ties of managing the project pipeline, engaging in the ap- dency to blur the boundaries of the IAs' roles is further exacer- proval process, and performing oversight responsibilities. The bated by the addition of the EAs that must find their way GEFSEC already has undertaken some organizational changes within the GEF project context. EAs have an uncertain man- during GEF-3 to facilitate this administration, including estab- date and a large learning curve to climb in order to function lishing a group to manage corporate and operational issues of competitively in the GEF "market." In fact, only four of the sev- the GEF. en EAs with expanded opportunities have signed a memoran- However, without adequate resources, the GEFSEC will not dum of understanding (MOU) with the GEF that officially sanc- be able to continue functioning effectively as the network ad- tions their ability to implement projects solely. EAs are the ministrative office of the GEF. Given that the Secretariat's cor- lead on only 38 of the more than 1,500 projects implemented porate budget has remained steady as an overall percentage by the GEF, which further underscores the nascent aspect of of the programming budget since the restructured GEF their involvement and speaks to the competitive playing field and dwindling funds under GEF-3. 10 OPS3: ProgressingToward Environmental Results While competition is, in some cases, straining the trust having responsibility for agricultural aspects. The current among corporate entities, collaboration among project pro- competitive climate has resulted in projects being developed ponents, including IAs and EAs, is being fostered by the GEF with a sole agency as the implementer. Additionally, under a as a means to improved functioning (and cost effectiveness) full-competition scenario, IAs may be less willing to fulfill and is specified in the "Instrument for the Establishment of their GEF corporate responsibilities. the Restructured Global Environment Facility" (GEF 1994). In discussions with the GEFSEC and the IAs, it was clearly stated Scientific and Technical Advisory Panel (STAP) to the OPS3 team that unlimited competition will be at odds with collaboration. For example, joint project implementa- Despite the recent efforts of the STAP to refine and focus its tion and the associated fee sharing imply collaboration, but work, and coordinate more closely with GEF entities, stake- competition implies developing and implementing wholly holders generally believe that the panel is still not nearly as re- owned projects with a single manager claiming the entire sponsive as it needs to be and is not able to provide consis- fee. On their own, the IAs will not likely be able to solve effec- tent value to the GEF. For instance, stakeholders at the tively the equation between competition and collaboration GEFSEC, the IAs, and within the STAP itself asserted that STAP on projects. In the POPs focal area, for example, it was envi- reports are not always relevant to the GEF and are not always sioned that the comparative advantages of the IAs and EAs provided to GEF entities in a timely enough way to be useful. would be brought to bear jointly in NIP development, with The current process for requesting STAP reports is circuitous, the United Nations Industrial Development Organization and the reports, when completed, may no longer be relevant. supporting activities with an industrial component and the Also, although the STAP roster is seen as a success in building Food and Agriculture Organization of the United Nations Main Findings 11 scientific capacity within the GEF system, the selective use of supports the idea of engineering quality into the M&E system the STAP roster is still not perceived as objective by project through the validation of IA M&E systems. Indeed, the consul- proponents, GEFSEC staff, and STAP members. In particular, tative process can be seen as a positive step toward develop- because project managers at the IAs are able to choose the ing an M&E "community of practice" throughout the network. roster expert who reviews their project, there is the appear- Through the consultative process, OME, the GEFSEC, and ance of a conflict of interest. IAs and EAs are also coming to agreements on how to cover Moreover, despite the efforts of STAP leadership to do so, M&E at many levels; to date, however, monitoring at the net- the STAP has not been able to sufficiently reach out to the sci- work level has not been addressed. As a partnership network, entific and research community for selected technical input as the GEF reacts in complex, inter-reliant ways to changes in its its mandate clearly prescribes, nor has it used the linkages own rules, such as the adoption of an RAF, changes in the IA with other scientists, a hallmark of the academic community, fee system, or separation of M&E functions. These systemwide to leverage its own resources. This leads to a conundrum be- impacts must be monitored by OME to ensure that such mod- cause STAP members frequently do not have sufficient time ifications of rules or procedures are not having unexpected to dedicate to their STAP work, but more networking with the negative effects on the functioning of the network. larger academic community could potentially alleviate this problem by leveraging additional experts. Finally, OPS3 finds Private Sector that positive progress is being made: a STAP retreat in Quito, Ecuador, is planned to discuss these issues with GEF partners, In recent years, GEF entities have explored the development including the IAs, OME, and the GEFSEC. of a more targeted approach to engage the private sector, in- cluding the preparation of a May 2004 OME report, "Review of GEF's Engagement with the Private Sector" (GEF/C.23/Inf.4 Monitoring and Evaluation (M&E) 2004). In response, GEF management requested the GEFSEC Some partners within the GEF system are in fact parts of part- to better articulate a private sector strategy, in collaboration ners. This can result in overlapping and competing proce- with the IAs and EAs, and in consultation with private sector dures. For example, evaluations performed as part of the re- stakeholders. Discussion is ongoing, but a clear, focused GEF quirements for an institution's own evaluation procedures strategy for engaging the private sector is still lacking. The de- may overlap with (but not fully supplant) GEF requirements velopment of such a GEF strategy ultimately may require diffi- for evaluation. Additionally, because IA evaluation systems cult decisions about the extent to which the GEF is prepared historically have not been validated by OME, there was some to reach out to industry and reconcile the differences in doing inefficiency related to evaluating evaluations. OPS3 finds that business--which include disparate drivers (profit versus envi- one of the most notable signs of recent success has been the ronment) and different, sometimes incompatible, modes of new leadership of, and strategic actions undertaken by, the operation and timeframes for action. In part as a result of this independent OME. The consultative process sponsored by absence of coherent strategy, the GEF has missed opportuni- OME is evidence of growing harmonization of goals and ties for potentially increasing catalytic effects through GEF processes across the GEF, but there are remaining tensions projects involving the private sector. OPS3 supports the GEF-4 and obstacles to overcome. OPS3 encourages the continua- Programming Document in its assessment that strengthened tion of this process, especially given the broad stakeholder in- engagement with the private sector should be a major ele- volvement approach that OME has chosen. In particular, OPS3 ment of the GEF-4 management agenda. 12 OPS3: ProgressingToward Environmental Results Small Grants Programme (SGP) The "Third Independent Evaluation of the GEF Small Grants Programme" (Wells et al. 2003) noted that "in many countries SGP has become the permanent public face or even de facto ambassador of the GEF." OPS3 also found that the SGP is well received by recipient countries and increases the visibility of the GEF. Indeed, the SGP remains one of the most appreciat- ed programs of the GEF, and many representatives of coun- tries, especially NGOs, that are not currently recipients of the SGP expressed to the OPS3 team that they wanted to be. Many recipient country stakeholders, including government representatives, NGOs, and project proponents, as well as in- country IA representatives, noted how effectively the SGP was responding to country priorities at the local level. The evaluation noted that "one of the most striking findings... is the high degree of fit between the services and benefits pro- vided by the SGP and the current priorities and needs in an extraordinary variety of country contexts in which the pro- gram operates." OPS3 found that the flexibility of the SGP has allowed for innovative thinking and design of activities to meet country needs and capacities in small island developing states (SIDS) and least developed countries (LDCs). Although many SIDS are only now gaining access to the SGP, they are optimistic about the impacts it will bring and feel strongly that wider access will lead to cost-effective strategies for ad- dressing focal area needs. The 2003 SGP evaluation also found that "the overall long- term global benefits from SGP activities will be considerable, SIDS with very limited capacities--and more in line with their and are likely to exceed the global benefits generated by capacity to absorb funds. The 2003 SGP evaluation also con- most larger projects with financial resources comparable to or cluded that the SGP's participatory approach to project devel- even exceeding the entire SGP budget." Although the OPS3 opment and implementation is very favorable to project sus- team has not itself aggregated the benefits associated with tainability. SGP activities, it finds this conclusion meritorious. The OPS3 team also heard anecdotal evidence from sever- Additionally, OPS3 concurs with BPS2004, which found evi- al groups of stakeholders, including IA country office repre- dence suggesting that smaller-sized projects may hold more sentatives and other project proponents, that SGP projects promise in achieving sustainability,3 perhaps because of their are more replicable than larger projects because their lower more targeted focus and limited objectives, or because of the cost makes them easier to adopt in other places. The 2003 more transparent, participatory, and country-driven approach SGP evaluation found that many SGP projects leveraged their to planning that characterizes SGP projects. Stakeholders at impact through scaling up, replication, and influencing gov- all levels and across multiple countries interviewed as part of ernment policies during GEF-2 (FY1999­2002). Without a ro- the OPS3 field study voiced very strong support for the SGP, bust set of data on replication, however, no conclusions can citing very high likelihood of sustainability due to their being be drawn on this issue by OPS3. more manageable and accessible--especially for LDCs and Main Findings 13 Major Recommendations T he major recommendations suggested by OPS3 are results of projects unless there is a strategic-level decision on based on the main findings discussed in the previous what should be classified as results, what measures are appro- section. The recommendations provided here are priate, what levels of expectations are appropriate, what the those that the OPS3 team viewed as most significant; within priorities are, and so on. the main report, there are additional recommendations that GEF guidance has been incremental and additive and, gen- have been identified and elaborated upon. Some of these erally, no direction has been provided on approaches for have been aggregated into major recommendations, if, for streamlining outdated guidance. The GEF Council will need to example, they cut across focal areas or across Terms of Refer- collaborate with the GEFSEC and IAs to determine how ence (TOR) areas. Others are minor or procedural in nature, streamlining should be accomplished (for example, through and are not "major" recommendations. Please refer to the elimination of guidance, harmonization of reports, and so on). main report for a discussion of all recommendations, organ- ized by TOR. The major recommendations are summarized in exhibit 1. Programming for Results--Focal Area Level As presented in exhibit 1, and throughout the text that fol- lows, a conceptual point is necessary to consider when inter- · Clarify strategic direction. The strategic direction and co- preting and/or implementing these recommendations. That herence of each focal area program should be clarified and is, there are strategic as well as operational recommendations, improved. In particular, some reformulation of the GEF's and although the operational aspect of the recommendation programming framework and priorities should be under- is how the way forward may be put into practice, it is critical taken to increase transparency and effectiveness of the that these be considered in light of the more strategic aspects programs. In the Biodiversity, Climate Change, and Interna- of the recommendation. For example, it is not sufficient to put tional Waters focal areas, the definitions of the OPs and the in place a detailed system for managing information on the manner in which they contribute to achieving impacts 15 Exhibit 1. Summary of Major Recommendations Topics Recommendation(s) Programming for results--focal area level · Clarify strategic direction · Define impacts Programming for results--country level · Cultivate a stronger country program focus · Incorporate RAF concepts in ranking projects at the country level · Track sustainability and catalytic effects Responsiveness to conventions · Strengthen two-way communication between the GEFSEC and Convention Secretariats Information management within the GEF network · Establish a formal information management function Network responsibilities and administration · Strengthen the role of the GEFSEC as the network administrative office · Clarify roles and responsibilities for all GEF partners,especially IAs and EAs · Clarify and strengthen the role of the STAP · Foster M&E at all levels · Launch a private sector initiative SGP · Allocate additional resources to the SGP should be clarified, and the relationship between OPs and · Define impacts. Given the difficulties experienced by OPS3 Strategic Priorities should be clarified. In the Biodiversity in measuring program impacts, it is apparent that more Program, the "Christmas tree effect" can be counteracted pragmatic project impact definitions are needed. In order by better describing the strategic vision and direction for to measure the results of the GEF, and to evaluate whether the Program. In the Climate Change focal area, clarification the GEF is optimally programming to achieve results, indi- of the way in which the long-term goal of market transfor- cators should continue to be developed and refined in all mation outcomes contributes to GHG emissions reduction focal areas to allow aggregation of results at the country or avoidance would increase transparency of the Program. and program levels--for instance, across the Strategic Pri- Also, while the strategic direction of the climate change orities. To cost-effectively deal with this daunting issue, the portfolio has shifted over time (for example, moving away GEF must rely on the efforts of others in the area of indica- from solar photovoltaics [PV] projects), this direction tor development, when possible. Collaborative efforts and should be better articulated to provide more program co- coordination of activities are the strengths that the GEF hesiveness. In addition to clarifying the OPs, the Interna- can leverage to ensure progress in this area. Finally, to fa- tional Waters Program should move from enabling activi- cilitate the aggregation of results, GEF partners should be ties to scaling-up of full operations to address agreed more proactive about ensuring that project proponents priorities for global critical transboundary water systems. In understand how to report on results and should be stricter the relatively new focal areas of Land Degradation and about the quality of project-level M&E. (TOR 1A, 1B, 1C) POPs, moving beyond enabling activities to implementa- tion should be undertaken. The Inter-Agency Task Forces should take up the matter of improving strategic direction and coherence at the program level. (TOR 1A, 1C) 16 OPS3: ProgressingToward Environmental Results Programming for Results--Country Level · Track sustainability and catalytic effects. Operational def- initions and indicators are needed for sustainability and · Cultivate a stronger country program focus. In countries the mechanisms of catalytic effects (for example, cofinanc- with robust GEF portfolios, the GEF should move toward a ing, leveraged resources, replication, and mainstreaming) stronger country program focus on local capacity, on part- to sharpen the focus on these goals. In particular, project nership in the GEF process, as well as on planning and de- design, implementation, and evaluation should explicitly velopment of clear country strategies and priorities for GEF consider sustainability and catalytic effects, and more sys- funding. Country programs should be developed within tematically report on these issues in project documents to existing structures, as an outgrowth of and in concert with allow for the tracking of the GEF's success. For example, in activities such as NCSAs and NDIs, and should be planned the project implementation reviews (PIRs), assessments of by a multistakeholder team coordinated by the GEFSEC all relevant factors of sustainability (that is, political and lo- and including IAs and EAs, national focal points, and other cal will, finances, design, and management) should be ex- local stakeholders. In this role, GEF partner agencies should plicitly included within the context of sustainability. While ensure that bottom-up requests in programming exercises PIRs currently report on the level of financing received to are reconciled with the GEF's global strategic objectives. date, they do not report on the levels of financing secured Additionally, country portfolio planning teams should pay for the future, or on efforts undertaken to secure next- attention to include local decision makers at the right lev- phase financing, which is important from a sustainability els in order to give the programs adequate weight and standpoint. A focus on sustainability would require that credibility in country. Special consideration must be paid stakeholder participation be reported on in terms of how to indigenous populations, allowing them to play a lead attitudes and behaviors have changed, and not simply on role in programs design and implementation. Also, ade- the number of workshops or meetings held. The OPS3 quate attention should be paid to focal area interlinkages Team recommends that the GEF establish a dedicated to optimize benefits both at the country and regional lev- team to explore indicators for sustainability and catalysis els. In countries with smaller GEF portfolios (such as LDCs for use in project documents; conduct systematic ex post and SIDS), an alternative strategy should be considered. Fi- monitoring of random samples of GEF projects; and track nally, to better understand the GEF portfolio performance and compile information on likely and actual sustainability at the country level, OME evaluation of selected country and catalytic effects that can be aggregated at the portfo- portfolios is recommended. The outcomes of such evalua- lio level. Additionally, information generated from these tions would not only indicate performance at the country processes must be shared within the GEF network to cat- level, but also could serve as valuable input to future pro- alyze additional global environmental benefits. (See also gramming at the country level. (TOR 1D, 2A, 2B, 2C, 4E) the recommendation on "Information Management within the GEF Network.") (TOR 2A, 2B) · Incorporate RAF concepts in ranking projects at the coun- try level. With regard to the proposed RAF, the GEF should continue to develop hierarchies of priorities and incorpo- Responsiveness to Conventions rate important concepts into any eventual RAF scoring sys- tem. For example, geographic balance and the relationship · Strengthen two-way communication between GEFSEC between global and local benefits (such as poverty allevia- and Convention Secretariats. Robust, collaborative, and tion) are important factors that can lead to success. Project regular two-way communications between the GEFSEC success factors and a weighting for innovation, as exam- and the Convention Secretariats should be further fostered ples, should be included in a scheme to rank projects with- to enable dialogue on priority setting, streamlining of in a country program (or for picking among projects for strategies, and institutional capacity sharing. In particular, countries with similar RAF ratings). This notion, which is dialogue should also be pursued between the GEFSEC and not currently incorporated into current conceptions of the RAF, emphasizes the need for measuring benefits at the country portfolio level. (TOR 1E, 4E, 5A) Major Recommendations 17 UNEP.net, UNDP's GEF portal, and the World Bank knowl- edge management system). In addition, in pushing for- ward KM and lessons learned, the GEF should make sure that adequate time is given to both the capture and, even more important, the dissemination and delivery of that in- formation to its appropriate targets. As a first step in the improvement process, the GEFSEC should mount a fo- cused effort with the IAs to update data that already exist in the current systems so that the latest and most accurate data are available. At the same time, the GEFSEC should begin the development of an information management strategy that will guide the long-term improvement and overall quality of the KM and MIS systems, and their sup- porting business processes. (TOR 5B) Network Responsibilities and Administration · Strengthen the role of the GEFSEC as the network admin- Secretariats of the UNCCD and Stockholm Convention to istrative office. The GEFSEC, as the network administrative monitor the observed differences between the Convention office, should administer and coordinate network activities and the way that GEF programs intend to implement the in a more comprehensive and strategic way. The GEF focal areas. This dialogue should also serve to clarify out- Council should adjust resources allotted to the Secretariat, standing issues such as guidance on how to calculate in- as necessary, recognizing that this function is critical to ef- cremental costs associated with POPs activities. These in- fectiveness and bears a cost. To that end, the GEFSEC, as teractions should be formally structured to ensure a the network administrative office, should consider formal- transparent and effective process. (TOR 4C) izing the following organizational functions: · Communication, coordination, and outreach--cover- Information Management within the ing communication with all the GEF partners in relation GEF Network to capacity and coordination, including country partner capacity; communication and outreach; coordination · Establish a formal information management function. To and outreach with other partners, including NGOs and address the current inadequacy of both the KM and MIS the private sector; and external entity outreach. functions in the GEF, OPS3 recommends that the GEF es- · Management, information, and policy--encompass- tablish a formal function for information management in ing the following functions: implementation of Council the GEFSEC (please see the recommendation on formaliz- and Assembly decisions, policy and planning, work plan ing the GEFSEC's function as a network administrative of- programming, information management strategies and fice) in the section on "GEF Procedures." This function systems, knowledge management and communities of would be responsible for KM and MIS with the bottom-line practice coordination, and project cycle management. goal of transparency at all levels of the GEF partnership (TOR 4A, 4D) network. The GEF should give this function appropriate time and resources, making it pragmatic by building on ex- isting KM and MIS systems in the GEF entities (such as 18 OPS3: ProgressingToward Environmental Results · Clarify roles and responsibilities for all GEF partners, es- · Foster M&E at all levels. OME should further foster collab- pecially IAs and EAs. Roles and responsibilities for all part- oration by institutionalizing the consultative process to ners must be clear, and outreach and collaboration must create a community of practice of M&E in the GEF, coordi- be encouraged. In particular, the GEFSEC needs to work nating with IAs and EAs on the science of evaluation, build- with the IAs and EAs to clarify roles and responsibilities and ing trust to foster harmonization and streamlining, and al- work through the challenge of competition and collabora- locating responsibility at the appropriate level. OME tion--an issue that has the potential to seriously affect the should also begin to monitor the health and the effective- quality of GEF results during GEF-4 if it is not managed ef- ness of the GEF partnership network itself, paying particu- fectively and proactively. Because there are already disin- lar attention to the ripple effects of changes in GEF proce- centives to collaborate, including competition for re- dures and rules, such as the employment of an RAF. (TOR sources and projects, and there is still poor transparency 4G) and less-than-full trust in the system, it is essential that the · Launch a private sector initiative. The GEF should launch GEFSEC take more of a leadership role in enunciating the a private sector special initiative to look for good models of positioning of collaboration and competition in the sys- cooperation with the private sector and to pilot projects. tem. OPS3 recommends that an ongoing dialogue be- Specifically, OPS3 recommends that the GEFSEC, in coordi- tween the GEFSEC, IAs, and EAs be undertaken to voice is- nation with the IAs and EAs, work directly with members of sues on the advantages and disadvantages of, and ways to the private sector to identify appropriate means and optimize, the competition versus collaboration nexus. This modalities to more effectively involve the private sector. dialogue could, for example, be in the form of a regularly Private sector representatives should be identified and se- scheduled workshop or contact group that convenes prior lected based on their previous involvement with the GEF, to GEF Council meetings. (TOR 4A, 4D) so that a blueprint that is sensitive to the needs and reali- · Clarify and strengthen the role of the STAP. The role of ties of industry can be formulated during a series of work the STAP must be better articulated and the relationship sessions scheduled throughout the year. The GEF should with the outside scientific community strengthened and aim to design a proposal for private sector engagement realigned. Positioning and accessibility must be conducive that includes a strategy for private sector outreach and to early and effective involvement. STAP should also coor- communication, as well as risk-sharing arrangements. In dinate more closely with the scientific bodies of the con- addition, the work sessions should address additional staff ventions (for example, the Intergovernmental Panel on Cli- expertise or resources that may be required within the mate Change [IPCC] and the POPs Review Committee), GEFSEC to actively engage the private sector in moving for- being careful not to overlap, duplicate, or supersede the ward, such as the potential development of a new staff po- mandates of those bodies. In addition, STAP should feed sition to identify, market, and facilitate new opportunities lessons learned, best practices, and science-based advice for private sector leveraging and partnerships. (TOR 3A) into the knowledge management system. To implement these recommendations and enable the STAP to provide relevant reviews in a timely manner, structural changes Small Grants Programme may be in order. One possibility is that the STAP could be streamlined to include only one member per focal area, · Allocate additional resources to the SGP. Building on the plus a chair, and that all members could give a higher per- findings of the Third Independent Evaluation of the SGP, centage of their time (for instance, 50 percent or greater) OPS3 recommends that additional resources be allocated to increase commitment and availability. STAP activities to the SGP and that the Land Degradation and POPs focal could be coordinated through these seven panel mem- areas, and that the adaptation Strategic Priority under the bers, who would draw on their networks with the greater Climate Change Portfolio be integrated into the program. scientific community, as well as on more junior scientists Because the need for the SGP has been particularly noted and consultants who have the time to undertake such ac- in LDCs and SIDS, where the need for adaptation funding tivities. (TOR 4A) is also particularly strong, the inclusion of the adaptation Strategic Priority is especially appropriate. (TOR 1A, 1C, 4F, 5A) Major Recommendations 19 Approach and Methodology T o support the development of findings and recom- OPS3 Role in Results Assessment mendations for OPS3, it was necessary to design a During initial consultations with OME, discussions were held robust approach and methodology. on how OPS3 would address the results assessment issue giv- en the objectives of the study, other major recent analyses that had contributed to the study (for instance, Program Stud- Evaluation Challenges and Strategies ies), and various other constraints such as the general unavail- ability of impact-level results data and the study timeframe. In addressing the various areas of the OPS3 TOR, there were Three consensus points emerged from these discussions: several distinct challenges and requirements that contributed · OPS3 should focus on assessing overall results of the GEF to the OPS3 approach. These are outlined below. at the focal area level, based on available data synthesized in reports such as the recent Program Studies, data gath- Results of GEF Activities ered through a series of country visits to assess results ob- served at the country level, and other available summary Given the increasing maturity level of certain GEF portfolios, data. and in the context of recent dialogue on the results of the GEF, there was a clear focus on assessing results as part of the · The recently completed Program Studies for Biodiversity, OPS3 study (TOR 1). In addition, this is an area where OPS1 Climate Change, and International Waters would serve as and OPS2 were not able to provide any comprehensive as- one of the primary existing sources of detailed data con- sessment, and where expectations for OPS3 were high. cerning specific results and related issues at the project and focal area level. Consideration of the Program Studies as part of the OPS3 assessment was supported by the GEF Council in the November 2004 summary meeting docu- mentation. 21 · The research conducted by OPS3 during the desk and field large gap in the effectiveness of such project-level mecha- study components would look to provide an overview of nisms in capturing results at the impact level. Apart from this GEF activities, and would not try to corroborate data at the constraint, there were no mechanisms in place to support the project level. Instead, OPS3 would use information collect- roll-up of impacts should they be identified. In summary, ed in the field to corroborate findings from the Program OPS3 was presented with a situation where basic questions Studies, OPS1 and OPS2, and the rest of the desk study. concerning what to measure, how to measure, and how to scale up results to the program level were not resolvable, and Key Challenges to Results Measurement results did not exist in a form conducive to clear aggregation. After conducting an initial desk review, it was clear that TOR1 Taken as a whole, these observations at the outset of OPS3 in- would be problematic. In particular, there would be problems dicated that results measurement within the GEF, in particular relating to reporting at the level of long-term quantifiable re- at the impact level, and in terms of global environmental ben- sults or impacts (global environmental benefits). This difficulty efits, remained a key challenge for the GEF, and would pose a had been reported by OPS2, and was similarly raised in the challenge for the OPS3 team. 2004 Program Studies, which also indicated that more recent projects have made progress in including baselines and indi- Recognizing Nonquantifiable Results cators. However, the results of these newer projects will not A subtheme to the challenges surrounding development of a be seen for several years. These observations by OPS3, in ad- more practical results measurement framework is how to dition to the scientific literature, pointed to problems such as treat nonquantifiable results of GEF activities. OPS3 was asked the following: to assess both quantitative and qualitative results. OPS3 rec- ognized that there is currently no agreed methodology avail- · Most projects do not generate information at the level of able in many focal areas to support the quantification and ag- long-term quantifiable impacts and, more important, gregation of qualitative or "soft" outcomes, although there many projects still do not have clear and agreed baselines, have been recent advances made in some focal areas (such as indicators of impacts, or methodologies to calculate them. Biodiversity) that may assist the GEF in aggregating such out- · Environmental change may take decades to be perceived comes in the future. or measured, while GEF projects on average span four or Studies point out certain outcomes that are either inher- five years. ently resistant to quantification, or that pose serious difficul- ties to quantification. Participatory stakeholder consultations · The GEF does not systematically conduct postcompletion under OPS3 were a major source of evidence to support this studies to look at long-term results. analysis. OPS3 identified many project-level outcomes that · The GEF, as an institution, does not have an overall results project participants recognized as nonquantifiable yet signifi- measurement framework or methodology to aggregate cant, and that do not link easily to upward aggregation. from project-level impacts to program-level or GEF-wide impacts. There is no unified framework in place for system- Sustainability and Catalytic Effects of the GEF atically defining, measuring, and aggregating results of GEF activities, particularly in terms of global environmental The key challenge in assessing issues relating to sustainability benefits for each of the GEF focal areas. and catalytic effects of the GEF was gaining input from key GEF stakeholders at all levels. The OPS3 desk study process OPS3 observed that while mechanisms appeared to be in provided some input, though this was limited. Project-level place to guide development of goals and results during proj- reporting data do not lend themselves to facilitating this as- ect design, implementation, and reporting (for example, proj- sessment since the data are not sufficiently dynamic in pres- ect log-frames), and individual projects have been assessed entation to articulate GEF project performance relating to against their implementation performance as part of various cross-cutting factors such as sustainability and catalytic effects annual, mid-term, and completion reporting, there remains a of GEF activities. Other sources such as the 2004 Program Studies were more useful. 22 OPS3: ProgressingToward Environmental Results The key methodological element of the OPS3 assessment Elements of the OPS3 Approach of these issues was the extensive series of stakeholder consul- tations undertaken over the course of the study. These consul- The principal components of the OPS3 methodology were as tations allowed OPS3 to probe the experience and observa- follows: tions of each set of stakeholders to arrive at a fuller picture of · Research agendas for each TOR. These were developed to the key questions underlying TORs 2 and 3, such as the key ele- guide both initial fact finding during the desk study com- ments contributing to sustainability of GEF activities, how ef- ponent and stakeholder consultations during the field fective leveraging occurs on the ground, how the GEF is cat- study component. alytic, and conversely how the GEF's processes or procedures may limit the GEF's effectiveness in achieving sustainability of · A desk study that included a review of key documents de- its efforts, or in maximizing its capacity to catalyze efforts on veloped to coincide with OPS3, and other documentation, the ground. The consultative process assisted OPS3 in opening in preparation for the field study component. dynamic and iterative lines of inquiry to support its analysis. · Extensive participatory stakeholder consultations. Efforts were made to consult all major stakeholders within the GEF Structure and Processes GEF family, to bring the full range of perspectives to bear on key areas of the TOR. Consultations were conducted on Recognizing that assessments of institutional structures, both an individual and group basis, and involved more processes, and effectiveness often require (and are very con- than 600 GEF stakeholders, including country focal points ducive to) the use of an overarching analysis framework, OPS3 and other government officials, NGOs, project managers, developed a two-part institutional framework to guide its as- GEF Council members, GEF Trustee members, GEF Secre- sessment of TORs 4 and 5. tariat, OME staff, outside academic representatives, IA and For its investigative framework OPS3 developed a Frame- EA staff, and staff from the CBD, UNFCCC, the Montreal work for Institutional Expectations Analysis, which provided a Protocol, UNCCD, and the Stockholm Convention, as well set of underlying performance expectations that OPS3 would as others. In many cases, OPS3 was able to triangulate find- expect to find in practice based on the GEF Instrument, previ- ings based on the feedback received from different types ous guidance and assessment, and the particular organiza- of stakeholders. The iterative process of the stakeholder tional traits and operating context of the GEF institution. For consultations assisted OPS3 in evolving an understanding each area of TORs 4 and 5 the institutional analysis compares of these issues, and provided OPS3 with the ability to de- these ideal expectations to actual performance of the GEF en- velop dynamic lines of inquiry over the course of the study. tities and the overall institution. The interpretive framework developed by OPS3 drew To develop key findings, evidence was reviewed from desk conclusions about the form of the GEF institution to identify a and field studies. Evidence from key studies such as the 2004 theoretical framework of the key challenges faced by such in- Program Studies was considered as a significant baseline; stitutions, and linked these theoretical challenges to specific however, OPS3 sought to corroborate such evidence through challenges observed with the GEF institution. This framework its field study component (participatory stakeholder consulta- assisted OPS3 in assessing results of its research in the context tions). Assessing the significance and validity of the evidence of the GEF institution's particular challenges, and communi- collected by the OPS3 field study required a separate process. cating its conclusions on the GEF structure and processes OPS3 determined significance and validity by assessing each based on those challenges. piece of evidence against specific criteria. Evidence was con- sidered valid and significant if it met two out of the following four criteria: (1) evidence corroborated the desk study; (2) evi- dence was supported within a stakeholder group; (3) evidence was supported across multiple stakeholder groups; and (4) ev- idence was supported across multiple field visits. Approach and Methodology 23 Focal Area Analysis Portfolio Analysis (TOR 1E) Excluding global and regional projects (which accounted for US$819 million), the GEF has funded 1,531 projects in 151 In general, the GEF has achieved significant results, particular- countries and regions since 1991, totaling US$4,431.6 million. ly at the outcome level, in the focal areas of Biodiversity, Cli- As shown by exhibit 4, particular countries have received sig- mate Change, International Waters, and Ozone Depletion. nificant shares of total funding and project numbers; indeed, From 1991 through March 2005, the GEF approved US$5.25 the top 20 recipients of those 151 (13 percent of countries) billion to over 160 countries to achieve global environmental represent 59 percent of total funding. benefits across the six focal areas, as shown in exhibit 2. Of this total figure, approximately US$1.7 billion (35 percent) has funded multicountry projects.4 Biodiversity (TORs 1A and 1B) The Biodiversity and Climate Change focal areas together account for the overwhelming majority of the GEF's project Findings on Impacts and Outcomes portfolio in terms of funding, representing 70 percent of over- The GEF Biodiversity Program supports the implementation of all GEF funds committed 1991 through March 2005. While the three objectives of the CBD: the conservation of biological funding for the GEF as a whole has increased over time, the diversity, the sustainable use of its components, and the fair share of total GEF funds attributed to the various focal areas and equitable sharing of benefits arising from of the utiliza- has remained relatively constant across the GEF periods. Exhib- tion of genetic resources. At the impact level, OPS3 believes it 3 presents GEF funding and number of projects by region that the program, as likely the world's largest government- since the inception of the GEF. As shown, project sizes are not funded mechanism for biodiversity conservation in develop- necessarily consistent across regions. For instance, while Africa ing countries, has had a notable impact on slowing or reduc- has received US$1,181.8 million, or 23 percent of the total ing the loss of biodiversity, although global trends in funds, 513 projects (31 percent of the worldwide total) have been completed or approved in Africa. 25 Exhibit 2.Total GEF Funding biodiversity continue to be downwards. At the outcome level, (1991 through March 2005) the GEF has produced significant achievements in biodiversity conservation through protected areas and in sustainable use Focal area U.S.dollars (millions) Percentage of biological resources. With regard to protected areas, by the Biodiversity $1,906.3 36% end of FY2004, the GEF had supported investments in 1,426 Climate Change $1,747.4 33% protected areas, covering nearly 269 million hectares International Waters $768.3 15% (GEF/R.4/7 2005), almost 17 percent of the total land area pro- Multifocal areas $457.9 9% tected globally (IUCN 2003). Indeed, as reported in BPS2004 Ozone Depletion $177.2 3% POPs $ 121.3 2% and also found by OPS3, the GEF has been credited by many Land Degradation $ 72.2 1% with helping to achieve the global goal of 10 percent of the Total $ 5,250.5 100% world's land area under protection. The GEF has also far ex- ceeded the mid-term targets set in the Third Replenishment Source: GEF Project Management Information System,accessed March 2005. Agreement for the Biodiversity focal area. Even though the GEF was given no direct guidance on pro- tected areas from the CBD until COP 7 (February 2004), pro- tected areas have featured prominently in the GEF portfolio. Exhibit 3. GEF Funding and Number of Between FY1991 and FY2003, approximately 75 percent of Projects by Region (1991 through the projects in the GEF biodiversity portfolio supported activi- March 2005) ties related to protected areas.5 These projects have resulted in the scaling-up of protected-area approaches to larger land- scape levels, such as in the Meso-American Corridor; strength- $1,600 ened legal and policy frameworks, such as in the Brazil Nation- al Biodiversity project; the establishment of innovative financing mechanisms to support the recurrent cost of pro- $1,200 tected areas; and some successful management and planning initiatives, such as the Cape Peninsula project in South Africa, (Millions) $800 expected to set international standards in the subject. With a few exceptions, BPS2004 found that few projects reported Dollars US success in implementing draft management plans. Moreover, $400 BPS2004 recommended that "despite GEF's very significant fi- nancial and technical contribution towards expanding the world's protected areas and protected area networks and en- $0 hancing their management, the GEF has yet to conduct a Asia Africa Caribbean entral Asia Global Regional study that looks at the aggregate contribution of local, proj- and andC ect, or site-level outcomes and impacts in [protected areas] to America Europe the GEF's overall contribution to higher level, global biodiver- Latin sity impacts" (GEF/ME/C.24/Inf.1 2004). Noting that the OPS3 team also found it impossible to aggregate to the level of im- Source: GEF Project Managment Information System,Accessed March 2005 pacts, OPS3 strongly endorses this recommendation. 26 OPS3: ProgressingToward Environmental Results In terms of the second objective of the CBD, the sustain- Despite these successes in achieving outcomes related to able use of biological resources, OPS3 found that several proj- the first two objectives of the CBD, outcomes related to the ects have demonstrated that the generation of alternative in- third, ABS arising from the use of genetic resources, have come is a good alternative for local populations. One example been more modest. BPS2004 found that few projects reported is the wildlife ranching project in Burkina Faso. However, on this topic, achievements or otherwise. During its field BPS2004 found that several projects reported activities not study, OPS3 also found that the GEF has not adequately ad- producing enough income, resulting in an increased demand dressed this objective. BPS2004 concluded that part of the for the targeted resource. Preparation and implementation of reason that more significant outcomes have not been natural resource management plans, such as in the Jozani achieved in this area is the current lack of clarity on ABS in the Chwaka Bay National Park project in Tanzania, is another im- context of the CBD. To this end, BPS2004 concluded that once portant outcome of the GEF Biodiversity Program. BPS2004 the COP negotiates and puts in place an ABS regime, the GEF recognized the opportunity described and then recommend- Biodiversity Program will be better situated to appropriately ed: "To improve chances of success, the operationalization of direct its resources. The GEF Biodiversity Task Force has start- the Addis Ababa Principles should encourage partnerships be- ed to discuss the possible impacts of these negotiations. tween GEF and other actors, particularly the private sector, at Outcomes of the GEF Biodiversity Program have also been all levels..." OPS3 endorses this action as an important means identified in the following areas: creating enabling environ- to increasing the outcomes in the field of sustainable use. ments and mainstreaming biodiversity (detailed below), inva- sive alien species, taxonomy, agrobiodiversity, and the SGP. Exhibit 4. GEF Funding and Number of Projects by Country, Excluding Global and Regional Projects (1991 through March 2005) $600 $500 44 $400 (Millions) $300 Dollars US 24 $200 21 20 22 23 $100 17 15 18 10 25 15 20 14 15 12 19 12 15 13 $0 .of o m China xico occ Peru tina Me India Brazil ation Africa Lanka Feder Rep Poland Ecuador Bulgaria Tunisia PhilippinesIndonesia ab Mor UgandaVietna Colombia Sri ,Ar Argen South Russian Egypt Source: GEF Project Managment Information System,Accessed March 2005 Focal Area Analysis 27 · Enabling environment--BPS2004 found that the majority of GEF-financed projects included components to improve the enabling environment for conservation and sustain- able use of biodiversity. Some of these achievements in- cluded (1) creating and implementing national priorities or legislative action, an accomplishment that more than half of the projects reported, although some setbacks were also reported; (2) generating public awareness and im- proving environmental education, an accomplishment that about two-thirds of the projects reported and that OPS3 also found during field visits (for example, the Sa- bana-Camaguey Ecosystem project in Cuba); (3) creating partnerships that broaden the catalytic effects of the GEF, an achievement that over half of the projects reported and that OPS3 also found; (4) knowledge generation, including at the level of environmental science and practice, and knowledge sharing, an achievement that about half of the projects reported; and (5) tool and technology develop- ment, in which about 40 percent of projects also reported achievements. · Mainstreaming biodiversity--The GEF designated this objective as one of its Strategic Priorities in GEF-3, and has proposed it as a GEF-4 strategic objective. During its field visits, OPS3 found instances of countries working hard to mainstream biodiversity, but the GEF has not been entirely successful in this regard. To reduce operational complica- tions in implementing the mainstreaming Strategic Priori- in the GEF-4 Programming Document constitute an improve- ty, BPS2004 recommended that "guidelines and clear defi- ment of the Strategic Priorities, and impact and coverage indi- nitions... be developed to clarify exactly what type of cators and targets, as well as the tools to measure them, which activities, processes, and interventions are covered under should improve management of the portfolio. These advances the mainstreaming concept in the GEF context." Having will provide future program evaluators with better tools with found country focal points, NGOs, and IAs discussing main- which to measure results. streaming in many ways and contexts, OPS3 agrees with Nevertheless, OPS3 agrees with BPS2004 that the Biodiver- this recommendation. sity Program still needs to refine, clarify, and strengthen the overall strategy and vision of the program, above and beyond Challenges and Strategic Trade-Offs the four Strategic Priorities. This is also related to the expecta- tions management issue discussed below. Furthermore, OPS3 Strategic direction and programming. OPS3 finds that the finds that the development of Strategic Priorities has served development of the Strategic Priorities for GEF-3 brought in- as additive guidance and has resulted in a broadening, rather creased strategic direction to the GEF Biodiversity Program than refining, of the overall strategic focus. Consequently, during GEF-3. Moreover, in part as a result of recommenda- there is confusion among stakeholders about the role of tions proposed by BPS2004, the strategic objectives identified Operational Programs and Strategic Priorities. In addition, projects that address a wide range of biodiversity outcomes can be funded through the GEF as a result of the broadening of strategic focus. 28 OPS3: ProgressingToward Environmental Results Strategic resource allocation. In the event that an RAF is ap- and can only be, one of many contributors to the achieve- proved by the GEF Council, it will have significant ramifica- ment of global environmental benefits; in Biodiversity as well tions on the GEF Biodiversity Program. Decisions on the allo- as the other focal areas. This reality seems to have been un- cation of resources to specific countries will need to be derstated in the GEF vision. cognizant of important nuances with regard to the develop- ment of biodiversity priorities. Though prioritizing countries Trade-offs in project outcomes. An implicit expectation of self-proclaimed as "megadiverse" has not been a stated policy the GEF that is directly related to its operating environment is of the GEF Biodiversity Program, these countries have re- that Biodiversity (and other focal area) projects should result ceived a large percentage of GEF resources for biodiversity in win-win situations; OPS3 verified this through stakeholder conservation. GEF funding also has a particular value in coun- consultations at all levels. However, biodiversity protection tries that are not of interest for bilateral funding from devel- and restoration competes with other factors that public and oped countries. Numerous representatives from countries private sector organizations consider when planning and im- that are not considered to be biodiversity "hotspots" reported plementing development activities, including market pres- to OPS3 that the GEF funding they received was largely re- sures and local poverty. As a result, as noted in the OP12 Pro- sponsible for enabling their country to focus on conservation. gram Study, there are trade-offs to biodiversity conservation that impede the GEF's capacity to achieve win-win situations. Measuring results. BPS2004 found that "though more work is Some projects, such as the Costa Rican Ecomarkets project, needed on the socioeconomic side, the new generation of have successfully managed these trade-offs. recently approved projects demonstrates progress in ensur- ing that important data are captured at the project level" and Recommendations recommended that "the establishment of baselines... be con- sidered mandatory... particularly to ensure that both biodi- In addition to the priority recommendations and recommen- versity and socioeconomic impact indicators are developed, dations on outcomes from BPS2004 that OPS3 has endorsed measured, and analyzed at all levels, from outputs to out- in the sections above, OPS3 recommends the following: comes to impacts." However, BPS2004 also found that at a · In the Biodiversity focal area, a balance needs to be struck be- higher level "there are still no clear guidelines, standardized tween even distribution of resources from the geographical or procedures, or measurable program-level targets or indicators regional standpoint and the likelihood of generating the to assess the impacts of the GEF portfolio on biodiversity sta- greatest global environmental benefits. While geographic ho- tus." Indeed, efforts by OPS3 to identify the global impact of mogeneity is not essential, targeting a few hotspots would not the GEF on biodiversity loss were not fruitful. At the outcome be in keeping with the GEF's objective for inclusivity and bal- level, the application of portfolio-level tracking tools devel- ance. Of course, generating global environmental benefits is oped to monitor and measure progress within each Strategic essential and must be linked to the project selection and prior- Priority for GEF-3 better enables the "rolling-up" of indicators itization process. National priorities and the implications that from the project level to the portfolio level. aspects of these priorities (such as poverty alleviation and al- ternative livelihoods) place on the generation of benefits must Expectations management. BPS2004 found that since the also be considered. In light of these issues, OPS3 recommends inception of the GEF there has not been a clear articulation of that these matters be considered in ongoing discussions the "expectations of the GEF or the level at which the GEF's about the implications of a potential RAF (please also see the performance--overall and at the three focal areas--would recommendation regarding strategic project selection under be assessed." As did BPS2004, OPS3 found the expectations an RAF scenario in the "Programming for Results--Country of the GEF Council, the Parties, and other stakeholders re- Level" subsection of "Major Recommendations"). garding the potential accomplishments of the GEF Biodiversi- ty Program to be unclear. BPS2004 concluded that the GEF is, Focal Area Analysis 29 Climate Change (TORs 1A and 1B) The experience of the renewable energy cluster is more varied, as the GEF is often trying to develop markets from a Findings on GEF Results much lower baseline. Renewable energy remains, in general, more expensive and less accessible than traditional fossil CCPS2004 found that the performance of the GEF portfolio in fuel­based energy sources, despite sustained efforts at vol- reducing GHG emissions was satisfactory, and OPS3 concurs. ume increases and market aggregation. CCPS2004 reported As reported by CCPS2004, direct and indirect6 reductions in that the GEF has been able to contribute to emerging market GHG emissions attributed to the GEF from closed, and pro- changes in specific energy sectors and countries; OPS3 also jected emission reduction targets in active, climate change found good examples of market transformation in renewable projects (1991 through April 2004) total about 1.9 billion met- energy, such as a wind power project in Russia. Regarding the ric tons (MT). The incremental cost effectiveness of GHG emis- advancement of new energy technologies, success has been sion reductions by the GEF has also improved over time limited; projects have proven difficult to design and imple- (based on GEF allocations alone). The GEF has met its mid- ment, in large part because key questions must be answered term performance targets as set by the Third Replenishment regarding trade-offs between innovation with higher risk ver- agreement. sus mainstreaming with lower risk (see also "Challenges and The intended GHG impacts vary widely across the Climate Strategic Trade-Offs" in the "Catalytic Impacts" subsection of Change Program's clusters, investment levels, country typolo- "The GEF as a Catalyst"). gy, and across individual projects. For instance, among active projects, the World Bank China Efficient Industrial Boilers proj- ect represents almost 40 percent of projected carbon dioxide Challenges and Strategic Trade-Offs (CO2) reduction targets. This is not that surprising given the Strategic direction and programming. While OPS2 concluded different scales and categories of project types that inherently that the GEF would benefit from a more focused program in have differing abilities to deliver GHG emission reduction. the Climate Change focal area, this does not appear to have Project approvals in FY2003­04 are expected to result in been fully achieved in GEF-3. CCPS2004 found that "the link- direct emission reductions of approximately 181 million MT ages between GEF's overall mission or goals, its strategic prior- and indirect reductions of about 409 million MT of carbon ities, OPs, project clusters, and performance measurement in- equivalent, which represent roughly 2 percent and almost 5 dicators are no longer conceptually clear, nor are they entirely percent of the 9 billion MT for global emissions in 2000, re- consistent." OPS3 also found a lack of clarity regarding the spectively (World Resources Institute 2004). Thus, while OPS3 links between GEF strategic directions reported at several finds the GEF's impact satisfactory given its limited resources, stakeholder levels, including IAs. CCPS2004 recommended, the GEF's role is relatively minor in slowing climate change. and OPS3 concurs, that GEFSEC take the lead to improve over- The organization can, however, play an important catalytic all strategic coherence by clarifying the goal of market trans- role in influencing, developing, and transforming the markets formation outcomes contributing to GHG reduction impacts, for energy and mobility in developing countries so that over and how the OPs and underlying strategies augment this goal. the long-term their economies are less carbon-intensive than they would have otherwise been. Strategic resource allocation. CCPS2004 found that the cur- Market transformation is a long-term challenge and a dy- rent project development system does not favor strategic namic process, and according to CCPS2004 and OPS3 findings choice, a condition that has resulted in a relatively dispersed it is starting to become evident in the GEF Climate Change portfolio and cases of missed opportunities in terms of poten- Program. CCPS2004 found that the greatest progress has tial impact. The study recommended that the GEF "improve been made within the energy efficiency portfolio; during its strategic choice and resource allocation... in order to ensure field study, OPS3 also found achievements that substantiate that the bulk of the portfolio is directed toward mitigation ef- this general finding, including electrical energy efficiency in forts in countries with relatively higher levels of GHG emis- Thailand. Indeed, for many evolving markets, the GEF is seen sions and market transformation potential." OPS3 agrees and as a driving force to help move changes forward. 30 OPS3: ProgressingToward Environmental Results finds that prioritization of the high-emitters should continue Lesson learning and replication. CCPS2004 concluded that by considering the relative availability of national funding and "learning within the GEF family has been neither systematic the specific opportunities for the GEF to add value by focus- nor system-wide, nor has it had strong outreach to outside ex- ing on specific market transformation projects. While the GEF pertise." While CCPS2004 found examples of good knowl- should pay careful attention to ensure that its portfolio is fo- edge-sharing initiatives within IAs and at the headquarters cused on achieving maximum impact, the GEF should main- level within the Climate Change Task Force, it suggested that tain the flexibility to assess individual countries' enabling en- better learning was needed between projects within the same vironments and determine whether to fund activities. clusters and within and between countries. As did CCPS2004, OPS3 also concurs with CCPS2004 that, to develop a more OPS3 finds that the GEF's approach to learning and replica- coherent GEF strategy for those countries that are likely to re- tion is not effective given the size of the portfolio and the ceive significant levels of funding, the GEF should consider valuable insights generated at the project level. As a positive developing country strategies to identify sectoral and project sign of progress, discussions of a pilot knowledge manage- priorities. Additionally, while the first NCs have tended to fo- ment initiative in the climate change program have been on- cus only on helping countries meet their reporting obliga- going among the GEF entities. tions under the UNFCCC and have generally not resulted in projects that can be taken forward through the GEF, the sec- Adaptation. Country focal points and NGOs in several region- ond NCs represent a significant opportunity for countries to al workshops, particularly the Pacific region, suggested to develop a national strategy that includes consideration of mit- OPS3 that the GEF must fund activities in the area of adapta- igation and adaptation elements. Indeed, OPS3 interviews tion to climate change since this goal is in the guidance from with IAs suggested that a higher level of country ownership UNFCCC and the mitigation of GHG emissions, owing to their and better stakeholder communication would result from the small volume, is not a high national priority. Stakeholders in revised approach that is being taken for the second NCs. these regions also noted, however, that adaptation will be a complicated new program area since adaptation issues are Measuring results. Although it has improved over the past typically local, and thus the calculation of global environmen- few years, the quality of data in the GEF Climate Change port- tal benefits and incremental costs will be difficult. Currently folio is still relatively poor. CCPS2004 recommended that "the three adaptation-related funds are managed by the GEF, in GEFSEC... provide explicit guidance regarding the realistic addition to the trust fund that includes adaptation in its calculation of GHG avoidance or reduction in project design Strategic Priority. The GEF is establishing pilot or demonstra- and implementation and the manner in which impacts should tion projects to show how adaptation planning and assess- be monitored and reported." OPS3 finds that discussions are ment can be translated into projects that provide benefits and already ongoing about how to move to a more harmonized that may be integrated into national policy and sustainable approach to the Climate Change focal area that will generate development planning. measurable quantifiable results, where possible, and clearer measures of impacts where quantification is more difficult. Carbon financing and its relationship with the GEF. Additionally, the approach developed by the GEFSEC, OME, CCPS2004 found that it would be "useful to clarify GEF in- and IAs and EAs to estimate GHG emissions avoided through volvement in carbon finance programs." OPS3 believes that GEF projects should be published as a guide for project pro- carbon finance will play an increasing role in improving the fi- ponents as soon as possible. OPS3 also supports the develop- nancial returns of many projects of the type that are in the ment of better methods to measure market development re- GEF portfolio, particularly as many regions and countries sults proposed in GEF-4 programming. (such as the European Union, Japan, and Canada) begin to im- pose carbon constraints on their industries, providing compa- nies with an incentive to locate low-cost emission reduction opportunities. With the entry into force of the Kyoto Protocol, some projects that have relied on the GEF may be able to tap carbon financing, which may offer more competitive terms and more easily leverage private sector engagement. Focal Area Analysis 31 Indeed, the GEF portfolio is beginning to see competition · Identify and synthesize lessons learned from activities imple- for funding arising from the Clean Development Mechanism mented under the GEF-3 adaptation Strategic Priority to in- (CDM) in market segments such as wind and landfill gas, form future activities and to feed into any potential revisions which are especially attractive for carbon financing because of the GEF adaptation strategy. In the further future, through these renewable energy technologies are almost cost com- the experiences in its adaptation pilot activities, the GEF will petitive with fossil fuels in power generation. This is not, how- need to develop plans for a more strategic response to adap- ever, likely to be the case for the greater portion of the GEF tation following the pilot program, given the cost paradigms Climate Change portfolio, particularly as it relates to off-grid each funding source requires. rural energy projects and longer-term technologies. The CDM · Evolve the climate change portfolio in light of the maturation pipeline does not feature many energy efficiency projects be- of the global carbon market by considering the exclusion or cause of the difficulty of proving additionality; however, limitation of specific technologies that are already attracting based on several new methodologies being submitted to the significant carbon finance in specific countries. In general, it CDM Executive Board, this may change as improvements in will be important for the GEFSEC and the IAs to clarify those energy efficiency are shown to earn carbon credits. For now, areas where carbon finance is competitive to inform the GEF's however, transforming the markets for energy efficiency will decisions regarding intervening in specific technologies and likely continue to be an area of focus for the GEF. countries that are already attracting carbon finance. Just as the GEF is experiencing competition in some market segments, the CDM and carbon finance are currently active in areas where the GEF is not. For example, COP 9 in Milan de- International Waters (TORs 1A and 1B) fined the rules for CDM-eligible carbon sequestration projects. To date, the GEF has not considered projects in these areas. Findings on the GEF's Contribution to the Health Geological sequestration through carbon capture and seques- of International Waters tration has been identified by the International Energy Agency's GHG research and development program, among As the GEFSEC noted in document GEF/C.24/3 submitted to others, as a key medium-term technology for sequestering car- the GEF Council in November 2004, the GEF International bon. As the technology evolves, it may be appropriate for the Waters Program was more than successful in achieving the GEF to undertake pilot carbon capture and sequestration proj- mid-term target set by the Third GEF Replenishment; indeed, ects given the significant scope for demonstrating this tech- in FY2003 and FY2004, projects were approved in six new nology in developing countries with large geological seques- transboundary water bodies with the aim of facilitating the tration potential. This is a key area for the STAP to monitor. establishment of a variety of management frameworks. These approvals represent a significant expansion of the geographic coverage of foundational projects in the portfolio. Recommendations In addition to endorsing the CCPS2004 recommendations on Impacts of the GEF International Waters Program strategic coherence, strategic direction, and lesson learning, In terms of environmental and socioeconomic status impacts, OPS3 recommends the following based on the discussions the International Waters focal area is making progress by de- above: termining relevant indicators and setting goals for improved water quality. However, only a few projects in the International · Exploit fully the unique opportunity provided by the second Waters portfolio have entered a SAP implementation phase, round of Non-Annex I NCs to develop shared agreements and while these projects are making valuable contributions to about priority policies, programs, and projects. Specifically, stress reduction that are expected to eventually result in envi- the GEF could use the opportunity to develop sectoral strate- ronmental status impacts, it is too early in the lifetime of these gies for those countries with existing or anticipated large and projects to report on such impacts. diverse project portfolios. 32 OPS3: ProgressingToward Environmental Results With regard to stress reduction impacts, IWPS2004 report- resistance. The International Waters focal area also serves as an ed being reasonably satisfied that monitoring of these im- example for other focal areas of the benefits of systematic pacts was happening in most projects. Stress reduction that identification and incorporation of lessons learned; IW:LEARN will help prevent future degradation of vital systems is occur- is in its second phase and has the potential to become increas- ring in some water bodies, such as the Black Sea­Danube7 and ingly effective at producing focused results and yielding useful Lake Victoria. In other water bodies, the results of GEF support products such as the Transboundary Diagnostic Analysis are still being quantified and are likely to be greater than can (TDA)/SAP course. be measured at this time. Challenges remain, however, to the successful achieve- ment of outcomes. In particular, IWPS2004 reported a need Outcomes of the GEF International Waters Program for better coherence between components of Strategic Part- In general, the program has achieved significant outcomes. nerships and better cooperation between IAs at the opera- There are numerous examples of International Waters focal tional level. IWPS2004 also found that the TDA/SAP can be an area outcomes that in time will result in stress reduction im- effective tool if the right elements are incorporated; however, pacts. While there is no global convention for which the GEF not all TDAs and SAPs examined by IWPS2004 included these International Waters focal area is the financial mechanism, the elements, limiting their effectiveness. Moreover, OPS3 found International Waters Program has itself supported the negotia- that greater attention is needed to ensure adequate IA man- tion and implementation of a number of global and regional agement and supervision during project implementation. conventions. This focal area has been an effective agent for policy, legal, and institutional reforms, and for valuable, but Challenges and Strategic Trade-Offs unquantifiable, results such as regional integration, political stability, and promotion of peace and security. Further, one of Clarification of programs, processes, and practices. The In- the strengths of GEF interventions is that they provide a forum ternational Waters Program Study undertaken in 2000 noted for countries to address external effects that their activities that "much more could be done to clarify the role of the vari- have on other countries sharing the same water body. Often ous Operational Programs," including OP8, OP9, OP10, and countries resist coming to the table, but the GEF International OP12. In reviewing these OPs, IWPS2004 found little evidence Waters Program can often be credited with overcoming this of progress to clarify their definitions, although IWPS2004 did Focal Area Analysis 33 note that the range of projects implemented under OP10 has use of comparative advantages of the IAs within each inter- expanded. To address these conceptual discrepancies, vention and also examine the selection of EAs in accordance IWPS2004 recommended "the production and use of an ac- with agreed criteria. The GEF management response, while cessible GEF International Waters Focal Area manual to clarify supporting the recommendation, suggested that additional the concepts, tools and processes that are giving rise to recur- corporate resources could be required to implement the rec- rent difficulties for project design and implementation." OPS3 ommendation. OPS3 supports these recommendations to im- endorses this recommendation, noting that such a document prove coordination within the International Waters Program. also could be useful in the other focal areas. Additionally, OPS3 observed the importance of IA manage- ment in fostering coordination on the ground, as well as yield- M&E and measurement of results. IWPS2000 recommended ing optimal project results. With this in mind, OPS3 recom- that "a streamlined oversight and tracking methodology... be mends that IAs give greater attention to ensuring adequate prepared and implemented." While the IWPS2004 identified project management and supervision. some areas of progress, including attempts at indicators and improved project identification, current deficiencies in the Recommendations M&E system were identified and attributed largely to the fact that M&E components have not integrated well into a system. In addition to the recommendations from IWPS2004 en- To address these shortcomings, IWPS2004 recommended that dorsed by OPS3, OPS3 recommends the following: the GEF "develop a comprehensive M&E system for Interna- · The GEF International Waters Program should shift from a tional Waters projects that ensures an integrated system for in- testing and demonstration mode (enabling activities) to scal- formation gathering and assessment throughout the lifespan ing-up of full operations in support of agreed incremental of a project." OPS3 endorses this recommendation and finds costs of reforms, investments, and management programs that this system should provide standard formats for reporting needed to address agreed priorities for globally critical trans- on stress reduction and environmental and socioeconomic in- boundary freshwater and marine systems. Focus should also dicators.8 However, OPS3 recommends that monitoring efforts be given to ensure adequate project management and super- not be excessively rigorous or cost prohibitive, so that they do vision during implementation. not detract resources from the GEF's primary objective, which is to catalyze action on the ground. Ozone Layer Depletion (TORs 1A and 1B) Coordination and management. IWPS2004 identified short- comings with respect to coordination in the International Findings Waters Program, specifically at the regional level and in terms of the role of an important coordination mechanism in the In- Countries with economies in transition (CEITs) have nearly ternational Waters Program--the GEF International Waters completed the full phaseout of Annex A and B substances, Task Force (IWTF). To that end, IWPS2004 recommended that and the GEF has essentially achieved its main objective in the a regional-level coordination mechanism for International Ozone Depletion focal area--to eliminate the consumption Waters projects be established. The GEF management re- (that is, production, exports, and imports) and emissions of sponse pointed out actions that are already being taken to ODS in CEITs. Consumption of Annex A and B substances9 in improve regional coordination and that regional coordination CEITs decreased from about 296,000 ozone depleting poten- has also been included as a feature in Strategic Partnerships. tial (ODP) MT in the late 1980s to less than 350 ODP MT by IWPS2004 also recommended that the GEF IWTF be redefined 2003--a reduction of more than 99.8 percent. (At the time of in a way that would (1) enhance its role in the definition of the publication of the 2000 Ozone Study, consumption was technical guidelines and policies, and (2) ensure the optimum estimated at 14,600 ODP MT.) 34 OPS3: ProgressingToward Environmental Results Total appraised direct ODS phaseout10 has amounted to ap- round of replacements. Moreover, when (or if) HCFC phase- proximately 19,000 ODP MT, an increase of almost 700 ODP out strategies are developed, dialogue will be needed to MT since 1999. This increase can be mostly attributed to the establish strategies to promote the implementation of the considerable progress of Estonia, Tajikistan, and Kazakhstan in most climate-friendly alternatives (that is, limit the use of implementing their Country Programs for ODS phaseout. For HFCs, to the extent possible). In addition, efforts are needed individual countries, assistance from GEF projects has ac- to ensure that the Approved Methodology AM001 (Incinera- counted for between 20 and 60 percent of the total ODS con- tion of HFC-23 Waste Streams) established under the Kyoto sumption phased out in the Country Program base years. On Protocol's CDM does not lead to perverse incentives related average, 28 percent of total ODS consumption phased out is to the phaseout of HCFCs (by increasing HCFC-22 consump- associated with GEF projects.11 Remaining consumption is be- tion, as well as the eligible incremental costs that will need to ing phased out through market forces, legislative measures, or be supported by the GEF and Multilateral Fund). the implementation of economic instruments by agreement. While all countries with the exception of Kazakhstan have Capacity sharing. Country strategies, which phased out ODS met the 70 percent reduction target for methyl bromide for and resulted in institutional strengthening, have enhanced 2003, additional efforts will be needed to completely phase country commitments; however, because incremental operat- out the consumption of methyl bromide. To aid in these ef- ing costs are not covered in GEF funding, capacity in CEITs forts, the GEF Council named methyl bromide reduction as may be short-lived, which may in turn have implications for one of its Strategic Priorities in its FY2005­07 Business Plan; in sustainability or the sharing of capacity across chemicals man- addition, a GEF-4 Strategic Objective, which addresses the agement treaties. residual use of methyl bromide as well as HCFCs and institu- tional strengthening activities, has been proposed. In light of Clear results chain. Lessons from the Ozone Depletion focal the significant activities still to be considered and agreed area regarding the establishment of an integrated results upon with respect to ozone depletion, and in light of the re- chain, as well as setting of baselines, can be learned particu- cent special report on ozone and climate prepared by the larly for the POPs focal area or other chemicals management joint IPCC/TEAP scientific panel (UNEP n.d.), which indicates agreements. the deleterious impact that climate change has on the mitiga- tion of ozone loss, OPS3 supports the continued attention Recommendations that the GEF-4 Strategic Priority on ozone implies. · The GEFSEC should coordinate with the MLF Secretariat re- garding methyl bromide and HCFC phaseout issues. The part- Challenges and Strategic Trade-Offs ners should ensure that funding eligibility requirements for There are few remaining issues in the Ozone Depletion focal HCFC users are kept current and appropriate, and that the area to be addressed, in large part because of (1) the clear re- most climate-friendly options are implemented to replace lationship between funded activities and ODS phaseout, (2) HCFCs. In addition, the GEFSEC should coordinate with the the maturity of the focal area, and (3) the strength of the CDM management bodies and the MLF Secretariat to ensure agreements made among Parties to the Montreal Protocol. that Method AM0001 does not undermine efforts pursued un- However, several issues may affect the achievement of future der the Montreal Protocol. For the methyl bromide phaseout, success in this focal area and/or have implications for success the GEFSEC should coordinate with the MLF Secretariat on in other focal areas, which are summarized below. project funding eligibility issues and phaseout opportunities. · IAs should prevent backsliding by ensuring that adequate HCFC phaseout. Continuing dialogue must be monitored customs training and border enforcement activities are con- with respect to convention guidance on HCFC phaseout; the ducted to thwart illegal trade of ODS. MLF stipulates that funding will not be provided for a second Focal Area Analysis 35 Land Degradation (TOR 1C) herence report (GEF/C.24/6/Rev.2 2005) that the following four challenges in the Land Degradation focal area will need Findings on Meeting Global Priorities to be addressed in GEF-4: Because the United Nations Convention to Combat Desertifi- · Defining criteria, programming and project performance cation (UNCCD) has been agreed upon by the international indicators, and a science-based priority-setting process community, the priorities outlined in the Convention are rec- that will ensure the selection, monitoring, and evaluation ognized as global priorities. These priorities are addressed by of viable initiatives with desired impact. the GEF in its OP for Land Degradation (OP15), the Land · Fostering systematic learning through a well-tailored Degradation Business Plan, and project documents. As shown knowledge management program. in exhibit 5, OPS3 finds that the GEF has generally addressed the UNCCD global priorities--with two exceptions: (1) UNCCD · Finding a mutually beneficial balance of trade-offs be- has a priority for combating desertification in Africa, whereas tween global environmental benefits and sustainable the GEF Land Degradation focal area strives for geographic livelihoods, while not diverting from the GEF core mandate balance; and (2) UNCCD focuses on combating desertification, to protect the global environment. but GEF projects tackle all causes of land degradation, includ- · Balancing the steadily growing demand for GEF support ing those occurring in humid areas. under this new focal area against limited available re- sources. Challenges and Strategic Trade-Offs OPS3 also believes that the lack of baseline data against To respond to global priorities, the GEF Land Degradation which to measure the results of GEF land degradation proj- Program must continue to strike a balance among its Strate- ects in terms of actual environmental improvement over time gic Priorities, project approaches (that is, "traditional" versus is a challenge that needs to be overcome in this focal area (as cross-cutting sectoral and programmatic approaches), and well as others). The GEF is well aware of these challenges, and land use systems (agriculture, rangeland, and livestock and is making strides in addressing them in its planning and de- forestry). OPS3 concurs with the findings of the Scope and Co- velopment of strategic objectives for GEF-4. Exhibit 5. Comparison among UNCCD, OP15, and the Land Degradation Business Plan Priorities UNCCD Priorities OP15 Priorities LD Business Plan Prorities Combat desertification and sustainable land management Indigenous involvement Creation of enabling environments Capacity building Mainstreaming into national priorities · Stakeholder involvement · Technology development and coordinated information collection · Geographic priority to Africa · · · = Not covered under the scope of OP15 or Business Plan; = Priority addressed by documents; = Priority emphasized by documents. 36 OPS3: ProgressingToward Environmental Results Recommendations mentioned in the Convention are addressed in the GEF strate- gy, with the exception of the potential need to identify and · As the GEF moves forward in its programming for GEF-4, there regulate the production of new chemicals with POPs charac- should be a clear evolution to identifying synergies among teristics. In addition, there are several differences in the em- the focal areas and to programming in the broader context of phases placed on priorities in the Convention versus those ar- integrated natural resource management. ticulated in OP14 and the POPs Business Plan: the GEF places · A system for prioritization of GEF funding under OP15 (sus- greater emphasis on capacity building and institutional tainable land management) should be established. strengthening, the need for innovative and cost-effective technologies for the disposal of POPs, and the aim of promot- · The GEFSEC should develop a set of output and outcome indi- ing synergies by integrating POPs management practices cators, with a focus on the global and incremental benefits at with other focal areas. the project, national, and global levels. Thus far, the initial focus of the GEF within the POPs focal area has been to assist developing countries to prepare their NIPs in response to the Stockholm Convention. As of January POPs (TOR 1C) 2005, 119 proposals for enabling activities for the develop- ment of NIPs had been approved. Findings Because the Stockholm Convention has been agreed upon by Challenges and Strategic Trade-Offs the international community, the priorities outlined in the Convention are recognized as global priorities. The Conven- Responsiveness. The GEF is responding to convention priori- tion's global priorities are addressed by the GEF in its OP for ties through its POPs strategy, but it is essential that the GEF POPs (OP14), the POPs Business Plan, and project documents. continue to monitor responsiveness and react to guidance As shown in exhibit 6, OPS3 finds that all global priorities from the first COP in May 2005, as appropriate. Based on such guidance, the GEF should work with the Stockholm Conven- tion Secretariat to finalize OP14. Moreover, because the NIPs Exhibit 6. Comparison between Stockholm Convention Priorities and OP14 and POPs Business Plan Priorities Convention Priorities OP14 and POPs Business Plan (GEF/C.22/6 2003) Priorities Protect human health and the environment Impose production limits Develop reduction strategies Monitor releases Develop and implement action and implementation plans Identify new POPs · Raise awareness Achieve multi­focal area benefits · · = Not covered under the scope of the Convention,OP14,or Business Plan; = Priority addressed by documents; = Priority emphasized by documents. Focal Area Analysis 37 have been developed in parallel with convention guidance · The GEF IAs should continue exploring options to reduce du- and the guidelines for Best Available Technologies and Best plicative and competing initiatives for chemicals manage- Environmental Practices, it is important that this guidance ment, as well as to find opportunities to leverage infrastructure feed into the implementation of the NIPs and the finalization and capacity in chemicals management from within other of OP14. Also, country focal points expressed concern that the chemicals management protocols and initiatives. Cross­focal quality of technical assistance provided by IAs and EAs in the area synergies involving POPs should also be pursued. POPs focal area varies widely. Demonstration projects have begun under OP14 without clear instruction from the Con- vention on how to calculate incremental costs, leading to Responsiveness of the GEF to Conventions some difficulties in terms of identifying and calculating incre- (TOR 4C) mentality for certain projects. Biodiversity (CBD) Determining baselines and a clear results chain. Adequate OPS3 finds--as did OPS2, the Second CBD Review of the indicators for measuring the global environmental benefits GEF,12 and the BPS2004--that the GEF has been generally re- associated with the elimination of POPs are not available; sponsive to COP guidance. As of March 2005, the GEF has pro- without such indicators and the development of baseline vided funding for 300 enabling activities in the Biodiversity fo- data, it may be difficult to raise the public and political profile cal area with a total allocation of US$122.6 million. The GEF of POPs issues globally, or fully understand the impacts of GEF has also funded activities in almost all of the areas of guidance activities on the global environment. provided by the COP. In particular, as the BPS2004 points out, the GEF has been particularly responsive to guidance on for- Synergies. An "untapped" opportunity exists for incorporat- est ecosystems and capacity building in biosafety. ing more cross­focal area synergies into project design and The GEF faces some challenges, however, in addressing implementation. For example, energy efficiency projects un- COP guidance. The BPS2004 found that the GEF remains fully der the climate change portfolio may have synergies and op- responsive in the areas of "implementing effective incentive portunities for cost sharing with POPs projects relating to measures, implementing national plans and strategies, devel- dioxin and furan reduction. Similarly, institutional synergies oping indicators, developing and applying baselines to moni- exist between the Stockholm Convention and other conven- tor changes in the status of biodiversity over time, and estab- tions dealing with the movement and management of chemi- lishing mechanisms for promoting the sustainability of cals and chemical wastes, particularly the Rotterdam and project outcomes." OPS3 also finds that the GEF has not ade- Basel Conventions. quately addressed the CBD priority on access and benefit sharing. Moreover, although the GEF has been responsive to Recommendations the CBD by assisting countries in preparing their national re- ports (as also noted by OPS2), IAs have reported to OPS3 that, · Regular dialogue between the GEFSEC and the Convention in some countries, the reports to the CBD can be isolated, and COP should be maintained to monitor differences between therefore neither mainstreamed nor influential. the Convention and GEF program implementation of the With respect to the more general challenges that the GEF POPs phaseout, identify priorities to be addressed in future must address, the general lack of prioritization of guidance GEF projects, and address any issues requiring clarification from the COP remains a challenge, according to BPS2004. Ad- (such as the calculation of incremental costs). ditional challenges identified by the BPS2004 include "forging · GEF should begin dialogue and engagement with the scientif- a participatory approach among relevant parties to enable an ic community about how to cost-effectively develop baseline agreement on the clarification and prioritization of COP guid- information on POPs concentration in the environment and ance" and "the apparent expectation that all COP guidance in human populations. will be supported by the GEF, at the same level and in perpe- tuity." The BPS2004 recommended "more concerted efforts to 38 OPS3: ProgressingToward Environmental Results improve the dissemination of information on how the GEF re- With regard to NCs, the GEF has been responsive by sup- sponds to guidance," noting that the GEF Country Dialogue porting 3 of 40 NCs from Annex I countries, and 105 of 115 Workshops could provide a good venue for these efforts. NCs from NAI countries. UNFCCC representatives reported to OPS3 supports this recommendation, noting that the revised OPS3 that 3 countries have also submitted their second NCs, NDI, which replaces the Country Dialogue Workshops (CDW), and that about 65 countries have reported that they have also provides for follow-up, which may present an opportuni- started the process of preparation of their second NC. Several ty for the GEF entities to assess whether guidance has been weaknesses, however, have been identified with respect to incorporated at the country level. the first round of NCs that should be remedied in the second round; please see the "Climate Change" subsection in the "Focal Area Analysis" section for OPS3's assessment of the NCs Climate Change (UNFCCC) and recommendations for the second round. In general, OPS3 finds--as did OPS2, the 2002 COP 8 review of With respect to the adaptation priority of the Convention, the GEF, and the CCPS2004--that the GEF has effectively per- the GEF has begun to respond. In November 2003, a Strategic formed its role as financial mechanism of the UNFCCC and has Priority piloting an operational approach to adaptation was been responsive to its mandate as defined by the Convention approved by the GEF Council, along with an associated US$50 and guidance and priorities as given by the COPs. GEF funding million in funding. Adaptation is also addressed under the of projects has been in direct response to the priorities out- LDC Fund and the SCCF, and is intended to be addressed un- lined by the COP. Moreover, communication and coordination der the Adaptation Fund, although there are no clear criteria between the UNFCCC and the GEFSEC has improved over the yet for how this fund will be operationalized. Indeed, the GEF past few years, aided by joint activities such as retreats. still has much to sort out in terms of its funding of adaptation The GEF has been particularly responsive in quickly mobi- activities--for instance, how it will mainstream adaptation lizing and implementing special trust funds, such as the Least into the other focal areas of the GEF portfolio. These issues are Developed Countries (LDC) Fund and the Special Climate addressed further in the "Climate Change" subsection, as not- Change Fund (SCCF), as mandated by the COP. Indeed, under ed in the previous paragraph. the LDC Fund to date, projects for the preparation of National Adaptation Programmes of Action (NAPAs) had been ap- Ozone Layer Depletion (Montreal Protocol) proved in 43 of the 48 LDC Parties to the UNFCCC, as well as two global support projects, for a funding total of US$9.4 mil- As discussed in the subsection on Ozone Layer Depletion lion. The GEFSEC has also prepared a proposal (GEF/C.24/Inf.7 above, almost all CEITs for which the GEF has provided assis- 2004) on the process for implementing NAPAs in response to tance are now in compliance with the Protocol. As such, the COP 9 guidance that the GEF support the implementation of GEF has essentially achieved the main objective of the Mon- NAPAs as soon as possible after their completion. Also at COP treal Protocol--to eliminate ODS consumption (that is, pro- 9, Parties agreed on guidance for the operation of the SCCF; duction, exports, and imports) and emissions. the GEFSEC, in consultation with the IAs and UNFCCC Secre- In regard to strategic responsiveness, OPS2 found that the tariat, prepared a programming paper describing how the GEF was both responsive and supportive of the Montreal Pro- SCCF will fund activities in the program areas of adaptation tocol. Since OPS2, only one specific request has been made of and technology transfer. This document was endorsed by the the GEF by the MOP, related to providing funding for South GEF Council at the November 2004 meeting. Africa to phase out methyl bromide. The GEF was fully respon- sive and agreed to provide the requested funding for South Africa. Focal Area Analysis 39 Achieving and Sustaining Global Environmental Benefits G lobal environmental benefits are defined in a 1996 However, in the more complex GEF projects, which repre- GEF policy paper as those benefits obtained "when- sent the majority of the portfolio--such as many of those in ever a global environmental objective is met," the Biodiversity, Climate Change, International Waters, and which includes the achievement of any of the conventions' Land Degradation focal areas--environmental improvements environmental objectives (GEF/C.7/Inf.5. 1996). The policy pa- can often only be perceived and measured over long time- per further explains that a "global environmental benefit" is frames (on the order of decades). As such, achieving global distinct from the achievement of development or local envi- environmental benefits necessarily requires that the out- ronmental benefits. comes be sustainable. For example, a biodiversity project may The achievement of global environmental benefits can be establish an effectively managed protected area as an out- measured at the outcome level (short- or medium-term ef- come, but the sustainability of that protected area is needed fects) or the impact level (long-term effects). In projects that in order for the project to result in reduced biodiversity loss. are highly technical in nature or have a well-understood re- Whether global environmental benefits can transpire from sults chain (for instance, projects in the Ozone Depletion focal a project in the short, medium, or long term, they will only be area), the achievement of global environmental benefits can achieved if the right set of conditions is in place on that partic- typically be realized in the short term, and will continue un- ular time scale. Sustainability is the continuation of those con- less specific action is taken to reverse the progress made. For ditions to allow the continuation of benefits. Specifically, in example, CFC production sector phaseout projects target the the context of the GEF, OPS3 defines sustainability to be the shutdown of CFC-producing facilities, thereby eliminating continuation of achieved benefits after project completion ODS supply and subsequent emissions, and resulting in quan- and, in particular, the persistence of conditions--sociopoliti- tifiable global environmental benefits. cal, economic, and environmental--brought about from the 41 project. The focus on conditions is important in underscoring · The "way"--viable means and tools to achieve and sustain the need to look beyond impacts when assessing sustainabili- environmental benefits, including resource factors, design ty, and to explicitly consider the context in which project ben- factors, and management factors. efits and activities will be able (or not able) to endure. In prac- "The will" is the sine qua non, the most basic requirement tical terms, this requires a focus on the assumptions and risks without which GEF benefits cannot be achieved or sustained. about initial conditions, and careful consideration of what the Political will among national governments can be demon- end goal should look like, and how to get there. strated and garnered in a number of ways. It is evident in the Given that the achievement and sustainability of global actions and decisions of political leaders that demonstrate environmental benefits are closely related, if not one and the country ownership (at the national government level) and same, OPS3 treats the discussion on factors for the achieve- country drivenness. This can be manifested through national ment and sustainability of global environmental benefits laws and policies (including economic development plans) jointly in this report. The overlapping nature of these con- that are aligned with GEF project goals, and institutional or fi- cepts was further demonstrated by an exercise conducted by nancial commitments to projects. Environmental mainstream- the OPS3 team. Specifically, the team developed a matrix of ing, awareness raising, and capacity building at the national factors, based on a review of the GEF literature, for both the level can help build political will among government decision achievement and sustainability of global environmental ben- makers. efits; the resulting matrices were virtually identical. Local will--including that of local governments, NGOs, the private sector, and civil society--is also critical to the achieve- ment and sustainability of all projects and programs, and es- Findings (TORs 1D, 2A, 2B, and 2C) pecially for those that are community based or require human behavior changes to achieve desired environmental impacts OPS3 found encouraging evidence that GEF is producing im- (as is common in the Biodiversity and Land Degradation focal portant, lasting global environmental benefits on the ground. areas). To this end, outreach and awareness raising, stake- OPS3 can point to a number of completed projects that have holder participation, and the generation of local benefits are achieved sustainability, and a number of ongoing projects critical. In building local will, particular attention should be that appear well on course to do so. Other examples of proj- paid to vulnerable groups such as indigenous peoples, ects that have not delivered, or are not likely to deliver, sus- women, and children. tainability have also been identified. Obtaining an accurate OPS3 conceptualizes "the way" to achieve and sustain ben- sense of the extent of sustainability beyond such pockets of efits as consisting broadly of three components, which do not evidence is difficult given the lack of data on actual or likely apply evenly to all projects: (1) resource factors, (2) design fac- sustained benefits from GEF projects. tors, and (3) management factors. Resource factors are assets Based on the desk and field reviews, the OPS3 team com- that must exist during a project's implementation and endure piled a list of key factors that influence the achievement and once GEF involvement comes to an end: financing, strong in- sustainability of projects. The team found that, at the most stitutions (institutional capacity), and capable people on the fundamental level, two factors are key to all GEF projects and ground (human capacity). Design factors include sound scien- programs, regardless of focal area or project type: tific, technical, and economic bases; proper identification and · The "will"--political will (at the national government level) targeting of root causes; realistic goals; appropriate project and local will (at local government and community levels); scope and scale; mitigation of external risks, and M&E. Man- and agement factors are adequate project management (includ- ing competence, technical know-how, and oversight) and the use of adaptive management approaches to react to new in- formation or changes in situations on the ground. 42 OPS3: ProgressingToward Environmental Results Challenges and Strategic Trade-Offs Improving Information Management Results versus conditions--Assessing sustainability requires a Several strategic tensions are associated with both "the will" dedicated focus to monitoring and measuring conditions, not and "the way" for achieving and sustaining benefits, as well as simply results. with the GEF's management information systems for tackling successes in these areas, as described below. Ex post project evaluations versus additional costs--Accu- rately determining the extent of benefit sustainability requires Ensuring the"Will" systematic ex post monitoring and data tracking. These activi- ties will require additional resources. National versus global priorities--Global environmental pri- orities, as advanced by the GEF, may not be consistent with country priorities, making it harder to shore up political will. Recommendations Broad country focus versus prioritizing political will--The · The GEF should establish a permanent "Sustainability Team" GEF's focus should be placed on countries where political will within the OME to develop and operationalize sustainability exists. In countries where it is not sufficient, efforts should be indicators; operationalize the reporting and collection of proj- spent on building political will, not on actual projects. ect-level information on likely and actual sustainability achieved, so that data can be tracked, aggregated, and as- Participation of vulnerable groups in policy versus practice-- sessed across the GEF portfolio; and conduct systematic ex Despite GEF guidelines on marginalized groups and stakehold- post monitoring. er participation, greater attention is needed to address the concerns and rights of indigenous peoples, and to more · Continued emphasis on activities designed to mainstream the strongly consider gender issues in project design and imple- environment at the national level and to strengthen focal mentation (particularly in rural, community-based projects). points is needed to enhance political will. To promote country ownership and drivenness, the GEF should move toward a stronger country program focus to plan the development of Ensuring the"Way" clear country strategies and priorities for GEF funding that is Resource factors: short- versus long-term capacity needs-- linked through dialogue with public and private stakeholders Often, the opportunity to increase the likelihood of achieving in multiple ministries and sectors. Country programs should and sustaining global environmental benefits by building in- consider the importance of using and further developing local country capacity is forgone in favor of international consult- capacity, to increase local will and build in sustainability. (See ants who can get the job done faster. "Recommendations" in the "National Priorities of Recipient Countries" subsection of "The GEF as a Catalyst" for more Design factors: environmental versus development objec- information.) tives--"Win-win" gains in both development and global envi- · The GEF should provide greater assistance in the procurement ronmental goods are rarely possible, and project designs of cofinancing and postproject financing (for example, a must recognize this and address it accordingly. clearinghouse of information on potential donors). Management factors: flexibility versus accountability-- Adaptive management is difficult to work into the complex GEF system, which must be somewhat rigid to ensure ac- countability. Achieving and Sustaining Global Environmental Benefits 43 The GEF as a Catalyst T he GEF attempts to create catalytic effects by using Findings its limited funds to produce faster or more global en- The main findings related to TORs 3A and 3B are presented in vironmental benefits than it could produce on its exhibit 8. own. The GEF can create catalytic impacts by increasing re- sources, fostering conditions, or encouraging processes that lead to the faster or greater achievement of global environ- Challenges and Strategic Trade-Offs mental benefits, as shown in exhibit 7. Financial versus nonfinancial mechanisms for catalytic ef- fects. Dialogue may be warranted to determine whether the GEF's across-the-board focus on financial modes for achieving Catalytic Impacts: Extent and Factors catalytic impacts is appropriate, or whether other modes for (TORs 3A and 3B) catalysis should be given equal weight in the project criteria equation, or at least greater weight than is currently attrib- TOR 3A specifies four mechanisms for catalytic action and uted. asks to what extent the GEF has been successful in fulfilling them: (1) leveraging additional resources from the public and GEF as "leverager" or "leveragee." Some GEF funds are used private sector; (2) catalyzing results by innovation, demonstra- to augment other funds already committed to projects, which tion, and replication; (3) fostering international cooperation would go forward with or without GEF contributions; GEF's on environmental issues; and (4) mainstreaming environmen- limited resources may produce greater catalytic effects if they tal issues into partner institutions. TOR 3B asks about the key are used for projects that would not get implemented with- factors that lead to catalytic impacts and what issues need to out the GEF, or may be more driven by the GEF mission. be addressed to improve catalytic impacts. 45 Exhibit 7. GEF Mechanisms for Producing Catalytic Effects Resources examples:cofinancing, leveraged resources Conditions CATALYTIC EFFECTS Catalytic reaction examples:mainstreaming at all levels, (faster or more global environmental (GEF projects) international cooperation benefits for each GEF contribution) Processes examples: demonstration, innovation, replication Source: Authors. Operationalizing mandate for the private sector. The devel- resources. (See the subsection "The Information Management opment of a GEF strategy that effectively engages the private Challenge for the GEF" of "GEF Procedures" for more informa- sector will require difficult decisions about the extent to tion on general challenges associated with information man- which the GEF is prepared to reach out to industry and recon- agement within the GEF.) cile the differences in modes, styles, and incentives for doing business. Recommendations Innovation versus risk avoidance. Innovation, which is asso- · Systematically track proxies for catalytic effects. The GEF Sec- ciated with risk, is actively promoted by the GEF, yet project retariat, in collaboration with the IAs and EAs, should develop criteria and cofinancers aim to minimize risk and ensure proj- operational definitions and indicators for the mechanisms of ect results. One area where this is a major concern is in lever- catalytic effects (for example, cofinancing, leveraged re- aging private sector involvement in GEF activities, where risk sources, replication, and mainstreaming) and an information may be high, but potential success may be substantial. Proper management system to collect and track these data so that incentives and guidelines are needed to promote the pursuit progress can be assessed. of higher-risk opportunities and foster greater innovation in · Promote catalytic effects through systematizing innovation, order to reconcile these competing bottom lines. demonstration, and replication. The GEF Secretariat, in col- laboration with the IAs and EAs, should organize annual Information management and knowledge sharing. GEF in- workshops for each of the three major focal areas, to share in- formation systems do not adequately track indicators for cat- formation on successful innovations, demonstrations, and alytic effects, and no GEF-wide knowledge-sharing system is approaches that have been demonstrated in the field, and to in place to impart information and foster replication into ap- strategize about how and where to promote their replication. propriate sectors or geographic areas. Developing and imple- Workshop proceedings should be widely disseminated menting such information management and knowledge- throughout the GEF network and, to the extent possible, sharing systems will require the dedication of additional GEF through in-country networks. 46 OPS3: ProgressingToward Environmental Results Exhibit 8. Main Findings of TOR 3A and 3B Innovation, demonstration, Leveraging and replication International cooperation Mainstreaming Extent World Bank and UNDP cofinancing ratios Many GEF projects have replication as an GEF's regional and global projects No GEF-wide system to measure and increasing over time. objective or activity.For example,replication are helping to foster international track extent of mainstreaming is needed to transform markets in the cooperation on environmental issues. throughout GEF portfolio,although a World Bank is responsible for lion's share Climate Change focal area,and to move tracking tool has been developed in of planned cofinancing,which is expected The OPS3 regional workshops in conservation into production landscapes the Biodiversity focal area under given its comparative advantage relative to themselves demonstrated an and sectors in the Biodiversity focal area. Strategic Priority 2. UN agencies in macroeconomic and private important function of the GEF in sector strategies,and its financial leverage. Evidence suggests that the GEF is spurring catalyzing international cooperation, Since OPS2,efforts undertaken to more innovation,demonstration,and replication; as they created a platform for directly target mainstreaming (for A comprehensive assessment of actual however,the actual extent to which this is enhanced organization,collaboration, example,NDI).Some Climate Change cofinancing is not possible given that occurring cannot be determined given the networking,and learning about the and Biodiversity EAs are credited with such data are not tracked,and often not lack of data on this topic.Moreover,an GEF within regions. increasing national government adequately reported in terminal operational definition and indicators for awareness and understanding of evaluations. The GEF has been critical in advancing replication do not exist across all focal areas environmental issues. new multicountry agreements for the GEF projects have leveraged much and project types within focal areas. management of shared water bodies. IAs have made efforts to mainstream nonfinancial support,including technical However,these agreements are often global environmental issues into their expertise,management capacity,equipment weak,with the related institutions operational programs,but level of GEF and technology,and other in-kind limited to advisory functions. influence on IAs and projects may vary contributions from NGO,academic,and based on significance of GEF's financial industry communities. contributions to the particular agency More opportunity to leverage private sector. or project.Partners providing large Visibility of GEF in this sector is low.Complex sums of financing relative to the GEF GEF modalities serve as a barrier. must ensure projects are aligned with GEF's mission. Factors Higher cofinancing does not necessarily Dissemination of information about proven Multicountry projects must be de- To integrate environmental mean greater catalytic effect; GEF may have innovations and successful demonstrations signed with the needs and realities considerations into decision making less influence over projects in which its that can be replicated in other appropriate of individual countries in mind.To at all levels,strong leadership and funds account for a small proportion of locales is needed to promote replication. maximize opportunities for genuine dialogue at all levels are needed. overall budget.Of particular concern is GEF Currently,no cohesive,GEF-wide system is cooperation and information Knowledge sharing and capacity financing used in World Bank projects to in place for such knowledge sharing. exchange,greater collaboration is building are needed to build augment funds already committed. needed both within and between political will. Inclusion of replication strategies in project countries.Trust and confidence must Focal points and NGOs indicated they design and implementation can promote also be built among stakeholders of are less willing or able to commit to GEF replication,although often overlooked in multicountry projects for the projects given the time lag between project GEF projects.Focal area task forces are to cooperation to become self-sustaining. design and approval,which is longer than refine and adapt project review criteria for their own budget cycles. replication to each focal area and project In the International Waters focal type,which may lead to improvements. area,the TDA/SAP process facilitates Cofinancing requirements can cause project long-term involvement of multiple delay or prevent project approval,especially Anecdotal evidence suggests that SGP countries at national and subnational for LDCs and SIDS. projects are more replicable owing to their government levels.Strong low cost.More data on replication are To enhance private sector involvement,GEF multicountry institutions are also needed to determine whether this is true. modalities must fit needs and realities of important factors. the business world. The GEF as a Catalyst 47 · Expedite development of a private sector strategy. The GEF · Clarify acceptable use of GEF funds to maximize leveraging. Secretariat, in coordination with the IAs and EAs, should work The GEF Council should develop a policy paper on the proper directly with members of the private sector to identify appro- use of GEF funds as they relate to the leveraging of financing priate means and modalities to more effectively involve the and other resources. This policy should also aim to ensure private sector, and develop a strategy for outreach and risk that cofinancing is targeted for GEF projects and not vice ver- sharing. Such means and modalities should be tried through sa. implementation of a private sector pilot program, which · Conduct further analysis into catalytic mechanisms. Pending should be funded as a special initiative. enhanced data availability, the GEF Secretariat should con- · Broaden focus on nonfinancial mechanisms for catalysis. The duct thematic reviews to evaluate (1) whether greater cofi- GEF should consider substituting cofinancing requirements nancing contributions are delivered when commitments are with other requirements related to nonfinancial mechanisms made once projects are under way instead of prior to their ap- for catalysis described in this chapter, to benefit LDCs and proval; (2) the extent to which different types of leveraging SIDS. Such changes would require that a task force be estab- have produced catalytic effects; and (3) the influence of proj- lished to identify appropriate indicators and requirements for ect (funding) size on replication and sustainability. The GEF nonfinancial mechanisms for catalysis. can thus assess whether SGP projects are more conducive to 48 OPS3: ProgressingToward Environmental Results fostering sustainability and replication. developed with a long-term national strategy. According to OPS2, and agreed by OPS3, such a country program approach should be pursued jointly by all key GEF partners, together National Priorities of Recipient Countries (TOR 4E) with the national operational focal point and other key in- country stakeholders, coordinated by the GEFSEC. To date, Findings the lack of portfolio planning and coordination at the country level has produced results that may not always be optimized The OPS3 field study found that, in some countries, GEF proj- or achieved in the most cost-effective manner; for example, ects are responding very well to national priorities. For exam- inconsistent focus within countries and somewhat duplicative ple, the GEF has prompted the setting of national environ- projects reduce the effectiveness of GEF projects. In the event mental priorities that fall within GEF focal areas in some that an RAF is approved, the GEF will likely have to systemati- countries, and IAs have been very responsive to country focal cally allocate resources among countries. In this context, for points. Overall, the GEF portfolio was found to best reflect countries with robust GEF portfolios, developing and manag- country priorities in countries that were able to move toward ing country strategic programs--within existing structures-- greater strategic partnership among IAs and among govern- could optimize results with the resources allocated to each ment ministries executing projects. In addition, the GEF NDI,13 country. the GEF NCSA, UNDP's Country Programme Action Plan (CPAP), and the World Bank's Country Assistance Strategy (CAS)14 have played a positive role in helping to create a cohe- Challenges and Strategic Trade-Offs sive framework by which to strategically plan and pursue GEF National versus global priorities. The GEF's mandate to fos- (and other) environmental projects. ter global environmental benefits may clash with its mandate However, there is much room for improvement. In the ma- to fund projects that are country driven and based on nation- jority of site visits conducted by OPS3, country focal points al priorities, since countries act in their sovereign interest-- and other government representatives raised concern over not in the interest of the global environment, per se. the responsiveness of IAs, their influence over project propos- als submitted to the GEF Council, and their conflicting motiva- Local and regional versus country priorities. Country priori- tions to serve the needs of countries versus the needs to de- ties should reflect priorities at the state and local levels, but in velop projects that are approvable and expand levels of many countries this is not the case. business. Moreover, weak links in the chain can occur at the country level if focal points or the environmental ministries in Mandating versus ensuring country ownership. The devel- which they are housed are weak or ineffective, if focal point opment and selection processes for GEF projects are not project selection is based on motivations other than national transparent, and as such, focal point approval does not neces- priorities, if national priorities do not reflect state or local pri- sarily translate into country ownership. Ensuring country own- orities, if government corruption is high, or if countries lack a ership, however, is critical to project success and is a mainstay stable central government. These various dynamics may often of ensuring alignment with national priorities. While there is lead to the focal point approval of GEF projects because they no way for project criteria to guarantee that GEF projects are are good funding opportunities, even though they may not country driven or truly have country ownership in the current be at the top of a country's priorities. Finally, some country fo- system, this is in fact essential. cal points and NGOs--particularly those in SIDS and LDCs-- noted that GEF has not been responsive to their country prior- Smaller projects. Continue promoting smaller projects (in ities, which, by necessity, place poverty elimination at the top. terms of funding) that fit the absorptive capacities of LDCs OPS2 supported a long-term programmatic approach un- and SIDS (for instance, by requiring less reporting and admin- der which all GEF projects in a country are coordinated and istrative burdens). The introduction of the pilot program for The GEF as a Catalyst 49 the financing of smaller MSPs (up to US$250,000) is a positive step in this direction. Recommendations · Continue efforts to link poverty alleviation with environmen- tal efforts, particularly for LDCs and SIDS. · Move toward stronger strategic GEF planning at the national level. Country programs should be planned within existing structures, by a multistakeholder team and coordinated by the GEF Secretariat. Country programs should be developed as an outgrowth of and in concert with activities such as the NDI and NCSA. The development of country programs could fit nicely into an RAF structure--if approved--since the RAF will likely require the GEF to allocate resources among coun- tries in a systematic manner. GEF will need to prioritize proj- ects at the country portfolio level by establishing and dissemi- nating a prioritization process and criteria, which could include innovativeness, replicability, cost effectiveness, and other parameters. Varying Capacities of SIDS, LDCs, and CEITs (TOR 4F) Findings OPS3 field studies revealed that the GEF has played a critical explain, at least in part, why the visibility of the GEF is ex- role in strengthening environmental institutions and develop- tremely low in the region. In Caribbean SIDS, many workshops ing capacity in LDCs and SIDS. The SGP, the rollout of NAPAs, have been held, but without adequate follow-up to spur con- and the introduction of OP15 have helped respond to the crete action. needs and capacities of these countries. In the Pacific SIDS, Institutional capacity in LDCs and SIDS, and to a lesser ex- the OPS3 workshop was in itself a great forum for information tent, in less developed CEITs, still remains largely inadequate exchange and collaboration for stakeholders in the region. to pursue GEF opportunities and fulfill GEF obligations. High Several issues were noted by the OPS3 team with regard to staff turnover has led to the loss of institutional capacity. Cofi- the GEF's consideration of the varying capacities of countries. nancing is another obstacle for LDCs, SIDS, and less devel- First, the vast majority (88 percent) of GEF projects in the Pa- oped CEITs. The GEF's outreach strategy has not permeated cific SIDS are enabling activities that have focused on fulfilling these regions, and country focal points do not have the re- international reporting requirements that have not benefited sources to perform their roles, given the difficulties and ex- countries (or the environment) in tangible ways, which may penses of travel and communication. Outreach to CEITs is also 50 OPS3: ProgressingToward Environmental Results insufficient, as evidenced by the confusion among countries institution for financial assistance, such as the IMF? Similarly, working toward EU accession regarding how they will be with regard to cofinancing, the GEF must consider whether its treated by the GEF upon "graduating" as a funding recipient. primary concern is the improvement of the global environ- ment, or if it is that the GEF should only cover incremental costs. Challenges and Strategic Trade-Offs The GEF communication and outreach strategy is not ade- Recommendations quate for reaching out to LDCs, SIDS, and CEITs, but how the GEF chooses to pursue (or not to pursue) an outreach strategy · Continue promoting smaller-scale projects that fit the capaci- in these countries calls into question the very nature of the ties of LDCs and SIDS. Additional funds are needed to imple- GEF: does the GEF identify itself more as an international ad- ment existing National Action Plans, build government ca- vocacy institution, such as the Joint United Nations Pro- pacity, and mainstream environmental issues in these gramme on HIV/AIDS (UNAIDS), or more as an international countries. · Augment outreach and communication efforts to facilitate accomplishment of the GEF's agenda with respect to the gen- eration of global environmental benefits, keeping in mind the needs of LDCs and SIDS. · Help LDCs, SIDS, and less developed CEITs identify external funding opportunities to better access GEF funding and/or re- duce cofinancing requirements to remove barriers. The GEF as a Catalyst 51 The Effects of the GEF Structures on Performance:The GEF Network Findings (TORs 4A, 4D, and 4G) successfully the challenges associated with managing and co- ordinating activity within a complex network. These chal- The GEF--based on its composition, structure, and division of lenges are permanent and dynamic. Over time, a network in- roles and responsibilities--has a network institutional form.15 stitution can address the challenges with increasing This form, established by the Instrument through its reliance effectiveness, but the challenges are always present. Consid- on multiple entities working collaboratively together to ac- ering the GEF in light of these challenges, which have been complish common results, seems appropriate for achieving derived from the literature on public sector networks17 and GEF goals. However, measures of network effectiveness are from the experience of the OPS3 team, OPS3 concludes that, just beginning to emerge in the literature.16 while there are a number of specific areas for improvement, The most important way to measure effectiveness of any the GEF overall is a more robust, stable, and effective institu- institution is to look at results. If an institution is producing re- tion today than it was at the time of OPS2. sults that contribute to the achievement of mission goals, the institution is effective to some degree. Other sections in this Communication and alignment of goals. The GEF has added report discuss results produced by the GEF network at the fo- guidance on strategic alignment through its Strategic Priori- cal area level and at the community and country stakeholder ties. Additionally, the OPs have achieved a degree of maturity level. and are understood clearly by the IAs. The business planning However, effectiveness can also be considered at the level process has become more standardized and provides a sense of the network itself (O'Toole 1997). GEF effectiveness as a of the programming challenges for the coming year. Work on mechanism for supporting, encouraging, planning, funding, a resource allocation framework for allocating a significant monitoring, and evaluating environmental action on a global amount of GEF resources continues. basis is associated with the GEF's overall ability to negotiate 53 However, the guidance has been additive in nature--no Evolving roles and responsibilities and managing collabora- previous guidance is changed when new guidance is added. tion and competition. Stability in a network comes from the This has yielded a framework that is too wide for strategic ac- growth in understanding and trust among partners (Milward tion. Furthermore, priorities indicate areas that are important and Provan 2003). In this regard, several representatives with- within the focal areas, but they do not constitute a "strategy" in the GEFSEC and the IAs commented on the degree of hon- that effectively guides programming choices. In the future, est, forthright conversation during task force meetings and OPS3 fully supports any GEF effort to streamline its strategic other consultations as a sign of increased trust and confi- guidance to focus on strategic goals, including guidance in dence in the other partners. OPS3 observed a fairly high de- the rather complex area of calculating incremental costs. gree of trust expressed from IAs and GEFSEC members, but trust- and transparency-related challenges remain, in particu- Coordinating partners on multiple levels and managing in- lar regarding competition versus collaboration among the IAs creasingly complex interdependence. Coordination of GEF and EAs. partners was largely done through the focal areas during the The network administration role, which has always been initial years of the GEF's existence. Over the past few years, co- the GEFSEC's responsibility, requires a more integrated ap- ordination has moved to the network level. An example here proach to administrative responsibilities than has been the is the increased coordination between the task forces and the case. OPS3's review of the literature finds that a strong net- Executive Coordinators meeting. In the past two years, the work administrative role is important for network stability and GEFSEC has established an Operations Team dedicated to effectiveness over time (Provan and Milward 2001). The role of handling administrative and management issues for the net- the GEFSEC as the network administrative office could use ad- work . OPS3 considers this an essential development in the ditional strengthening to support the continuing expansion GEF's evolution and encourages the GEF to enhance this ca- of responsibilities. pability. The country outreach program is an example of a co- The GEFSEC and the STAP have been working since OPS2 ordination function that spans all focal areas. Also, the IA and the Third Replenishment to improve and clarify the func- staffing within countries has improved the potential for effec- tioning of the group. New staff, a new roster of experts, and tive decision making and coordination at the country level. new operational rules have been developed for the STAP over However, while coordination has increased with the IAs, it is the past three years. However, despite the efforts of the STAP less clear how the GEFSEC is coordinating efforts with the EAs. (and its recent chairpersons) in trying to refine and focus the The role of the country partners, while stronger now than work of the STAP and coordinate more closely with GEF enti- in the past, continues to take on increasing importance as the ties, OPS3 found a general perception among stakeholders primary mechanism for ensuring the country-drivenness of that the STAP is still not nearly as responsive as it needs to be projects. Implementation of an RAF will heighten the need for to provide consistent value to the GEF. STAP members appear country-GEF coordination considerably, including coordina- to be keenly aware of the perception of the STAP's marginal- tion of the various IAs that have staff in the country. Also, re- ized role and commented to OPS3--as did IA, OME, and dundancies that affect efficiency do exist in various areas of GEFSEC representatives--that the STAP mandate still needs activity. For instance, many stakeholders at every level men- to be clarified and its mission redefined. tioned that project reviews over the course of the design phase by the GEFSEC, the IAs, and the GEF Council all focused on the same technical level and led to frequent, sometimes contradictory requests for design changes. 54 OPS3: ProgressingToward Environmental Results Structured informality (balance between control and em- Maintaining an inclusionary approach. The GEF has contin- powerment). The main mechanism for managing the balance ued in an incremental growth pattern in terms of partner- between control and empowerment within the GEF is the ships and responsibilities that contribute to increased vitality. project cycle, through which projects are approved and the Over the past few years, the GEF has added focal areas and majority of GEF funds are disbursed. Efforts have been made expanded opportunities to some EAs. In addition, the GEF to clean up and add discipline to the pipeline process, which has strengthened the country focal point program to help is the critical link from work programs to actual projects. countries play a more effective role. The maturation of the Progress has also been made toward distributing the moni- SGP has increased the access of smaller, national NGOs to toring function to the IAs and EAs that are responsible for GEF activities. project implementation. A study was recently concluded on An increasing number of network members place addi- the project cycle that questioned whether the design phase tional demands on the coordination capabilities of the net- of the project cycle could be shortened without affecting the work, but growth beyond the capability of the network will quality of approved projects. The length of the design phase eventually lead to frustration and inefficiency. For example, and the difficulty of knowing where projects are in the design currently, EAs are not involved in the weekly Executive Coor- process are still two of the most highly criticized aspects of dinator meetings, and participants in those meetings recog- the project cycle by GEF stakeholders at all levels. nized that including the EAs would make the meeting more difficult to schedule and manage. Logistics become more challenging with extra partners. A balance must be sought between inclusiveness and the ability of the system to coordi- nate additional partners and stakeholders. The Effects of the GEF Structures on Performance:The GEF Network 55 The capacity of the countries to work effectively within the GEF framework has also increased, although the improve- ments are somewhat mixed depending on the region and the country. Stakeholders at all levels in a range of countries still identify the lack of basic infrastructure capacity, including communication technology, as a major challenge. Language barriers can also affect the absorptive capacity of countries; al- though OPS1 and OPS2 noted that documentation should be provided in all UN languages, this has not yet transpired. As noted before, the capacity of focal points is still uneven; with some not in sync with GEF activities in their countries or not well integrated into other ministries. The loss of institutional memory resulting from high turnover of GEF Council mem- bers and focal points also threatens the stability of countries' capacity. Developing clarity in measures and outcomes. As of Novem- ber 2004, the GEF M&E Unit (GEFM&E) has been established as an independent office, with its own director. The transition to a fully functioning OME reporting directly to the GEF Council is expected in FY2008. M&E responsibilities have been adjust- ed to increase the efficiency of their execution; IAs and EAs still have responsibility for project monitoring, but the GEF Secretariat is assuming more responsibility for portfolio moni- toring. OPS3 considers monitoring the effects of changes on the GEF network itself to be of very high value. Changes in a complex network--such as those that will result from the im- plementation of an RAF--can have wide-ranging and unin- Overcoming capacity shortages. The GEF has benefited to tended effects. Only by monitoring the changes over time will date from a core of participants that have been working and the GEF be able to act in a timely fashion to intervene with re- maturing in the partnership network for a significant amount finements and corrections. This level of network monitoring, of time. These representatives understand how the GEF func- which takes place now in a slightly ad hoc manner, needs for- tions and are able to communicate effectively. Additionally, mal definition and assignment, most likely to the GEF Secre- the transfer of learning among focal areas is helpful in ampli- tariat, although OME will also play a role. fying the knowledge of GEF ways and means that already ex- ists. IAs and EAs working at the local level in countries have Managing in a permanently evolving world. At the enter- contributed to capacity at that level, although country focal prise level, the GEF has shown itself to be a continually self-re- points and NGO representatives testified to the varying abili- flective and evolving institution, as evidenced by its regular ties of the officers in these positions, many of whom were undertaking of program studies; overall performance studies, knowledgeable and well prepared, but others of whom were such as this one; and other evaluations. Often, especially with fairly inexperienced. major program studies, recommendations are turned into management actions (see the "Varying Capacities of SIDS, LDCs, and CEITs" subsection of "The GEF as Catalyst"). In addi- tion, the newly appointed Corporate Executive Officer and 56 OPS3: ProgressingToward Environmental Results Director of OME bring fresh perspectives that will likely con- hand, many stakeholders, some of whom were the same as tinue to open the GEF to new ideas. However, the STAP's role those praising strong design, acknowledged that project cir- as a liaison with the scientific community for generating and cumstances change rapidly and lead to significant changes in introducing new ideas needs to be strengthened. the project plan during implementation. One repeated sug- In many cases, the major effects of changing circum- gestion heard by OPS3 was that the logical framework should stances are on the projects themselves. Recent studies have be simplified, with goals and objectives--including cofinanc- highlighted difficulties associated with project inflexibility and ing--defined completely only during implementation. have stressed that greater flexibility and room for innovation is needed to allow projects to achieve optimal results in the IA and EA collaboration versus competition. OPS3 found that face of changing circumstances. These reviews have empha- roles and responsibilities were not always clear for IAs and sized the need to clarify policies and procedures to encourage EAs, especially with regard to collaboration and competition. adaptive management of projects (GEF/C.24/Inf.5. 2004). On On the one hand, IAs are aware of their stated comparative the other hand, adaptive management cannot be a substitute advantage, and OPS3's review of the project portfolio across for effective project planning during the design phase. all focal areas found that the majority of projects in the pipeline are well aligned with these comparative advantages. Maintaining effective relations with external stakeholders. However, the OPS3 review also found that there were a num- In the FY2005­07 Business Plan, the GEFSEC proposed the de- ber of projects for which it was not possible to discern from velopment of a Communications Strategy to bring coherence looking at the characteristics of the project why a particular IA to the activities currently undertaken by the Secretariat and was the implementer of record. OPS3 also heard testimony the IAs. The monthly GEF newsletter is distributed to a much from the IAs themselves and other stakeholders, that compe- wider audience now, including NGOs and other stakeholders. tition for projects and resources was forcing IAs to look ever The GEF still needs an aggressive outreach campaign, how- wider for projects and investigate new lines of business to ever, to broadcast GEF accomplishments both to external and support their sustained growth, even when those projects internal stakeholders, especially those stakeholders such as crossed over into the comparative advantage of one of the the NGOs that are nodes away from the source of the commu- other IAs. This tendency to blur the boundaries of the IAs' nication. OPS3 field visits revealed that some NGOs working roles is further exacerbated by the addition of the EAs that on GEF projects--especially smaller, national NGOs--were must find their way within the GEF project context. EAs have not even aware that the GEF was a source of their project's an uncertain mandate and a large learning curve to climb in funding. There are many in the global environmental commu- order to function competitively in the GEF "market." In fact, nity that have only a vague idea of the GEF and its role. only four of the seven EAs with expanded opportunities have signed an MOU with the GEF that officially sanctions their ability to implement projects solely. The fact that EAs are the Challenges and Strategic Trade-Offs lead on only 38 of the more than 1,500 projects implemented by the GEF further underscores the nascent aspect of their in- Goal definition and structural stability versus adaptive flexi- volvement and speaks to the competitive playing field and bility. The logical framework is structured to guarantee that dwindling funds under GEF-3. GEF projects clearly define their goals and support the GEF operational principles. Getting a project approved is often a long process precisely because of the hard work done during this phase. The results of this effort, according to the testimo- ny of many stakeholders throughout the network, are well- thought-out projects that are stable and strong. On the other The Effects of the GEF Structures on Performance:The GEF Network 57 At the same time that competition is, in some cases, strain- Inclusiveness versus cost effectiveness and network capaci- ing the trust among corporate entities; collaboration among ty. The GEF operational principles require that "[T]he GEF... project proponents, including IAs and EAs, is being fostered ensure the cost-effectiveness of its activities" and at the same by the GEF as a means to improved functioning (and cost ef- time "provide for consultation with, and participation as ap- fectiveness) and is specified in the "Instrument for the Estab- propriate of, the beneficiaries and affected groups of people." lishment of the Restructured Global Environment Facility" The degree of inclusivity required for GEF projects can easily (GEF 1994). In discussions with the GEFSEC and the IAs, it was lead to extra time and cost in the preparation and execution clearly stated to OPS3 that unlimited competition will be at of projects. odds with collaboration. For example, implementing projects jointly and the associated fee sharing implies collaboration, while competition implies developing and implementing Recommendations projects in a wholly owned manner with a single manager claiming the entire fee. Left to their own devices, the IAs will · The GEF should strengthen the role of the GEF Secretariat as not likely be able to solve the equation between competition the network administrative office. The GEFSEC already serves and collaboration on projects effectively. Additionally, under a the function of network administrator, but in a mostly as- full competition scenario, IAs may be less willing to fulfill their needed, ad hoc way. However, the network administrative of- GEF corporate responsibilities. fice role includes consciously managing all the challenges as- sociated with a complex network. OPS3 encourages the GEF Council to acknowledge the critical contribution of the net- work administrative office within the GEF and to adjust re- sources where necessary to allow the GEFSEC to manage net- work challenges in a more comprehensive and strategic way. 58 OPS3: ProgressingToward Environmental Results To that end, the GEFSEC, as the network administrative office, · The role of the STAP, currently a marginalized GEF partner, should consider formalizing the following organizational should be revisited, and structural and organizational functions: changes should be undertaken to allow the STAP to provide documentation in a timely fashion. Specifically, the STAP · Communication, coordination, and outreach--cover- must be realigned to ensure that it can function as the liaison ing communication with all GEF partners in relation to ca- with the external scientific community. Additionally, the STAP pacity and coordination, including country partner capac- should be responsible for choosing experts from the roster for ity; communication, coordination, and outreach with project review and must undergo structural and organiza- other partners, including NGOs and the private sector; and tional changes to enable it to provide relevant reports in a external entity outreach. timely manner. · Management, information, and policy--encompassing · OME must continue to evolve the understanding of M&E with- the following functions: policy and planning, work plan in the GEF and international organizations. To accomplish programming, information management strategies and this, OME should formalize the consultative process, while systems, knowledge management and communities of OME, the GEF Secretariat, and the IAs and EAs should ensure practice coordination, and project cycle management. that M&E are covered at all levels (for example, at the project, These functions consolidate and focus efforts that are current- portfolio, country, network, and even at the global environ- ly under way within the GEFSEC. The recommended functions mental management practice levels). Finally, OME should en- do not eliminate any current responsibilities of the GEFSEC in deavor to ensure that its evaluations are easily "digestible," leading the focal areas, communicating with the conventions, actionable, and relevant to stakeholders. and supporting the Council and the Assembly, although real- location of resources among functions may be required. · The GEFSEC should work with the IAs and EAs to clarify roles and responsibilities and work through the competition and collaboration challenge--an issue that can seriously affect the quality of GEF results during GEF-4 if it is not managed ef- fectively and proactively. Because there are already disincen- tives to collaborate, including competition for resources and projects, and there is still poor transparency and less than per- fect trust in the system, it is essential that the GEFSEC take more of a leadership role in enunciating the positioning of collaboration and competition in the system. OPS3 recom- mends an ongoing dialogue among the GEFSEC, IAs, and EAs to voice issues on the advantages and disadvantages of, and ways to optimize, the competition versus collaboration nexus. This dialogue could, for example, be in the form of a regularly scheduled workshop or contact group that convenes prior to Council meetings. The Effects of the GEF Structures on Performance:The GEF Network 59 GEF Procedures T his section treats the processes, procedures, and sys- Pipeline. The pipeline is the GEF's tool for work program tems required for two critical aspects of GEF activity: tracking. All concepts for GEF projects have to be reviewed the project lifecycle and knowledge and information and entered into the GEF pipeline prior to further preparation management. and Council review for work program inclusion. Because of time lags between pipeline entry and inclusion in work pro- grams, many projects currently entering the pipeline in GEF-3 GEF Project Cycle (TOR 5A) will not be included in work programs until GEF-4. In fact, the FY2005­06 Business Plan (GEF/C.24/9/Rev.1 2004) indicated Findings that almost US$300 million worth of concepts would fall into this category. The GEFSEC is currently initiating a process to Since its introduction in 1995, the GEF project cycle has been manage the pipeline more closely, including possibly requir- a guide to project designers and implementers, a focus of ing projects to have an indication of which work program evaluation in the GEF, and a source of both achievement and they will be included in or identifying a maximum number of frustration for many GEF stakeholders. Both OPS1 and OPS2 years that a project can remain in the pipeline. A proposed recognized the efforts of the GEF entities to streamline the time horizon (perhaps three years as the standard) would be project cycle; however, there is wide recognition within the imposed for all projects in the pipeline, unless a strong argu- GEF family that significant improvement is critical for ensuring ment based on the difficulty or complexity of the project a successful future for the GEF. start-up activities is provided. 61 Design phase. The design phase typically draws the most fire ground, and IWPS2004 observed a "move toward projects from stakeholders who feel that the phase takes too long, is that combine strategic planning with demonstration projects nontransparent, and requires too much specialized expertise to maintain stakeholder interest and articulate the adaptive to write a design document that meets all of the GEF require- management process." ments. The design phase does have notable aspects, includ- To be effective, however, adaptive management needs to ing the following: be based on a good monitoring system that provides infor- mation a manager can use to make informed decisions. De- · Project stakeholders are extensively consulted during the spite many requests, OPS3 could not find consistent evidence design phase. that such clear information exists. Therefore, monitoring at · Comments from the Council and the GEFSEC are helpful at the project level should be strengthened to serve as an input the concept level. for mid-course correction and for tracking changes in the ini- tial project baseline over time. · There is a clear appreciation among GEF partners of the concern with the project cycle at the design phase, and ef- Modalities. The GEF has approved expedited procedures for forts are being undertaken to conduct further study and approval and disbursement for several modalities (such as flow those findings into action steps. MSPs and the SGP) and is currently considering instituting an- An unintended consequence of the GEF's attempts to im- other modality--mid-size projects between US$100,000 and prove the design of projects by providing more guidance on US$500,000. The GEF has met its recent targets for increasing project development is the additive nature of that guidance. the number of countries that participate in the SGP and these This proliferation of guidance appears to cause some confu- projects may be the most easily replicable types of projects. sion among those involved in developing projects, both at The SGP modality is well received by recipient countries and the IA and country level. Additionally, the number of compli- increases the visibility of the GEF; however, stakeholders have cated items to be addressed in project design leads to a situa- pointed out that the scale of SGP projects may lead to the re- tion in which many project proposals are developed by exter- alization of limited global environmental benefits. Conversely, nal consultants and specialists who understand the FSPs are designed to maximize the realization of global envi- increasingly complex criteria involved with GEF proposal writ- ronmental benefits, but these projects tend to take the ing, but who may not understand the nuances of a particular longest time to design and approve. Additionally, not all re- country culture. cipient countries can support FSPs in terms of need or country By encouraging the use of adaptive management tech- capacity. MSPs were designed to ameliorate some of the criti- niques, the GEF has acknowledged that the conditions under cisms directed at FSPs by reducing project scope and stream- which a project is designed and those under which it is imple- lining the approval process, but according to the GEF Annual mented may change and that management must adapt. Cur- Performance Report (GEF/ME/C.25/1 2004), the time lag for rently, however, policies and procedures such as the logical approval of these projects is well beyond what was originally framework, incremental cost calculations, and cofinancing re- expected. Indeed, stakeholders interviewed by OPS3 noted quirements do not reflect the variable and adaptive stage of that MSPs were being subjected to almost the same degree of implementation. scrutiny as FSPs. In spite of the strengths and weaknesses of these modalities, they enable the GEF to respond to the di- Implementation phase. Operational Principle 5 requires that verse needs of its stakeholders. Moreover, the GEF has the GEF "maintain sufficient flexibility to respond to changing demonstrated a commitment to evaluating these modalities circumstances...." In the project realm, this principle has led constantly for effectiveness and improvements. to the support of "adaptive management"18 techniques during implementation. The 2004 Program Studies cited adaptive management as providing flexibility to implementers on the 62 OPS3: ProgressingToward Environmental Results Challenges and Strategic Trade-Offs the cycle, and it appears that available funding in various OPs is as much a driver of project development as are country pri- Risk aversion versus innovation. The tension between GEF's orities or results. In addition, the generally additive nature of commitment to both innovative approaches that inherently guidance over time has led to a substantial focus by IAs on involve risk and to ensuring project results that inherently in- following approval procedures. volve minimizing risk is discussed in "Challenges and Strategic Trade-Offs" in the "Catalytic Impacts" subsection of "The GEF Adaptive management and maintenance of the accounta- as a Catalyst." Consultations with both GEFSEC and OME staff bility chain. There is a dynamic tension between detailed indicated that this issue of emphasis across all GEF programs project design in the logical framework and the need for proj- is a high-level strategic issue that has yet to be resolved. ect managers to have flexibility during project implementa- tion to adjust project elements (staff, resources, goals, and so Approval culture versus results culture. OPS2 noted that the on) as required to meet changing circumstances. The GEF en- GEF should be heading more in the direction of a results-ori- courages managers to use flexibility in implementing projects ented culture than an approval culture. As pointed out else- through the use of "adaptive management"19 techniques. where in this report, a results culture is not fully in evidence at However, often there is only a limited record of any changes this time. While moves have been made to shift focus to re- in a revised project plan, if indeed the revised plan exists at all. sults, and to better assess baselines and indicators for results, This leads to difficulties in tracking the achievement of project OPS3 noted that the emphasis among key stakeholders such results, if there is an MIS to permit such analysis. Furthermore, as IAs and their recipient country counterparts is on the ap- an adaptive management approach must not become a sub- proval element of the project cycle. Considerable time and re- stitute for effective and in-depth project design. sources are spent during the design and approval phase of GEF Procedures 63 Recommendations · OME should ensure that the monitoring tools of the IAs and EAs allow them to effectively manage projects in an adaptive way. Also, the GEF's project guidance for the design phase should be rethought in the spirit of adaptive management. OPS3 recommends that the GEFSEC develop better guide- lines, in consultation with the IAs, to back up the concept of adaptive management--there should be genuine flexibility and recognition of the need to avoid delays in mid-course cor- rections. The following aspects of project design could poten- tially be affected: · Logical framework--While setting out the goals and ob- jectives of the project, some of the detail required in the logical framework may be transferred to implementation. · Cofinancing--Building more flexibility into the timing of cofinancing could actually improve overall levels of cofi- nancing. It may be easier to secure cofinancing during im- plementation when the project has some tangible life of its own. · Incremental costs--Streamlined guidance on incremen- tal costs could aid in the more rapid design of projects and could have a significant impact on the value of projects proposed. The Information Management Challenge for the Lessons learned and knowledge management. Lessons GEF (TOR 5B) learned are identified through semiformal mechanisms, in- cluding Annual Project Implementation Reviews (PIRs); Pro- Modern networks are absolutely dependent on a solid, stable ject Performance Reports (PPRs), which have been replaced information management infrastructure that helps to moni- by the recent Annual Performance Review; terminal and final tor, improve, inform, and direct the network. The goal of net- project evaluations, and Terminal Evaluation Reviews (TERs) work administration is to ensure that information is transpar- performed by OME. Additionally, as BPS2004 pointed out, ent--clear, complete, easily accessible, on time, and "opportunities for institutional lesson learning and direct in- accurate--for all partners, within the constraints of reason- corporation within the GEF Secretariat and the IAs" exist in the able information security. This requirement aligns directly Inter-Agency Task Forces. OPS3 finds that the Executive Coor- with Operational Principle 6, which calls for the "full disclosure dinators' meeting is a mechanism for exchanging lessons of all non-confidential information." Part of being able to dis- learned on multiple aspects of the project lifecycle. UNDP has close information is having the ability to identify, collect, and a system that allows project proponents to post questions disseminate the right types of information to meet the needs and receive answers from various participants across the of the network. In that regard, the GEF is significantly defi- globe--and several focal areas, particularly International cient. This section will address this issue of information trans- Waters, have developed systems to support knowledge shar- parency in two areas--lesson learning and knowledge shar- ing. The IAs have collected and published lessons learned on ing, and MIS. 64 OPS3: ProgressingToward Environmental Results specific topics (for example, the UNDP publication on solar PV Management information systems (MIS). The establishment projects in Africa). In addition to these somewhat formal of a GEF Project Tracking and Management Information Sys- mechanisms, lessons learned at the project level are identified tem (PMIS) came about because it was felt that the GEF's in a sporadic and ad hoc manner, in small groups, on projects, "unique structure and the diverse, open, and transparent part- and so on. nerships" required new management techniques, including The GEF has incorporated some of these lessons learned modern information technologies (GEF/C.14/9 1999). into project design and implementation. BPS2004 noted that However, OPS3 finds that serious problems with MIS still the GEF had or is incorporating findings and recommenda- exist. No MIS captures project information effectively tions on issues such as "stakeholder participation [and] the throughout the lifecycle and makes that information available improvement of linkages with other sectors of the economy." to GEF partners regularly. General information management The GEF has shown itself to be effective in generating and in- at the GEF has been lacking since its inception and makes ac- corporating lessons learned through the ongoing review and curate monitoring of GEF activities at the portfolio level very amendments of the project lifecycle. GEFM&E has completed difficult. Stakeholders at all levels of the GEF partnership com- more than five major evaluations since OPS2, including three mented that the PMIS maintained by the GEF Secretariat is an program studies. In terms of incorporating lessons learned, approval system only and is inadequate to meet the manage- Action Plans are developed to respond to the recommenda- ment and monitoring needs of the GEF. Many high-level GEF tions of the OPSs, Replenishments, and Assembly, and other staff felt that the PMIS could not even be considered a man- evaluations, and progress on implementing the recommen- agement information system in its current form. dations is periodically assessed. The direction that the OME is The lack of information transparency is networkwide. De- taking with respect to harmonizing and standardizing M&E, spite the recommendation of OPS2, very little information is the formation of an Operation Coordination Team in the available to recipient country stakeholders regarding where GEFSEC, and the re-formation of the Country Dialogue Work- their projects are in the project cycle. Moreover, CCPS2004 shops into the NDIs, are all signs that members of the GEF net- found that "the GEF database is not an analytical tool accessi- work identify lessons and integrate them into future activity. ble to parties outside the GEF Secretariat, updating is irregular, However, despite the evidence of the GEF as a learning or- it has limitations in data on results, and data inconsistencies ganization, there seems to be broad consensus among stake- between GEF and IA databases are frequent. This function is holders--OPS3 heard similar comments in this regard from seriously under-resourced in the GEF Secretariat." The absence GEFSEC personnel, from the IAs, especially at the country lev- of a well-managed and comprehensive MIS in the GEF has crit- el, from the country focal points, and from many NGOs--that ical implications for the GEF network. As CCPS2004 also com- more learning can be done more systematically. Recent evalu- mented, the shift from an approval culture to a result and ations have highlighted the inadequacy of current processes quality orientation, recommended by OPS2, would "remain for capturing lessons learned and the need to systematize elusive as long as it is so difficult for any stakeholder to gain a what now takes place within limited areas. Given that there full overview of what is going on in the portfolio at any given has not been an adequate systematic process for capturing time." The current inability of the GEF to monitor its portfolio lessons learned over time, there is a real risk that substantial at a macro level inhibits the development of strategic vision- lessons learned and capacity, as well as institutional knowl- ing and planning. Indeed, OPS3 itself has struggled to provide edge among individuals, will be lost if it is not recorded. an analysis of results in the focal areas and of the GEF portfolio In May 2004, the Council acknowledged KM as a corporate as a consequence of the inadequate MIS in the GEF. GEF task to be led by the GEFSEC with the support of the IAs by approving the FY2005 Corporate Budget (GEF/C.23/9), which included US$0.49 million for a KM specialist position in the GEF Secretariat's budget. This position, however, has not yet been filled. GEF Procedures 65 Recommendations · Trends and challenges affecting the GEF in information management; · The GEFSEC should develop an overall information manage- ment function for the GEF that encompasses both KM and · Assessment of the strengths and weaknesses of the current MIS functions; this information management function should information management implementation; build on existing systems in the GEF entities and be based on · Plans for significantly improving the GEF MIS and knowl- a comprehensive information management strategy to be de- edge-sharing systems; veloped by the GEFSEC. To address the current inadequacy of both the KM and MIS · Technical description and principles for a GEF-wide techni- functions in the GEF, OPS3 recommends that the GEF estab- cal architecture that would support the goals and objec- lish a formal function for Information Management in the tives of the MIS and knowledge-sharing systems;21 GEFSEC. This function would be responsible for KM and MIS · Any organizational processes, staff structure, and GEF cul- with the bottom-line goal of transparency of information at ture changes needed to effectively implement MIS and KM, all levels of the GEF partnership network. The GEF should give including quality assurance and content management this function appropriate time and resources, make it prag- procedures to ensure that information is accurate, appli- matic by building on existing KM and MIS systems in the GEF cable, and current; and entities (such as, UNEP.net, UNDP GEF portal, World Bank knowledge management system), and, in pushing forward · Performance measures and milestones to assess the KM and lessons learned, make sure that adequate time is giv- progress of the information management function. en to both the capture and, even more importantly, the dis- The execution of the information management strategy semination and delivery of that information to its appropriate should lead to the implementation of the following: targets. As a first step in the improvement process, the GEFSEC · An institutionwide MIS that makes available information should mount a focused effort with the IAs to update data on the status of projects at every stage from pipeline entry that already exist in the current systems so that the latest and to completion. This MIS should be accessible to all project most accurate data are available. The KM and the MIS sys- proponents, including operational focal points, so that tems should build on and meld the existing, but currently in- they are able to track their (and other) projects through compatible, information systems of the GEF entities into a the various stages of the project cycle, thereby improving useful GEF-wide resource. At the same time, the GEFSEC country ownership. This project-level information would should begin the development of an information manage- also be aggregated and analyzed to aid the process of col- ment strategy20 that will guide the long-term improvement lecting and measuring results, determining cost-effective- and overall quality of the KM and MIS systems, and their sup- ness, and comparing and evaluating project results. The porting business processes. The strategy should broadly ad- data in the MIS should also reflect actual situations in the dress all aspects of information and knowledge management GEF rather than expected outcomes (such as cofinancing and contain at least the following points: at the conclusion of projects in addition to at approval). · A knowledge-sharing infrastructure designed to support the capture and dissemination of lessons learned and the exchange of information and knowledge at all levels and for specific communities of practice and interest within the GEF network. 66 OPS3: ProgressingToward Environmental Results References GEF Documents Other Documents BPS2004. See GEF/ME/C.24/Inf.1 (2004). Goldsmith, Stephen, and William D. Eggers. 2004. "Governing by CCPS2004. See GEF/ME/C.24/Inf.2 (2004). Network: The New Shape of the Public Sector." Washington, DC: Brookings Institution Press. GEF. 1994. "Instrument for the Establishment of the Restructured Global Environment Facility." IUCN. 2003. "2003 United Nations List of Protected Areas." Cam- bridge, U.K.: IUCN and UNEP WCMC. GEF/C.7/Inf.5. 1996. "Incremental Costs." February. Milward, H. Brinton, and Keith G. Provan. 2003. "Managing Networks GEF/C.14/9. 1999. "FY01­FY03 Corporate Business Plan." November. Effectively." Paper presented at the National Public Management GEF/C.22/6. 2003. "GEF Business Plan for FY05­07." October. Research Conference, Georgetown University, Washington, D.C. GEF/C.23/4. 2004. "Terms of Reference for the Third Overall Perfor- October. mance Study of the GEF." May. Morgan, Gareth. 1986. Images of Organization. Beverly Hills, Calif.: GEF/C.23/Inf.4. 2004. "Review of GEF's Engagement with the Private Sage. Sector." Prepared by the GEFM&E. April. O'Toole, Laurence J., Jr. 1997. "Treating Networks Seriously: Practical GEF/C.24/Inf.5. 2004. "GEF Project Cycle Update: Clarification of Poli- and Research-Based Agendas in Public Administration." Public cies and Procedures for Project Amendments and Drop/Cancella- Administration Review 57(1): 45­52. tions." October. Provan, Keith, and Brinton Milward. 1995. "A Preliminary Theory of In- GEF/C.24/Inf.7. 2004. "Elements to Be Taken into Account in Funding terorganizational Network Effectiveness: A Comparative Study of the Implementation of NAPAs under the LDC Fund." October. Four Community Mental Health Systems." Administrative Science GEF/C.24/9/Rev.1. 2004. "GEF Business Plan FY05­06." October. Quarterly 40(1): 1­33. GEF/C.24/6/Rev.2. 2005. "Scope and Coherence of the Land Degrada- ________. 2001. "Do Networks Really Work? A Framework for Evaluat- tion Activities in the GEF." May. ing Public-Sector Organizational Networks." Public Administration Review 61(4): 414­23. GEFM&E. 2002. "Monitoring and Evaluation Indicators for GEF Inter- national Waters Projects: M&E Working Paper 10." November. Rank, Olaf, and Andreas Wald. 2000. "A Methodological Framework for the Analysis of Network Organizations." 26th EIBA Confer- GEF/ME/C.24/Inf.1. 2004. "GEF Biodiversity Program Study 2004 ence, Maastricht (Netherlands). (BPS2004)." September. Reinicke, Wolfgang H., and Francis Deng. 2000. Critical Choices: The GEF/ME/C.24/Inf.2. 2004. "GEF Climate Change Program Study 2004 United Nations, Networks, and the Future of Global Governance. Ot- (CCPS2004)." September. tawa, Canada: International Development Research Centre. GEF/ME/C.24/Inf.3. 2004. "GEF Program Study on International Waters UNEP. n.d. "IPCC/TEAP Special Report. Safeguarding the Ozone Layer 2004 (IWPS2004)." October. and the Global Climate System: Issues Related to Hydrofluorocar- GEF/ME/C.25/1. 2004. "GEF Annual Performance Report." bons and Perfluorocarbons." Available at GEF/R.3/38. 2000. "Third Replenishment Agreement." October 2000. http://www.ipcc.ch/press/SPM.pdf. GEF/R.4/7. 2005. "GEF-4 Programming Document." June. Wells, Michael G., Mehreen Hosain, Bolaji Ogunseye, and Julio C. Tre- IWPS2004. See GEF/ME/C.24/Inf.3 (2004). sierra. 2003. "Third Independent Evaluation of the GEF Small Grants Programme." GEF Small Grants Programme, New York. World Resources Institute. 2004. "Climate Data: Insights and Observa- Convention Documents tions." Presentation by Jonathan Pershing at a COP 10 Side Event, Buenos Aires. December 13. UNEP. 2002. CBD COP Document UNEP/CBD/COP/6/INF/4. 2002. "Se- cond Review of the Financial Mechanism for the CBD." Available at http://www.biodiv.org/doc/meeting.aspx?lg=0&mtg=cop- 06&tab=1. 67 Notes 1. Although the exact nature of the RAF is yet to be determined, 11. This estimate is approximate since the basis for assessing such a system will necessitate the development of processes at all appraised phaseout may vary among subprojects. levels, including structures for decision making, M&E, outreach, 12. UNEP (2002). and administration. It is with this in mind that OPS3 makes recom- 13. The NDI aims to achieve greater mainstreaming of GEF activities mendations about the RAF. The need for such systems to support into national planning frameworks and coordination and syner- the RAF does not depend on the exact nature of the RAF, and gies among the GEF focal areas at the national level through a OPS3 makes recommendations regarding the RAF at this abstract multistakeholder dialogue process. level. 14. The CAS establishes a program of support linked to the country's 2. This report uses O'Toole's (1997, 45) definition of networks as own development strategy and to the Bank Group's own com- "structures of interdependence involving multiple hierarchical parative advantage in the context of other donor activities. arrangement." Forms of network may include coalitions, al- liances, partnerships, and so on. Rank and Wald (2000, 3) define 15. O'Toole's (1997, 45) definition of networks is used here (see note a network as "a well-defined, persistent, and structured set of 2). See also Rank and Wald (2000, 3), as mentioned in note 2. semi-autonomous corporate actors engaged in numerous mutu- Unfortunately, it does not appear from the literature that a well- al exchange relationships in order to jointly reach the common accepted topology of these subtypes exists. Therefore, OPS3 will network objectives." treat the GEF as a network in the general sense outlined by O'Toole (1997). Indeed, it is at this general level that most of the 3. BPS2004 found that, of the projects assessed that reported literature and research seems to be addressed. achievements regarding the overall likelihood of sustainability, medium-size projects (MSPs) outnumber full-size projects (FSPs) 16. A network institution requires a different set of evaluative criteria by approximately two to one, and FSPs outnumber MSPs approx- than does a hierarchical organization. See Provan and Milward imately two to one for projects that reported shortcomings on (1995, 2001); Rank and Wald (2000). sustainability. 17. Network challenges have been compiled and adapted principally 4. Multicountry projects include more than one country, whereas from Morgan (1986); Rank and Wald (2000); Provan and Milward regional projects span more than one region and global projects (1995, 2001); Milward and Provan (2003); Reinicke and Deng span more than two regions. (2000); Goldsmith and Eggers (2004). 5. Statistics such as these reported below are based on the cohort 18. Adaptive management is defined as "accommodating changes in of projects reviewed by BPS2004. project design and implementation to changes in context (imple- mentation environment), if any, with the overall objective of 6. Direct reduction is defined as "tangible carbon dioxide reduc- meeting project goals and objectives" in GEF/C.24/Inf.5 (2004). tions directly attributable to specific project activities and the lifetime of technology promoted by the project," while indirect 19. See note 18. reduction is the estimated replication effect catalyzed by the GEF 20. The GEFSEC is currently undertaking a requirements analysis for intervention. MIS. OPS3 believes this study is necessary but not sufficient to 7. While good impacts and outcomes are associated with the Black build a full-dimensioned information management function. The Sea­Danube project, the reduction of nitrogen runoffs is partially information management plan described herein will provide a a result of the collapse of the former Soviet Union. more comprehensive framework for information management within which the MIS requirements analysis will fit. OPS3 sees no 8. GEF management noted that the project-level indicators that problem with developing these activities separately, as long as were included in M&E Working Paper 10 (GEFM&E 2002) could the MIS requirements analysis is developed with the awareness be used as indicators of progress in International Waters projects. that this larger context is necessary and is under consideration. 9. Annex A and B substances include CFCs, halons, carbon tetra- 21. The technical architecture will have to be developed based on chloride, and methyl chloroform. World Bank support, and it will take into account connectivity 10. Appraised ODS phaseout is the amount of ODS phaseout that with architecture that already exists within the partners. each country's respective GEF project is expected to phase out. 69 Annex 1: Clarification of OPS3 Terms of Reference OPS3 has interpreted many of the TOR questions for clarifica- TOR question 1E: Historically, how have GEF resources been tion purposes. These interpreted TOR questions will serve as allocated geographically, and is this allocation consistent with OPS3's working definition of the TOR. The original TORs and strategic priorities? the interpreted language are provided below. · No interpretation required. TOR Question 1: Operational and Program Results TOR Question 2: Sustainability of Results TOR question 1A: What have been the quantitative and quali- TOR question 2A: To what extent have desired global environ- tative impacts and results of GEF activities at the local, region- mental benefits continued following completion of GEF al, and global levels in the areas of biodiversity, climate projects? change, international waters, and ozone depletion? · No interpretation required. · OPS3 interprets the words "impacts and results" as "re- sults" for results to be consistent with the definition of TOR question 2B: What are the key factors that determine the results provided in the original TOR. This change has been sustainability of GEF projects? made because results are defined as "outputs, outcomes, · OPS3 interprets this question to be asking about the sus- and impacts" in the original TOR.1 tainability of global environmental benefits, rather than TOR question 1B: If impacts and other results are not quantifi- sustainability of GEF projects. able, what are the reasons? TOR question 2C: To what extent do country ownership, · OPS3 interprets the words "impacts and results" as "re- stakeholder involvement in project development and execu- sults" for results to be consistent with the definition of tion, and the generation of local benefits improve the sustain- results in the original TOR (see discussion for TOR question ability of activities supported through the GEF? 1A above). · OPS3 interprets this question to be focused on sustainabili- TOR question 1C: Do projects developed under the new focal ty of results, rather than on sustainability of activities. areas of land degradation and persistent organic pollutants reflect global priorities? TOR Question 3: Effects of GEF Operations · No interpretation required. on Other Institutions and Related Issues TOR question 1D: What are the key factors that have con- tributed to the achievement of global environmental benefits? TOR question 3A: How successful has the GEF been in fulfilling its catalytic role by leveraging additional resources; catalyzing · No interpretation required. results by innovation, demonstration, and replication; foster- ing international cooperation on environmental issues; main- streaming environmental issues into partner institutions; and involving the private sector in both projects and cofinancing? 71 · OPS3 interprets this question to be focused on four dis- TOR question 4E: Are the GEFSEC and its partner agencies tinct areas, including (1) leveraging additional resources effectively responding to national priorities? from public and private sectors; (2) catalyzing results by · OPS3 interprets this question to focus on the national innovation, demonstration, and replication; (3) fostering priorities of recipient countries. international cooperation on environmental issues; and (4) mainstreaming environmental issues into partner institu- TOR question 4F: Is the GEF taking into account the varying tions. Based on additional discussion of scope in the TORs, capacities of countries including, for example, SIDS, LDCs, and the first and the last portions of this question were com- emerging economies? bined into item (1). · OPS3 interprets "emerging economies" to mean CEITs. TOR question 3B: What are the key areas that lead to catalytic TOR question 4G: How effective has the M&E unit been, and impacts, and what issues need to be addressed to improve how effective has the M&E process been? catalytic impacts? · No interpretation required. · OPS3 interprets this question to be asking about key fac- tors, rather than key areas. TOR Question 5: Effects of GEF Implementation Processes TOR Question 4: Effects of the GEF's Institutional Structure and Procedures on Results TOR question 5A: What are the factors that influence perform- ance at all stages of the GEF project cycle? TOR question 4A: Are the GEF entities--the IAs and EAs, the GEFSEC, the STAP, and the Trustee--performing their respec- · No interpretation required. tive functions in a satisfactory, cost-effective, and responsive TOR question 5B: Have lessons learned and feedback been ad- manner? equately integrated into project design and implementation? · No interpretation required. · No interpretation required. TOR question 4B: Are there conclusions that can be drawn with TOR question 5C: What progress has been made on the imple- respect to cost effectiveness and responsiveness of the GEF mentation of key policy recommendations from the GEF projects in comparison to similar international institutions? Council? · No interpretation required. · No interpretation required. TOR question 4C. Are the GEF's policies and programs ade- quately responding to the objectives of the conventions to which it serves as a financial mechanism? Note · No interpretation required. 1. "Results are defined as the outputs, outcomes and impacts TOR question 4D: Are the GEF's composition, structure, and achieved by the implementation of projects and programs. These should include the assessment of both positive and nega- division of roles and responsibilities effective in meeting its tive outputs, outcomes and impacts that are both intended and mandate and operations? unintended." (Transcribed from footnote 2 of OPS3 TORs.) · No interpretation required. 72 OPS3: ProgressingToward Environmental Results Annex 2: Complete Terms of Reference for the Third Overall Performance Study of the GEF (Approved by the GEF Council on May 21, 2004) Introduction to the Global Environment Facility 4. During its first decade, the GEF has provided about $4.5 (GEF) billion in funding for almost 1,200 projects in more than 140 countries. The Third Replenishment of the GEF Trust Fund 1. The mission of the Global Environment Facility (GEF) as provided an additional $3 billion for the period July presented in the Instrument for the Establishment of the 2002­June 2006, the third phase of the GEF (GEF3), and is tied Restructured GEF and as amended by the Second GEF Assem- to an agreement on a set of policy recommendations includ- bly is: ing a set of targets to be achieved by November 2004. The GEF shall operate, on the basis of collaboration and part- nership among the Implementing Agencies, as a mechanism for 5. The GEF underwent an independent evaluation of its Pilot international cooperation for the purpose of providing new and Phase in 1993. Overall evaluations of the restructured GEF additional grant and concessional funding to meet the agreed were conducted in 1997 and 2001. All of these evaluations incremental costs of measures to achieve agreed global environ- contributed to the decision-making processes of the GEF Re- mental benefits in the following focal areas: climate change, bio- plenishment and Assembly. The GEF Council has decided that logical diversity, international waters, ozone layer depletion, land a Third Overall Performance Study (OPS3) of the GEF should degradation, primarily desertification and deforestation, and be independent and external, and that it should be complet- persistent organic pollutants (POPs). ed in time to inform negotiations of the Fourth GEF Replen- ishment of Funds expected to begin in 2005. 2. The GEF is governed by the GEF Council, which is serviced by the GEF Secretariat. It has three GEF Implementing Agen- cies (UNEP, UNDP, the World Bank), seven Executing Objectives of OPS3 Agencies1, and a Scientific and Technical Advisory Panel (STAP). The GEF works in close collaboration with national 6. The purpose of the Third Study of GEF's Overall Perfor- governments and also cooperates with NGOs, civil society and mance (OPS3) is to assess the extent to which GEF has the private sector. achieved, or is on its way towards achieving its main objec- tives, as laid down in the GEF Instrument and subsequent de- 3. The GEF is a financial mechanism for the UN Convention cisions by the GEF Council and the Assembly, including key on Biological Diversity, the UN Framework Convention on Cli- documents such as the Operational Strategy and the Policy mate Change, the Stockholm Convention on Persistent Or- Recommendations agreed as part of the Third Replenishment ganic Pollutants and the UN Convention on Desertification. of the GEF Trust Fund. As the portfolio continues to mature, OPS3 will focus more than its predecessors on program and project outcomes, the sustainability and replicability of those outcomes and the move towards impacts. There are five main topics for the study, namely: 73 a) Operational and Program Results:2 What have been the partner agencies effectively responding to national priori- quantitative and qualitative impacts and results of GEF ac- ties? Is the GEF taking into account the varying capacities tivities at the local, regional and global level in the areas of of countries including for example small island developing biodiversity, climate change, international waters and states (SIDS), least developed countries (LDCs), emerging ozone depletion? If impacts and other results are not economies? How effective has the Monitoring and Evalua- quantifiable, what are the reasons? Do projects developed tion (M&E) Unit been and how effective has the process of under the new focal areas of land degradation and persist- monitoring and evaluation been? ent organic pollutants reflect global priorities? What are e) Effects of GEF Implementation Processes: What are the fac- the key factors that have contributed to the achievement tors that influence performance at all stages of the GEF of global environmental benefits? Historically, how have project cycle? Have lessons learned and feedback been ad- GEF resources been allocated geographically and is this al- equately integrated into project design and implementa- location consistent with strategic priorities? tion? What progress has been made on the implementa- b) Sustainability of Results: To what extent have desired global tion of key policy recommendations from Council? environmental benefits continued following completion of GEF projects? What are the key factors that determine the 7. Specifically, the Study will: (a) provide an overall assess- sustainability of GEF projects? To what extent do country ment of the impacts and other results achieved since the GEF ownership, stakeholder involvement in project develop- restructuring in 1994 till June 2004; (b) assess the effective- ment and execution and the generation of local benefits ness of GEF policies, strategies and programs; (c) examine the improve the sustainability of activities supported through main factors influencing the achievement of GEF objectives the GEF? and consider possible shortcomings to the achievement of objectives; (d) draw key lessons and provide clear and action- c) Effects of GEF Operations on other institutions and related is- able recommendations to the GEF Council on how to render sues: How successful has the GEF been in fulfilling its cat- GEF support more effective in contributing to global environ- alytic role by leveraging additional resources, catalyzing re- mental benefits. sults by innovation, demonstration and replication, fostering international co-operation on environmental is- 8. The Study will also focus on GEF`s performance in imple- sues, mainstreaming environmental issues into partner in- menting the GEF Operational Strategy, notably its ten opera- stitutions and involving the private sector in both projects tional principles listed in the GEF Operational Strategy, as well and co-financing? What are the key areas that lead to cat- as their continued relevance to the development and imple- alytic impacts and what issues need to be addressed to im- mentation of GEF`s programs. The study will also review and prove catalytic impacts? evaluate the effectiveness of instruments of assistance e.g., d) Effects of GEF's Institutional Structure and Procedures on Re- small grants program etc. sults: Are the GEF entities--the Implementing and Execut- ing Agencies, the GEF Secretariat, the STAP and the Trus- tee--performing their respective functions in a Scope and Focus of OPS3 satisfactory, cost-effective and responsive manner? Are there conclusions that can be drawn with respect to cost- 9. The scope of OPS3 will cover five main themes: effectiveness and responsiveness of the GEF projects in comparison to similar international institutions? Are GEF`s (1) Results of the GEF activities. The OPS 3 team will: policies and programs adequately responding to the ob- jectives of the Conventions to which it serves as a financial a) Assess both the quantitative and qualitative environmen- mechanism? Is the GEF`s composition, structure and divi- tal impacts and other results at local, regional and global sion of roles and responsibilities effective in meeting its levels as well as other results from completed and on-go- mandate and operations? Are the GEF Secretariat and its ing GEF efforts in biodiversity, climate change and interna- tional waters, including progress made towards achieving 74 OPS3: ProgressingToward Environmental Results GEF3 targets3 by projects approved since July 2002. To the (4) GEF policies, institutional structure and partnerships. extent the impacts are not quantifiable, the study will ex- The OPS3 team will: plain why e.g., inadequate project design, inadequate baseline data. a) Evaluate whether GEF policies and programs are ade- quately responding to the objectives and guidance of the b) Regarding the two new GEF focal areas--land degradation Conventions to which GEF is a financial mechanism. and persistent organic pollutants (POPs)--review initial steps and assess to what extent GEF projects reflect global b) Assess the performance and cost-effectiveness of GEF Im- priorities for action in these two areas. plementing and Executing Agencies in project design, ap- proval and implementation, as well as the delivery of GEF c) With reference to the GEF M&E Unit's ozone study in the corporate services by the Implementing Agencies. year 2000,4 provide an update on performance and impact with regard to the phase-out of ozone depleting sub- c) Assess whether GEF's current composition, structure, divi- stances. sion of roles and responsibilities are effective for meeting its mandate and operations. d) Identify key factors that influence the achievement of re- sults, such as policy frameworks, strategies, institutional ca- d) Assess how the cooperation among the GEF entities--the pacities, and implementation issues. GEF Secretariat, Implementing and Executing Agencies, STAP and the Trustee--impact on the development of the GEF portfolio and its results. Also consider the effective- (2) Sustainability of results at country level. ness of communication and information dissemination be- The OPS3 team will: tween or among entities and stakeholders, including the Conventions to which it is a financial mechanism. a) Assess the extent to which global environmental benefits from GEF projects have or are likely to continue after proj- e) Assess the follow-up of Council decisions by the GEF Secre- ect completion. The analysis should be done for each focal tariat as well as the Implementing and Executing Agencies. area. f) Assess the extent to which the GEF Secretariat and its Im- b) Identify the key factors that determine sustainability, plementing and Executing Agencies are responsive to na- whether they are of a financial, economic, social, ecologi- tional priorities and GEF strategies and the extent to which cal, institutional or other nature, and review their effects on they are effective in mobilizing co-financing and main- GEF performance. streaming GEF objectives and operations within their own development assistance programs. (3) GEF as a catalytic institution. The OPS3 team will: g) Review the effectiveness of the GEF strategies, outreach activities and dialogue with different stakeholders to build a) Assess the performance of the GEF in fulfilling its catalytic partnerships and catalytic action for global environmental role in each focal area by: (a) leveraging additional financial benefits. Focus should be given to evaluating the partici- resources from the public and private sectors; (b) catalyz- pation of the private sector, civil society, and local commu- ing results by innovation, demonstration and replication; nities as well as recommending ways to improve upon this (c) fostering international cooperation on environmental participation. Gender issues should be addressed in this issues; and (d) leading to the mainstreaming of global en- context. vironmental issues into the operations of Implementing h) Assess how the cooperation amongst GEF entities at the Agencies, Executing Agencies and the policies and pro- country level impact on the development and implemen- grams of national governments. tation of country portfolios. b) Identify key areas and/or issues related to potential catalyt- i) Assess the effectiveness of the M&E Unit both as an institu- ic impacts and analyze how GEF has acted or positioned it- tion and in terms of its processes. self in response to such opportunities. CompleteTerms of Reference for theThird Overall Performance Study of the GEF 75 (5) GEF implementation processes. The OPS3 team will: 12. The team will function in an inclusive and open manner. To help ensure transparency and good communication, the a) Review the factors and strategies that influence perform- Team will propose appropriate mechanisms for stakeholder ance and results at the three main phases of the project cy- involvement, consultations and cooperation with the key cle: (a) project design and approval; (b) implementation; partners. Such mechanisms may include consultations in con- and (c) monitoring and evaluation. Wherever relevant and junction with Council Meetings and appropriate international possible, the study should distinguish among various or regional environmental meetings; regional or local events, types of recipient countries e.g., emerging market and public information through electronic or other means. economies, small island developing states (SIDS) and LDCs. 13. In preparation for OPS3, the GEF M&E Unit is implement- b) Assess whether lessons learned and feedback from Council ing program studies in the three main GEF focal areas of bio- members have been adequately integrated into project diversity, climate change, and international waters. In addi- design and implementation. The review should seek to tion, the Unit has also conducted annual project performance recommend ways to simplify, standardize, and render the reviews, several special studies, including the extensive analy- process more effective. Particular attention should be paid sis of local benefits of GEF projects and of financial arrange- to evaluating the capacity of countries to develop and im- ments of biodiversity projects. The program studies and the plement GEF projects. special studies will be conducted by teams of independent c) Assess the progress made on the implementation of the consultants and staff members from the GEF M&E Unit. The Policy Recommendations Agreed as Part of the Third Re- OPS3 team will review the existing body of work of the M&E plenishment of the GEF Trust Fund. Unit and the lessons it has generated, consider the usefulness of this material to establish the status of the current portfolio and determine the need for additional activities including Execution of the Study and Stakeholder data collection and analysis, desk reviews and field work. The Involvement lessons learned from analysis of existing documents should also guide the extent to which additional activities need to be 10. The Third Study of the GEF's Overall Performance (OPS3) undertaken in order to fill gaps in current knowledge. will be undertaken from May-June 2004 through April 2005 by an independent and external team of experts.5 OPS3 Team 11. The work of the GEF involves a range of stakeholders. Stakeholder participation is one of the key GEF review criteria 14. The composition of the Study Team, with 6­9 members,6 of performance. Accordingly, the OPS3 Team will consult with should reflect the independence and the substantive focus of the GEF Secretariat, the GEF M&E Unit, Implementing and Exe- the exercise. The Study will be undertaken by a firm or consor- cuting Agencies, GEF member countries, the Trustee, STAP, tium of firms which is internationally recognized in evaluation the Secretariats of the Conventions for which GEF serves as a assessment work, has a strong reputation and experience in financial mechanism, direct beneficiaries of GEF projects as international development, has a substantial track record in well as some of the agencies co-financing GEF projects and undertaking complex assignments, and can demonstrate it programs. Other key partners, such as the private sector and has no conflict of interest (para. 16, below). The firm or con- the NGO community, will also be consulted. sortium will be expected to have a qualified senior member/partner that would oversee the work of the study team, liaise regularly with the GEF M&E Director, and ensure the timely delivery of the contracted products by the Study Team. The composition of the Study Team, with 6 principal members, should reflect the independence and the substan- tive focus of the exercise. 76 OPS3: ProgressingToward Environmental Results In accordance with the scope of the Study, the Team com- · Demonstrated ability to deliver analytical papers of high position should reflect: quality, · Strong competence in evaluation, especially in techniques · Be available to work for at least 80 days between June appropriate for organizational performance evaluations, 2004 and May 2005, in periods essential to the Study, program or thematic evaluations; · Excellent analytical writing and communication skills in · Expertise in global environment and sustainable develop- English; proficiency in Spanish or French is desirable. ment; 16. The OPS3 evaluation exercise is intended to be an inde- · Competence in related aspects of policy analysis, econom- pendent assessment, offering professional, objective and im- ic and social development; partial advice. The evaluation team must therefore be free of · Financial expertise; any conflict of interest, real or apparent, with other, past, pres- ent or considered future assignments or own corporate inter- · Institutional and organizational expertise; and est. The evaluation team must demonstrate a clear arms- · Knowledge of project management approaches of interna- length relationship with GEF-related entities and member tional development assistance, in order to contextualize governments. No team member or firm shall be engaged in the analysis and assess project contributions. the evaluation exercise if this would be in conflict with prior or current obligations to other clients, or if they would be in a The Team Leader must be a senior professional with sub- position of not being able to carry out the assignment inde- stantial and relevant international experience in evaluation pendently. assessment, must have a demonstrated capacity and experi- Without limitation on the generality of this rule, the con- ence in strategic thinking and policy advice, in evaluation of tracting party and members of the team shall not: complex programs and in team leadership. · have any concurrent assignment (as contracting party) 15. All team members and consultants will be chosen on the with the GEF Secretariat or with the STAP, the GEF-related basis of merit. In composing the team and conducting the departments of the Implementing Agencies or Executing study, consideration will also be given to geographical and Agencies (hereafter collectively referred to as "GEF Institu- gender balance as well as the involvement of local and re- tions")8, or be current employee of any member govern- gional consultants to enhance the quality of the study. The ment or member government-owned entity. Publicly- team members will be involved for the entire duration of the owned academic institutions, including universities or their study. The minimum qualifications and expertise required equivalents and research institutions affiliated with univer- from each team member are: sities, can be considered exceptions to this, provided they have a record of independence. · Masters degree or equivalent in natural or social sciences or related fields, · have previous work experience with the GEF Secretariat, or previous oversight/managerial responsibility for GEF-relat- · Minimum of 10 years of experience in evaluation, research, ed activities, including any external review/audit or evalua- environment and development; at least 15 years of experi- tion, or have a work history with more than 15% of work- ence for the Team Leader and key members of the team, ing days with GEF-related activities over the last five years9. · Minimum of five years of experience in developing coun- · derive, on average over the last five years, directly, or tries or countries with economies in transition, through their employer, more than 35% of their funding, · Experience with multilateral, bilateral or national develop- contracts or income from any single source with environ- ment programs, ment-related activities or the international affairs/develop- ment assistance agencies of any one member govern- · Capacity to work effectively and flexibly in a team and with ment, or more than 20% from any single GEF Institution. tight timeframes. Ability and readiness to work in a multi- cultural environment and to travel internationally, CompleteTerms of Reference for theThird Overall Performance Study of the GEF 77 In addition, the successful applicant will be barred from 21. The methodology should be consistent with best prac- bidding for contracts related to GEF activities for a period of tices as outlined in the independent M&E Unit's Terms of Ref- one year after the completion of this evaluation, except for ex- erence (approved by Council July 28, 2003), including random tensions of the evaluation as may be requested by the GEF selection methodologies, and should respond to the five key Council. areas of scope, with an appropriate balance of the following key elements: 17. The Operations Evaluation Department (OED), a fully in- · Analysis of existing data and information, gathered from dependent unit of the World Bank, will manage the selection desk reviews and field visits; process (including advertising and reviewing Expressions of Interest (EOIs), reviewing EOIs, and undertaking a competitive · Consultations should be undertaken using appropriate and transparent selection process) and will select the team to participatory techniques with international, national and undertake the study consistent with the Terms of Reference. local stakeholders and beneficiaries, directly though re- The aim is to assemble the full team by June 2004. gional/national meetings, interaction at international events, surveys or questionnaires, focus group meetings 18. Once the work is underway, the incoming head of the etc. newly independent GEF M&E Unit will provide oversight of · Interviews with key stakeholders, in particular project ben- the process, basically in ensuring that the Terms of Reference eficiaries, community groups, NGOs and civil society or- for OPS3 are followed. The GEF M&E Unit will brief the OPS3 ganizations and the private sector as well as the GEF Secre- Team on relevant GEF M&E documents and data sources. It tariat, GEF Implementing and Executing Agencies, will provide technical backstopping and administrative sup- members of the GEF Council and representatives of the port to the OPS3 Team including preparing contracts and lo- Secretariats of UN Conventions to which the GEF serves as gistics for field visits and regional consultations. The Unit will a financial mechanism. also provide written comments to the OPS3 team on all its de- liverables. · Selected field visits, tentatively in 10­15 countries, identi- fied on the basis of criteria specified in the Inception Re- 19. The GEF Secretariat and GEF Implementing and Execut- port and best practices with respect to random selection. ing Agencies will also be expected to respond promptly and Those criteria might include: (a) number of GEF projects fully to OPS3 requests for information and documentation, as and size of funds allocated; (b) representation of projects well as render some logistical support to the OPS3 team for in the various focal areas; (c) project performance; (d) vari- field visits. ous institutional models for responding to GEF initiatives; (e) geographical and other variations between regions and countries; (f) information gaps. Methodology · Any other approaches that may yield essential information (select project reviews, case studies, country studies, time- 20. The complexity of the issues, the organizational frame- series analysis etc.). work and impact levels covered in OPS3 call for a coherent and focused evaluation methodology. The key challenge is to establish a Study framework that captures the hierarchy of re- Deliverables sults produced at the levels of projects, programs, and focal areas, while analyzing overall performance at the organiza- 22. Within three months of its establishment, the OPS3 team tional and institutional level and across common areas and is- will produce an Inception Report to be shared with the GEF sues. Such a framework will be detailed in the Inception Re- Council,10 the GEF Secretariat, and GEF M&E Unit for feedback. port of the OPS3 team, with a plan of implementation for the The Inception Report will provide a detailed, transparent and Study. practicable framework for the Study, and should include, as a minimum: 78 OPS3: ProgressingToward Environmental Results · Purpose and scope of the Study as defined in the Terms of Notes Reference. 1. The Executing Agencies comprise the four major regional devel- · Other areas/issues to be covered in-depth (focal areas, opment banks (the African Development Bank, the Asian Devel- strategic priorities, cross-cutting issues, global, regional opment Bank, the European Bank for Reconstruction and Devel- and country projects), criteria for selection. opment, and the Inter-American Development Bank) and three specialized agencies (the International Fund for Agricultural · Methodology to address the five key areas of OPS3. Development (IFAD), the Food and Agriculture Organization of the United Nations (FAO), and the United Nations Industrial · Data and information needed, availability, and how to Development Organization (UNIDO). address data gaps. Data and analysis should be gender- 2 Results are defined as the outputs, outcomes and impacts disaggregated when relevant and possible. achieved by the implementation of projects and programs. · Strategy and mechanisms for stakeholder involvement, These should include the assessment of both positive and negative outputs, outcomes and impacts that are both intended communication, consultations and cooperation with the and unintended. key partners. 3 GEF3 targets were presented in GEF/C.21/Inf.11. · Organization of the Study and work program, including 4 Study of Impact of GEF Activities on Phase-Out of Ozone Deplet- country field visits and specific responsibilities and deliver- ing Substances (GEF Evaluation Report #1-00). ables of the Study team members. 5 This is in accordance with GEF M&E Unit's TORs approved by Council: OPSs are external and managed by the Unit. · Proposed interaction and interim reporting to the GEF 6 The estimated inputs in the budget are full-time for the team Council. leader and 5 person-months for the five additional study team consultants between June 2004 and May 2005. 23. The main expected output is the comprehensive draft re- 7 It is not feasible to compose the team of technical expertise in port of the Study that should be circulated to the Council no the numerous focal area or OPs of the GEF. Where required, later than the first quarter of 2005, and final report of the short-term technical expertise may be called upon. Study that should be circulated to the Council, including rele- 8 The term "GEF-related departments of the Implementing Agen- vant annexes with detailed data, by the second quarter of cies or Executing Agencies" is intended to include units of those 2005. The final report by the OPS3 Team should contain: agencies which undertake GEF project activities, GEF activities or environmental activities more generally. For example, a contract- · Executive Summary including specific and actionable con- ing party could have a concurrent contract on AIDS projects with clusions and recommendations the World Bank, but could not have a concurrent contract for en- vironmental projects with the World Bank. · Background, with analysis of context 9 The term "GEF-related activities" is understood to be limited to activities funded by GEF or activities funded by the Implement- · Analysis of results and performance in the five areas of ing Agencies or Executing Agencies that are undertaken in con- scope junction with GEF-funded activities. · Key policy issues 10 The Inception Report will be circulated to the Council to provide feedback on an informal basis. · Lessons Learned · Main Findings and Recommendations · Annexes, as appropriate 24. The OPS3 team will also be responsible for providing progress reports to the GEF M&E Unit as proposed in its Incep- tion Report, or as requested by the Council during the execu- tion of OPS3. Upon completion of the study, the OPS3 team will make available, on request, any supporting documenta- tion or studies as background material to the main report. CompleteTerms of Reference for theThird Overall Performance Study of the GEF 79 OPS3 Acronyms and Abbreviations ABS Access and Benefit Sharing BPS Biodiversity Program Study CAS Country Assistance Strategy (World Bank) CBD Convention on Biological Diversity CCPS Climate Change Program Study CDM Clean Development Mechanism CDW Country Dialogue Workshop CEIT Countries with economies in transition CFC Chlorofluorocarbon CO2 Carbon dioxide COP Conference of the Parties CPAP Country Programme Action Plan (UNDP) EA Executing Agency EU European Union FSP Full-size project GEF Global Environment Facility GEF-1 Restructured GEF (FY1995­98) GEF-2 FY1999­2002 GEF-3 FY2004­2006 GEF-4 FY2007­2010 GEFM&E Global Environment Facility Monitoring & Evaluation Unit GEFSEC Global Environment Facility Secretariat GHG Greenhouse gas HCFC Hydrochlorofluorocarbon HFC Hydrofluorocarbon IA Implementing Agency IPCC Intergovernmental Panel on Climate Change IUCN International Union for the Conservation of Nature and Natural Resources (World Conservation Union) IW:LEARN International Waters Learning Exchange and Resource Network IWPS International Waters Program Study IWTF GEF International Waters Task Force KM Knowledge management KRA IUCN Key Results Area LDC Least developed country LME Large Marine Ecosystem M&E Monitoring and evaluation MIS Management and information system(s) MLF Multilateral Fund of the Montreal Protocol MOP Meeting of the Parties to the Montreal Protocol MOU Memorandum of Understanding MSP Medium-size project 80 OPS3: ProgressingToward Environmental Results MT Metric ton(s) NAI Non­Annex I (countries) NAPA National Adaptation Programme of Action NC National communication NCSA National Capacity Self-Assessment(s) NDI National Dialogue Initiative NGO Nongovernmental organization NIPs National Implementation Plans (under POPs) ODP Ozone-depleting potential ODS Ozone-depleting substance(s) OME GEF Office of Monitoring and Evaluation OP Operational Programs OP1 Arid and Semi-Arid Zone Ecosystems OP2 Coastal, Marine, and Freshwater Ecosystems OP3 Forest Ecosystems OP4 Mountain Ecosystems OP5 Removal of Barriers to Energy Efficiency and Energy Conservation OP6 Promoting the Adoption of Renewable Energy by Removing Barriers and Reducing Im- plementation Costs OP7 Reducing the Long-Term Costs of Low Greenhouse Gas Emitting Energy Technologies OP8 Water Body­Based Operational Program OP9 Integrated Land and Water Multiple Focal Area Operational Program OP10 Contaminant-Based Operational Program OP11 Promoting Environmentally Sustainable Transport OP12 Integrated Ecosystem Management OP13 Conservation and Sustainable Use of Biological Diversity Important to Agriculture OP14 Persistent Organic Pollutants OP15 Sustainable Land Management OPS Overall Performance Study(-ies) PIR Project Implementation Review PMIS GEF Project Tracking and Management Information System POPs Persistent Organic Pollutants PPR Project Performance Report PV Photovoltaics RAF Resource Allocation Framework SAP Strategic Action Programme SCCF Special Climate Change Fund SGP Small Grants Programme SIDS Small island developing states STAP Scientific and Technical Advisory Panel TDA Transboundary Diagnostic Analysis TEAP Technology and Economic Assessment Panel TER Terminal Evaluation Review TOR Terms of Reference UN United Nations UNAIDS The Joint United Nations Programme on HIV/AIDS UNCCD United Nations Convention to Combat Desertification UNDP United Nations Development Programme UNEP United Nations Environment Programme UNFCCC United Nations Framework Convention on Climate Change WCMC World Conservation Monitoring Centre OPS3 Acronyms and Abbreviations 81 Photo Credits Front cover background and inside front cover: © Grant V. Faint/Getty Images. Front cover (clockwise from upper left): © Plush Studios/Getty Images, © S. Meltzer/PhotoLink/ Getty Images, © Don Farrall/Getty Images, © H. Castro, © Curt Carnemark/World Bank. Interior: page xii, © Glen Allison/Getty Images; pages 2, 8, 11, 13, 18, 20, 28, 40, 44, 50, 55, 60, 63, © Curt Carnemark/World Bank; pages 5 and 48, Daniel Lieberman/ICF Consulting; page 18, © Thomas Sennett/World Bank; page 24, © Paul Chesley/National Geographic/Getty Images; page 33, © Alex Baluyut/World Bank; page 52, © Tim Cullen/World Bank; page 58, © Alex Baluyut/ World Bank; page 64, © Tran Thi Hoa/World Bank. 83 © 2005 Office of Monitoring and Evaluation of the Global Environment Facility 1818 H Street, NW Washington, DC 20433 Internet: www.thegef.org E-mail: gefteam@thegef.org Printed using soy inks on recycled paper.