57521 GEF Annual Performance Report 2007 october 2008 Evaluation Office G LOBAL ENVIRONMENT FACILITY Global Environment Facility Evaluation Office GEF Annual Performance Report 2007 October 2008 (The main findings and recommendations of this evaluation were presented to the GEF Council in April 2008.) Evaluation Report No. 40 © 2008 Global Environment Facility Evaluation Office 1818 H Street, NW Washington, DC 20433 Internet: www.gefeo.org Email: gefevaluation@thegef.org All rights reserved. The findings, interpretations, and conclusions expressed herein are those of the authors and do not necessarily reflect the views of the GEF Council or the governments they represent. The GEF Evaluation Office does not guarantee the accuracy of the data included in this work. The boundaries, colors, denomi- nations, and other information shown on any map in this work do not imply any judgment on the part of the GEF concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The GEF encourages dissemination of its work and will normally grant permission promptly. ISBN-10: 1-933992-16-6 ISBN-13: 978-1-933992-16-7 Credits Director of the GEF Evaluation Office: Robert D. van den Berg Task Manager: Aaron Zazueta, Senior Evaluation Officer, GEF Evaluation Office Evaluation Team: Neeraj Kumar Negi, Evaluation Officer; David Michael Todd, Senior Evaluation Officer; Anna Viggh, Evalua- tion Officer; Ines Angulo and Cynthia Cummis, consultants Editing and design: Nita Congress Printing: Graphic Communications Cover photo: Urucan seeds from Brazil, Igor Castro da Silva Braga, World Bank Evaluation Report No. 40 A FREE PUBLICATION Contents Foreword ...................................................................................................................................... vii Acknowledgments ....................................................................................................................... ix Abbreviations ................................................................................................................................ x 1. Background, Main Conclusions, and Recommendations .................................................... 1 1.1 Background ...................................................................................................................................................... 1 1.2 Findings and Conclusions............................................................................................................................. 2 1.3 Recommendation ......................................................................................................................................... 10 1.4 Issues for the Future ..................................................................................................................................... 10 Notes ........................................................................................................................................................................ 11 2. Scope and Methodology ....................................................................................................... 12 2.1 Scope................................................................................................................................................................ 12 2.2 Methodology.................................................................................................................................................. 13 3. Project Outcomes and Sustainability .................................................................................. 17 3.1 Rating Approach ........................................................................................................................................... 17 3.2 Project Outcomes ......................................................................................................................................... 18 3.3 Project Outcomes Sustainability............................................................................................................... 21 3.4 Outcomes and Outcomes Sustainability................................................................................................. 23 4. Factors Affecting Attainment of Project Results ................................................................ 24 4.1 Materialization of Cofinancing ................................................................................................................. 24 4.2 Delays in Project Completion .................................................................................................................... 26 Note .......................................................................................................................................................................... 29 5. Evaluation of Capacity Development ................................................................................. 30 5.1 Findings from the Country Case Studies ................................................................................................ 30 5.2 Findings from the Terminal Evaluations ................................................................................................. 31 5.3 Summary ........................................................................................................................................................ 33 v 6. Quality of Project Monitoring............................................................................................... 34 6.1 Rating Approach ........................................................................................................................................... 34 6.2 Overall Findings ............................................................................................................................................ 34 6.3 Factors Affecting Quality of Monitoring ................................................................................................ 35 7. Quality of Terminal Evaluations............................................................................................ 37 7.1 Rating Approach ........................................................................................................................................... 38 7.2 Findings ........................................................................................................................................................... 38 7.3 Agency versus Evaluation Office Ratings................................................................................................ 41 8. Assessment of GEF Agency Climate Policies and Guidelines ............................................ 42 8.1 Assessment Approach ................................................................................................................................. 42 8.2 Findings ........................................................................................................................................................... 43 9. Management Action Record ................................................................................................. 44 9.1 Rating Approach ........................................................................................................................................... 44 9.2 Findings ........................................................................................................................................................... 45 10. Performance Matrix ............................................................................................................. 48 10.1 Rating Approach ........................................................................................................................................... 48 10.2 Findings ........................................................................................................................................................... 51 Annexes A. Terminal Evaluation ReportReview Guidelines ...................................................................................... 53 B. Terminal Evaluations Reviewed during FY 2007 ..................................................................................... 59 C. Greenhouse Gas Footprint Policies and Guidelines Assessment ........................................................ 61 D. Methodological Notes on the Performance Matrix ............................................................................... 71 E. Management Response .................................................................................................................................. 74 References ................................................................................................................................... 77 Figures 3.1 Percentage of Projects with Outcomes Rated Moderately Satisfactory or Above ........................ 17 3.2 Percentage of Projects with Sustainability Rated Moderately Likely or Above ............................. 17 3.3 Number of Terminal Evaluation Reports by GEF Agency and Focal Area..................................... 18 3.4 Project Outcome Ratings by Number of Projects and Project Funding ......................................... 19 3.5 Percentage of Projects with Outcome Dimensions Rated Moderately Satisfactory or Above .. 20 3.6 Sustainability Ratings by Number of Projects and Project Funding ................................................ 22 3.7 Perceived Risks Underlying Projects Receiving Low Sustainability Ratings .................................. 22 3.8 Projects with Outcomes/Sustainability Rated as Moderately Satisfactory/ Moderately Likely or Above ....................................................................................................................... 23 4.1 Reported Materialization of Cofinancing per Dollar of Approved GEF Grant ............................. 25 4.2 Average Delay in Project Completion by GEF Agency and Focal Area, FYs 2005­07 ................ 27 vi GEF Annual Performance Report 2007 5.1 Types of Capacity Development Activities in Projects........................................................................ 32 6.1 Percentage of Projects Rated Moderately Satisfactory or Above for Quality of M&E ................ 34 6.2 Quality of M&E during Implementation ................................................................................................ 35 7.1 Number of Terminal Evaluation Reports Submitted for Review ...................................................... 37 7.2 Percentage of Terminal Evaluation Reports with Quality Rated Moderately Satisfactory or Above .............................................................................................................................................................. 39 7.3 Percentage of Terminal Evaluation Reports with Quality Rated Moderately Satisfactory or Above, by GEF Agency and Focal Area................................................................................................... 40 7.4 Percentage of Terminal Evaluation Reports with Quality Rated Moderately Satisfactory or Above on Each Quality Dimension .......................................................................................................... 40 9.1 Change in Ratings on Adoption of Council Decisions since APR 2006 Assessment................... 46 Tables 1.1 Summary of Project Outcomes and Sustainability Ratings.................................................................. 3 1.2 Materialization of Cofinancing ................................................................................................................... 5 1.3 M&E Ratings for Projects ............................................................................................................................. 6 1.4 Quality of Terminal Evaluation Reports ................................................................................................... 7 1.5 Ratings of GEF Progress in Adopting Council Decisions ..................................................................... 9 3.1 Project Outcomes by GEF Agency and Focal Area .............................................................................. 19 3.2 Outcome Ratings for FY 2007 by GEF Investment .............................................................................. 20 3.3 Sustainability of Project Outcomes by GEF Agency and Focal Area ............................................... 21 3.4 Sustainability of Outcome Ratings for FY 2007 by GEF Investment ............................................... 22 4.1 Materialization of Cofinancing ................................................................................................................. 24 4.2 Materialization of Cofinancing by GEF Agency, 2002­07 ................................................................. 26 6.1 Quality of M&E during Project Implementation by GEF Agency and Focal Area ....................... 35 7.1 Quality of Terminal Evaluation Reports by GEF Agency and Focal Area ...................................... 39 8.1 Assessment of GEF Agencies' Climate Policies..................................................................................... 43 9.1 Ratings of GEF Progress in Adopting Council Decisions ................................................................... 45 10.1 GEF Agencies and Entities Performance Matrix .................................................................................. 49 Contents vii Foreword The Evaluation Office of the Global Environ- Office. The 2007 APR also responds to a specific ment Facility is pleased to present its fourth concern expressed in previous reports: the verac- annual performance report (APR) evaluating the ity of the terminal evaluation reports submitted performance of the GEF portfolio. Building and so far had been checked through desk reviews responding to work initiated in the three preced- only. To address this issue, the Evaluation Office ing APRs, the evaluation based its conclusions piloted the approach to direct verification in the on information collected by the monitoring and field of terminal evaluations described in this evaluation systems of the GEF Agencies. report. The report describes results of an indepen- Findings presented in this APR show that the dent assessment of three topics highly relevant quality of terminal evaluation reports--on to the success of GEF efforts: project outcomes which the information and conclusions of this and sustainability, materialization of cofinanc- report so rely--has steadily improved. This can ing, and quality of monitoring and evaluation be attributed to efforts by the GEF Agencies and arrangements. First and foremost, the Evalua- feedback provided to them by the Evaluation tion Office assessed the extent to which capac- Office. However, the 2007 APR indicates that ity development activities in GEF projects are this fiscal year's cohort has lower outcome and relevant, effective, and efficient. The report also sustainability ratings than for previous years. It describes the results of these activities and their is too early to determine if this will constitute sustainability. a trend, given the small sample size and differ- ences in the typology of projects included in the This year's report goes further than its prede- 2007 cohort. cessors. With increasing attention worldwide to reducing greenhouse gas emissions, it is impor- The GEF Council reviewed the 2007 APR and tant that the GEF partnership address this issue its management response (see annex E) at its in its own operations. This APR thus reviews the 33rd meeting in April 2008. The Council in par- carbon footprint policies and guidelines of GEF ticular congratulated the GEF Agencies on their entities and Agencies. In addition, for the first progress in reducing greenhouse gas emissions time, a performance matrix summarizes the per- in internal operations and invited them to pro- formance of the GEF Agencies and the Secretariat vide information on progress in June 2009. Some on various parameters tracked by the Evaluation Council members requested that the Evaluation ix Office address cofinancing in detail in future like to thank all those involved for their support evaluations. and criticism. In preparing its final report, the Evaluation Office discussed various intermediate drafts with the GEF Secretariat and Agencies, which led to Rob van den Berg improvement in the quality of this APR. I would Director, Evaluation Office x GEF Annual Performance Report 2007 Acknowledgments This year's annual performance report reflects emission policieis and guidelines of GEF Agencies the able efforts of a number of Global Environ- and drafted the related chapter 8. ment Facility (GEF) Evaluation Office staff and Appreciation is also due to colleagues in the Eval- consultants. Serving as task manager for general uation Office who reviewed the terminal evalu- development of the report was Aaron Zazueta, ation reports under the coordination of Neeraj Senior Evaluation Officer in the GEF Evaluation Kumar Negi. They include Evaluation Officers Office. Neeraj Kumar Negi, Evaluation Officer, Sandra Ramboli, Lee Risby, Neeraj Kumar Negi, drafted the overall document. David Michael and Anna Viggh; Evaluation Specialist Divya Nair; Todd, Senior Evaluation Officer, and Anna Viggh, and consultants Ines Angulo, Alejandro Imbach, Evaluation Officer, conducted the capacity-build- Maria Soledad Mackinnon, and Timothy Ranja. ing evaluation and drafted chapter 5. Ines Angulo, consultant, helped conduct the desk reviews for The Evaluation Office also wishes to thank the the capacity development evaluation and drafted staff of the GEF Agencies for their valuable com- chapter 9, which presents findings on manage- ments and contributions to this evaluation. Nev- ment action records. Cynthia Cummis, consultant, ertheless, final responsibility for the report rests conducted the pilot assessment of greenhouse gas firmly with the Office. xi Abbreviations APR annual performance report MAR management action record FSP full-size project MSP medium-size project FY fiscal year OPS4 Fourth Overall Performance Study GEF Global Environment Facility PIF project identification form GHG greenhouse gas SGP Small Grants Programme IA Implementing Agency UN United Nations IEG Independent Evaluation Group UNDP United Nations Development Programme M&E monitoring and evaluation UNEP United Nations Environment Programme xii 1. Background, Main Conclusions, and Recommendations 1.1 Background of cofinancing, and quality of monitoring. For the assessment of project outcomes, project This document is the fourth annual performance sustainability, and delays in project comple- report (APR) that the Global Environment Facil- tion, 148 projects for which terminal evaluation ity (GEF) Evaluation Office has developed. The reports were submitted by the GEF Agencies to report presents a detailed account of some aspects the Evaluation Office in FY 2005 (41 projects), of project results, of processes that may affect FY 2006 (66 projects), and FY 2007 (41 projects) these results, and of monitoring and evaluation were considered. The GEF investment in these (M&E) arrangements in completed GEF projects. 148 projects totaled $713 million. The focus In the APR for fiscal year (FY) 2005,1 the Evalua- of this APR's reporting is on the 41 projects for tion Office presented an assessment of the M&E which terminal evaluation reports were submit- arrangements across the GEF partnership. The ted in FY 2007; these represent a total GEF invest- APR for FY 2006 contained an assessment of proj- ment of $199 million. ect supervision arrangements across the GEF part- nership. In this APR, the Evaluation Office pres- For assessment of materialization of cofinanc- ents an assessment of the extent to which capacity ing, all 223 projects for which terminal evalua- development activities in GEF projects are rel- tion reports have been submitted since January evant, effective, and efficient and the results, and 2001 were considered. Of these, information on the sustainability of the results, of these activities. actual materialization of cofinancing was pro- It also presents a review of Agency carbon foot- vided for 157 projects (70 percent), either in the print policies and guidelines. For the first time, a terminal evaluation reports and/or by the rel- performance matrix is presented, which summa- evant Implementing Agency (IA). The GEF has rizes the performance of the GEF Agencies and the invested a total of $782 million in these 157 proj- GEF Secretariat on various parameters tracked by ects; the IAs reported that an additional $2.5 bil- the Office. This matrix will be a regular feature of lion was leveraged for these projects in the form future APRs. This year, the Evaluation Office also of cofinancing. piloted an approach to direct verification of termi- To assess capacity development activities in nal evaluations. GEF projects, the Evaluation Office conducted The 2007 APR continues the annual presentation detailed country case studies in Vietnam and of assessment of project outcomes, project sustain- the Philippines. In addition, 42 terminal evalua- ability, project completion delays, materialization tions, including 41 submitted to the Office during 1 FY 2007, were reviewed. The Office was thus able Office to make the performance ratings more con- to assess the extent to which findings based on the sistent with the evidence provided in the terminal analysis of the data from these two approaches evaluation reports, it does not allow it to estab- were consistent. lish the veracity of the evidence presented. Direct verification permits the Office to address this The GEF Evaluation Office commissioned a survey concern. In addition, the field verification process of the policieis and guidelines of GEF Implement- provides the Office with an opportunity to gather ing and Executing Agencies to broadly assess the new information to draw lessons from the com- extent to which they are addressing the green- pleted projects. To develop its approach to direct house gas (GHG) emissions generated by their verification further before adopting it as a regular internal operations. GHG footprint­related pol- activity of the APR, the Evaluation Office this year icieis and guidelines of the 10 GEF Agencies were piloted the approach for three full-size projects reviewed. (FSPs) for which terminal evaluation reports had This year's management action record (MAR) been submitted in FY 2007. tracks the level of adoption of 46 Council deci- One of the limitations noted in the earlier APRs sions based on 12 GEF Evaluation Office docu- has been that, on many issues, the number of ments. The Office was able to verify 41 of these projects for which data were available was not suf- decisions and plans to carry out thematic assess- ficient to allow for in-depth assessment of differ- ments in the future to assess adoption of Council ences in performance and the underlying factors decisions that were not verified this year. that affect performance. With the inclusion of the This APR marks the first time a performance FY 2007 cohort in the data pool, this constraint matrix providing a summary of the performance has been mitigated to a large extent for such issues of GEF Agencies and the GEF Secretariat on rel- as trends pertaining to the quality of terminal evant parameters has been included. Several of evaluation reports and reported materialization of the parameters included in the matrix are already cofinancing. Analysis of other issues remains con- assessed on an annual basis by the Evaluation strained, but additional years of data will similarly Office. For the remainder, the Office--in col- answer to these. laboration with the evaluation units of the GEF Agencies--is developing assessment approaches. 1.2 Findings and Conclusions Reporting on such parameters will be provided in future APRs. Results The APRs for 2005 and 2006 found that the Conclusion 1: The percentage of completed absence of direct verification of completed proj- projects with outcome ratings in the satisfactory ects prevents a more comprehensive assessment range is close to the 75 percent target specified in the GEF-4 replenishment agreement. of the veracity of the submitted terminal evalu- ation reports. The APR evaluation primarily Among the completed GEF projects assessed and involves review of the evidence presented in the rated this year, 73 percent were rated moderately terminal evaluation reports, with verification satisfactory or above in achievement of outcomes, of performance ratings based primarily on desk and 59 percent were rated moderately likely or reviews. Although this methodology enables the above in sustainability of outcomes. Although the 2 GEF Annual Performance Report 2007 outcome ratings for the FY 2007 cohort are lower outcomes of 60 percent of FSPs and 85 percent than those for FYs 2005 and 2006, no conclusions of MSPs were rated moderately satisfactory or can yet be drawn as to whether this will constitute above. This is unlike the FY 2006 cohort, where a trend. the outcome ratings for FSPs and MSPs were similar. Attainment of project outcomes. The Evalua- tion Office rated the achievement of project out- Project performance for the FY 2007 cohort is comes on criteria of relevance, effectiveness, and close to the target set for the fourth GEF replen- efficiency. Of the 41 terminal evaluation reports ishment of 75 percent of projects having sat- submitted in FY 2007, 40 reports (98 percent) pro- isfactory outcomes (GEF 2006).3 Although the vided sufficient information to allow assessment completed projects assessed for FY 2007 do not of the level of attainment of project outcomes (see fall under the provisions of the fourth replenish- table 1.1).2 The key findings of this assessment are ment, their performance indicates that the target as follows. outcome achievement is realistic. z Of the 40 projects whose outcomes were rated Sustainability of project outcomes. The Evalua- by the Evaluation Office, 73 percent were rated tion Office rated sustainability based on its assess- moderately satisfactory or above. Although this ment of level of risks to sustainability of outcomes figure is lower than that for FY 2005 (82 per- on four dimensions: financial, sociopolitical, cent) and FY 2006 (84 percent), given the rela- institutional and governance, and environmen- tively small number of observations and the tal. Of the terminal evaluation reports submitted differences in project mix, the difference is not in FY 2007, 39 (95 percent) provided sufficient statistically significant. information to allow assessment of sustainabil- z Of the total investment in the rated projects ity of project outcomes. The key findings of this ($198 million), 69 percent ($137 million) was assessment follow: allocated to projects that were rated moder- z Of the 39 projects rated, the sustainability ately satisfactory or above. of outcomes of 59 percent (23 projects) was z There were differences in the outcome ratings deemed moderately likely or above. Although of FSPs and medium-size projects (MSPs): the outcomes of 65 percent of projects from the Table 1.1 Summary of Project Outcomes and Sustainability Ratings Factor FY 2005 FY 2006 FY 2007 Number of terminal evaluation reports submitted 41 66 41 Number of projects with outcome rating 39 64 40 % rated moderately satisfactory or above in outcome ratings 82 84 73 Number of projects with sustainability of outcomes ratings 39 54 39 % rated moderately likely or above in sustainability of outcomes ratings 49 65 59 Number of projects rated on both outcomes and sustainability of outcomes 39 54 39 % of rated projects with moderately satisfactory/moderately likely or above for both 44 61 51 Note: the methodology used to rate project outcomes for FY 2007 was the same as that used for FY 2006. 1. Background, Main Conclusions, and Recommendations 3 FY 2006 cohort had been similarly rated, the Of the 41 terminal evaluation reports submitted difference between the two cohorts is not sta- for FY 2007, 33 reported on materialization of tistically significant. cofinancing. Of the eight projects for which ter- minal evaluation reports did not provide infor- z Of the total GEF investment in rated projects mation on cofinancing materialization, this ($182 million), 55 percent ($99 million) was information was retrieved for six by the Evalu- invested in projects that were rated moderately ation Office through follow-up with the respec- likely or above in terms of the sustainability of tive IA. For the FY 2007 cohort, an average of their outcomes. $2.60 had been promised at the point of project The Evaluation Office assessed the extent to which inception in cofinancing per dollar of approved projects that were rated moderately satisfactory GEF investment. Of this, $1.90 was reported to or above in achievement of outcomes were also have materialized. For FY 2006, in comparison, rated moderately likely or above in sustainability $2.10 had been promised and $2.40 was reported of outcomes. Of the terminal evaluation reports in to have materialized (see table 1.2). the FY 2007 cohort, 39 (95 percent) provided suf- If all terminal evaluation reports submitted to ficient information on both the parameters to per- the Evaluation Office to date are taken into con- mit assessment. The Office found that of the rated sideration, information on cofinancing is available projects from the FY 2007 cohort, 51 percent for 157 projects. For these, the Agencies prom- were rated both moderately satisfactory or above ised an average of $3.50 in cofinancing per $1.00 in outcomes and moderately likely or above in sus- of approved GEF grant. The actual cofinancing tainability. In terms of GEF investment, 44 percent reported was slightly lower: $3.20 per $1.00 of was invested in these projects. The corresponding approved GEF grant. Thus, an average of 92 per- figures for the FY 2006 and FY 2005 cohorts were cent of promised cofinancing was reported to have 61 and 56 percent, respectively. materialized. Processes On average, the projects of the FY 2007 cohort were completed after a delay of 13 months; Conclusion 2: The materialization of cofinancing 14 percent were completed after a delay of at reported by the IAs was about three-fourths of least two years. The Evaluation Office began that promised at project approval. tracking project completion delays in FY 2005. The There are great variations among projects in average project completion delay was 19 months terms of level of GEF investment, promised cofi- for the FY 2005 cohort and 13 months for the nancing, and reported materialized cofinancing. FYs 2006 and 2007 cohorts. Compared to 44 per- The figures for a cohort could easily be skewed cent of the projects in the FY 2005 cohort, only by a few projects. Consequently, the average fig- 17 percent of the projects in the FY 2006 cohort ures for cohorts may fluctuate despite the absence and 14 percent of the FY 2007 cohort had delays of an underlying trend. The cofinancing figures of two years or more. It is too early to determine reported by the IAs for the FY 2007 cohort need if this drop in average delay in project completion to be noted with this caveat. constitutes a trend. 4 GEF Annual Performance Report 2007 Table 1.2 Materialization of Cofinancing FY of report submission Up to All Factor 2002 2003 2004 2005 2006 2007 2006 years Number of projects with cofinancing data available 11 7 30 23 47 39 118 157 Data obtained from terminal evaluation reports 11 7 30 23 47 33 118 151 Data otherwise obtained 0 0 0 0 0 6 0 6 Approved GeF grant per project (million $) 6.2 5.5 5.9 6.4 3.5 4.8 5.0 5.0 Actual GeF grant per project (million $) 6.1 4.6 4.6 6.3 3.3 4.4 4.6 4.5 Promised cofinancing per project (million $) 29.5 8.4 43.2 9.5 7.2 12.8 19.0 17.4 Promised cofinancing per $1.00 of approved GeF grant 4.70 1.50 7.40 1.50 2.10 2.60 3.80 3.5 reported materialized cofinancing per project (million $) 29.2 7.1 38.9 10.0 8.2 9.2 18.3 16.0 reported materialized cofinancing per $1.00 of approved 4.70 1.30 6.60 1.60 2.40 1.90 3.60 3.2 GeF grant Materialized cofinancing per $1.00 of promised cofinancing (%) 99 85 90 106 114 72 96 92 Conclusion 3: The results of capacity develop- regularly. Second, once staff have been trained ment activities in completed GEF projects are and put their new skills into practice, they often generally positive and in some areas significant. need higher level skills which require additional However, in many instances, the gains made are learning opportunities. The case studies revealed not sustained. a common underlying weakness in the training In FY 2007, the Evaluation Office began work to programs, namely the tendency to plan and exe- assess GEF capacity development activities. To cute training as a "one-shot" solution. Training date, the evaluation team has completed literature approaches should be built on the basis of existing reviews and country case studies for the Philip- capacity in a country or region, such as universi- pines and Vietnam. The case studies showed that ties or specialist public or private sector training the results of capacity development activities are institutions. generally positive and, in some areas, significant. GEF capacity development support has been rel- Forty-two terminal evaluations, including 41 sub- evant and in line with national policy priorities; it mitted to the Evaluation Office in FY 2007, were has also reflected a significant amount of national reviewed for this assessment. The findings of the ownership. The effectiveness of capacity develop- terminal evaluation review supported those of the ment activities has varied, although even in areas country case studies. The results of capacity devel- that did not produce immediate benefits, such may opment activities have generally been positive. develop in the longer term. The cost-effectiveness Support has overall been relevant to national devel- of capacity development activities was difficult to opment goals, with the exception of some specific assess. Although there are many improvements to training exercises in five projects. The efficiency capacity at the individual, institutional, and sys- of capacity development activities has usually temic levels, there are doubts about sustainability. been satisfactory in meeting output and outcome First, staff turnover in many government institu- targets; some projects, however, have suffered tions is high, so there is a need to repeat training implementation delays. The cost-effectiveness of 1. Background, Main Conclusions, and Recommendations 5 capacity development activities was difficult to Table 1.3 assess, mainly because budget information was M&E Ratings for Projects reported for such only when it was a separate proj- FY of terminal evaluation ect component. The effectiveness of such activi- report submission ties has varied, and there are doubts concerning Factor 2004 2005 2006 2007 the sustainability of results. The sustainability of No. of terminal evaluation 42 41 66 41 capacity improvements from training remains low reports submitted because few national or regional long-term train- Did not report on M&e 11 8 20 8 ing programs were developed. The review of ter- Did report on M&e 29 32 46 33 minal evaluations found that there is inadequate reporting not required 2 1 0 0 % rated moderately satis- 55 66 78 61 reporting on the performance of capacity develop- factory or above ment activities implemented by projects. The weak evidence base of capacity development activities makes it difficult to learn lessons or track account- Of the 41 projects, quality of M&E arrangements at ability of project investments. entry was assessed for 40. Of these, 27 (68 percent) were rated as moderately satisfactory or above. The country case studies and the review of termi- nal evaluations provide the following lessons on The Evaluation Office continued with the analy- capacity development activities. Capacity develop- sis presented in the FY 2006 APR where the asso- ment at the systemic level must be realistic. Overly ciation between the ratings for quality of M&E ambitious goals to change policies or laws may, arrangements at entry and quality of project mon- when not attained, adversely affect implementa- itoring during implementation was assessed. Such tion of other project components. Institutional an assessment was possible for only 32 projects. strengthening requires baselines and assessments Of the 24 projects in the 2007 cohort whose qual- to determine how new capacity will be absorbed. ity of M&E arrangements at entry was rated to be Long-term training programs and capacities moderately satisfactory or above, for 18 (75 per- should be based on a national or regional context cent), the quality of project monitoring dur- rather than a dependence on one-shot training. ing implementation was rated to be in the same range. In contrast, of the eight projects whose Monitoring and Evaluation quality of M&E arrangements at entry was rated to be moderately unsatisfactory or below, only one Conclusion 4: There is a strong association (13 percent) was rated as moderately satisfactory between quality at entry of M&E arrange- or above in quality of project monitoring during ments and actual quality of monitoring during implementation. implementation. This confirms the findings of the FY 2006 APR, where a strong association between Of the 41 terminal evaluation reports submitted the two had been reported. during FY 2007, 33 (80 percent) provided suffi- cient information on M&E to allow the Evaluation Conclusion 5: There has been significant Office to rate quality of project monitoring during improvement in the overall quality of terminal implementation. Of the rated projects, 61 percent evaluation reports. However, further improve- were rated moderately satisfactory or above on ments are required in the reporting of financial quality of monitoring (see table 1.3). information. 6 GEF Annual Performance Report 2007 In FY 2007, 41 terminal evaluation reports were Despite improvement in the overall quality of submitted. This is considerably lower than the 66 reporting, financial reporting and analysis pre- submitted during FY 2006. The Evaluation Office sented in the terminal evaluation reports remains will investigate this phenomenon and report on it an area where there has been little improvement. in the next APR. Almost half of the terminal evaluation reports (44 percent) did not provide adequate information Of the 41 terminal evaluation reports submit- on aspects related to financial performance. Conse- ted in FY 2007, the quality of 39 (95 percent) was quently, it is difficult to assess the variance in actual rated to be moderately satisfactory or above. This financial performance and the effect of such varia- performance is better than that of earlier cohorts tion on results achieved by these projects. (see table 1.4). On all performance parameters assessed, a greater percentage of the terminal Financial reporting is primarily contingent on the evaluation reports provided sufficient informa- extent to which the respective Agency provides tion, thus enabling the Evaluation Office to assess timely financial information for terminal evalu- the performance of a greater percentage of proj- ation. This again is dependent on the financial ects. The underlying trend of improvement in the systems of the respective Agency. For example, of quality of terminal evaluation reports submitted the reports submitted to the Office since FY 2005, to the Evaluation Office becomes stronger when 70 percent of those from the World Bank provide the year of terminal evaluation report completion, adequate financial information, compared to only instead of year of submission, is used as a basis 41 percent from the United Nations Environ- for comparison. This improvement is a result of ment Programme (UNEP) and 37 percent from the quality control measures adopted by the GEF the United Nations Development Programme Agencies and guidance provided by the GEF Eval- (UNDP). This finding is not surprising, in that the uation Office. Bank is a financial institution that accords greater attention to financial reporting. Since it is difficult to make changes in financial systems, this remains Table 1.4 an area where more effort is required. The Evalu- Quality of Terminal Evaluation Reports ation Office will devote more attention to finan- cial reporting and will work with the GEF Agen- FY of report submission cies to find ways to improve performance on this Factor 2004 2005 2006 2007 parameter. No. of reports submitted 42 41 66 41 % without sufficient ­ 5 3 2 In November 2007, after the "Joint Evaluation of information on project outcomes the Small Grants Programme" was presented to % without sufficient infor- ­ 12 18 5 the GEF Council, information emerged regard- mation on sustainability ing ongoing investigations of which the evaluation of outcomes team was not aware. These investigations were % that did not report on 29 44 29 20 confidential. The Evaluation Office will fine-tune cofinance % without sufficient infor- 25 20 30 20 the guidelines on how confidential information mation on M&e on GEF activities should be shared with evalua- % rated moderately satis- 69 88 84 95 tors while maintaining the confidentiality of the factory or above in quality Agencies. 1. Background, Main Conclusions, and Recommendations 7 Greenhouse Gas Footprint criteria considered for assessment. Overall, the banks have made more progress in addressing the Conclusion 6: The GEF Agencies are on track in climate impacts of their internal operations than addressing the greenhouse gas emissions of have the other Agencies. their internal operations. However, most are in early stages of developing and adopting a All Agencies need to advance their efforts to comprehensive greenhouse gas management address their internal climate change impacts and strategy. further develop processes and systems to track The GEF Evaluation Office commissioned a sur- and manage their internal GHG inventory data. vey of the policieis and guidelines of GEF Agen- The Agencies should first focus their efforts on cies to broadly assess the extent to which they are completing a robust GHG inventory. Once they addressing the GHG emissions generated by their have assessed their climate risk through such an internal operations. GHG footprint­related pol- inventory, they will be well positioned to develop a icieis and guidelines of the 10 GEF Agencies were cost-effective and disciplined approach to reduc- reviewed. ing their impacts. This assessment was primarily based on a review of publicly available information. Supplementary Management Action Record information was gathered for some Agencies Conclusion 7: All 41 verified Council decisions by requesting relevant documents and through in the 2007 Management Action Record show a phone conversations. Since several of the Agen- level of adoption of medium and higher. For the cies are in the process of defining their approach to older Council decisions, 14 of 27 show substan- addressing GHG emissions from their operations, tial progress in level of adoption as compared to last year. some of the information in this report might be outdated and contain gaps. A set of climate evalu- This year's MAR tracks the level of adoption of ation criteria was developed for a review of the 46 Council decisions based on 12 GEF Evalua- GHG reduction approaches of the 10 Agencies. tion Office documents by presenting ratings from The climate policieis and guidelines of the Agen- GEF management and verification of these ratings cies were compared to these criteria. by the Evaluation Office. The Office was able to verify the adoption of 41 of these 46 decisions (see The findings show that the GEF Agencies are on table 1.5). Decisions that could not be verified yet track in addressing the GHG emissions of their are mainly related to project M&E standards and internal operations, but that most are in an early quality of supervision. The Evaluation Office will stage. Some Agencies do have commitments carry out thematic assessments in the future to to further address their internal impacts in the assess adoption of these Council decisions. future. For example, Agencies under the United Nations (UN) umbrella will be working toward The Office rated 18 (44 percent) of the 41 veri- reducing their GHG emissions as part of a UN- fied decisions as having been adopted by manage- wide climate-neutral target that they plan to meet ment at high or substantial levels. This represents in the future. Among the Agencies reviewed, the an improvement over last year, when 33 percent European Bank for Reconstruction and Develop- were in this range. In addition, 12 Council deci- ment, the Inter-American Development Bank, sions thus far have been fully adopted and there- and UNEP have begun to address most of the fore have been graduated from the MAR. Six of 8 GEF Annual Performance Report 2007 Table 9.1 Ratings of GEF Progress in Adopting Council Decisions GEF Evaluation Office rating Sum of Not possible management Management rating High Substantial Medium Negligible to verify yet ratings High 5 8 8 0 3 24 Substantial 1 4 3 0 1 9 Medium 0 0 12 0 0 12 Negligible 0 0 0 0 1 1 Not possible to verify yet 0 0 0 0 0 0 Sum of Office ratings 6 12 23 0 5 46 Note: Highlighted fields show agreement between the ratings of management and the GeF evaluation office; fields to the right of the diagonal represent higher ratings by management than by the evaluation office (except in the case of not possible to verify yet). these were adopted during FY 2006 and pertained The Evaluation Office found noticeable progress to the Evaluation of the Experience of Executing over last year in adoption of Council decisions Agencies under Expanded Opportunities in the requesting increased transparency in the GEF GEF. The remaining six were based on other eval- project approval process through an improved uations and were fully adopted during FY 2007. management information system. The Office assessed the recent launch of a new system, which Table 1.5 shows that the Evaluation Office and includes a country portal to provide information GEF management agreed on the rating of progress related to GEF projects at the country level, to be of adoption for 51 percent of Council decisions a concrete improvement. (21 of 41). For 46 percent of the decisions (19 of 41), the Evaluation Office gave a lower rating than Eight out of the nine Council decisions for which that given by management. Many of these lower ratings have not shown improvement since last ratings reflect the fact that even though the GEF year are related to the local benefits study and the has taken measures to address the decisions, these biosafety evaluation. Regarding the former, Coun- cannot yet be considered to demonstrate a high cil decisions stress the need to establish a system level of achievement. They also reflect the fact that ensures that local benefits are addressed in a that some proposals to the GEF Council have not more systematic way at all stages of the GEF proj- yet been approved. If and when the Council does ect cycle. The GEF Secretariat currently cannot approve these proposals, substantial adoption is verify the quality of this aspect in project design likely to occur. or implementation because it still has no system in place to involve specialist social development Thirty Council decisions from seven GEF Evalua- expertise in its project review processes. Regard- tion Office documents were included in both last ing biosafety, the Evaluation Office will rate sub- year's and this year's MAR. The Evaluation Office stantial adoption once the Council approves the was able to follow the progress of adoption of 27 of "Program Document for GEF Support to Biosafety these Council decisions (3 were rated as not pos- during GEF-4." sible to verify yet). Of these 27 Council decisions, 52 percent (14 out of 27) showed progress in their The Small Grants Programme (SGP) management level of adoption. has started to address various Council decisions 1. Background, Main Conclusions, and Recommendations 9 from the SGP joint evaluation through the GEF higher than the 75 percent target specified in the SGP Steering Committee. However, because the GEF-4 replenishment agreement, even though country coordinators are not represented on the these projects are not subject to this provision. Steering Committee, there is a risk that, inadver- The percentage of World Bank projects (87 per- tently, new proposals will not consider country cent) with ratings in the satisfactory range has operational issues and perspectives sufficiently. been significantly higher than the target. The inclusion of some senior country coordina- For parameter 13, improvement in performance, tors on the Steering Committee would allow for the quality of terminal evaluations has been the country program perspective to be heard in the assessed. As noted in conclusion 5 above, there has discussions that lead to SGP proposals addressing been substantial improvement in performance on GEF Council decisions. this dimension. UNEP and UNDP demonstrated A complete version of the MAR is available at the considerable improvements, and the quality of a GEF Evaluation Office Web site (www.gefeo.org). high percentage of terminal evaluation reports submitted by the World Bank continues to be in Performance Matrix the satisfactory range. The performance matrix provides a summary of the performance of the GEF Agencies and GEF 1.3 Recommendation Secretariat on 13 parameters, covering key areas Recommendation 1: The GEF Secretariat, in such as results, processes affecting results, effi- coordination with the GEF Agencies, should ciency, M&E, and learning. Several of the param- conduct a formal and in-depth survey to more eters included in the matrix are already assessed accurately and thoroughly assess GEF Agency efforts to reduce GHG emissions. by the Evaluation Office on an annual basis. Since performance ratings on these parameters fluctu- 1.4 Issues for the Future ate from year to year, running averages of two to four years, depending on the parameter, will be z The GEF Evaluation Office will investigate why used in the matrix (see chapter 9 and annex D the number of terminal evaluations submitted for methodological details on the performance during FY 2007 was lower than expected and matrix). Of the 13 parameters included in the per- will report on this in the next APR. formance matrix, ratings have been provided on z The GEF Evaluation Office will work with the 9. Note, however, that the information provided GEF Secretariat to develop better systems to for parameter 13, improvement in performance, track project completion. addresses only 1 of the parameter's 12 dimensions. z The GEF Evaluation Office will fine-tune the In the future, as data for more years become avail- guidelines on how confidential information on able, it will be possible to track improvement in GEF activities should be shared with the evalu- performance on a greater number of dimensions. ators while maintaining the confidentiality of Based on the review of terminal evaluation reports the Agencies. submitted to the Evaluation Office since FY 2005, z The GEF Evaluation Office will work with the the Office rated outcome achievement (parame- GEF Secretariat, GEF Trustee, and GEF Agen- ter 1) in 81 percent of the projects to be moderately cies to facilitate better financial reporting for satisfactory or above. As noted, this percentage is completed projects. 10 GEF Annual Performance Report 2007 Notes 2. The complete list of the terminal evaluation reports reviewed for FY 2007 is provided in 1. The GEF fiscal year, like that of the World Bank, annex B. runs from July 1 to June 30. FY 2007, the primary focus of this report, comprises the period from 3. The GEF-4 replenishment period runs from 2006 July 1, 2006, to June 30, 2007. to 2010. 1. Background, Main Conclusions, and Recommendations 11 2. Scope and Methodology 2.1 Scope to which cofinancing promised at the point of project endorsement has materialized and on Each year, the APR provides an assessment of delays in project completion. The assessment the performance of completed GEF projects, on materialization of cofinancing is based on analysis of the processes that affect accomplish- figures reported by the respective GEF Agen- ment of results, and findings of the GEF Evalua- cies. The Evaluation Office will continue to tion Office's oversight of project monitoring and report on these issues on an annual basis. evaluation activities across the portfolio. The APR also provides the GEF Council, other GEF z A detailed assessment of the extent to which entities, and stakeholders with feedback to help capacity development activities in GEF improve the performance of GEF projects. Some projects are relevant, efficient, and effec- issues are addressed by the APR annually, some tive and on the results and sustainability of biennially; others are addressed whenever there these activities (chapter 5). This is the first is a need to do so. The 2007 APR includes the time the Evaluation Office has undertaken this following: assessment. The experience gained through recent field studies undertaken by the Office z An overview of the extent to which GEF in the Philippines and Vietnam facilitated the projects are achieving their objectives (chap- development of an effective methodology for ter 3). This overview consists of the Evalua- reviewing GEF project capacity development tion Office's assessment of the extent to which targets and achievements as reported by the the completed projects for which terminal terminal evaluation reports submitted to the evaluation reports were submitted in FY 2007 Office. For this assessment, 42 terminal evalua- achieved expected outcomes and the risks to tions, including 41 submitted to the Office dur- sustainability of achieved outcomes. The APR ing FY 2007, were reviewed. will continue to report annually on attainment z An assessment of the quality of project mon- of objectives and outcomes, and on sustainabil- itoring (chapter 6). This includes an examina- ity of outcomes. tion of the quality of M&E at project comple- z A presentation of the materialization of tion and an assessment of the quality at entry project cofinancing and delays in project of M&E arrangements of completed projects. completion reported by the GEF Agencies The APR reports on these M&E issues on an (chapter 4). This APR reports on the extent annual basis. 12 z An assessment of the quality of terminal particularly for those topics that are reported on evaluation reports submitted by the GEF annually. Ensuring the reliability of these reports Agencies to the Evaluation Office (chap- is therefore critical. The Evaluation Office seeks ter 7). This assessment, which is presented to assess and strengthen this reliability in several annually, provides information on the quality ways, as described below. of terminal evaluation reports by focal area and Agency and assesses trends in terminal evalua- The Evaluation Office reviews terminal evalua- tion report quality. This year, the Office piloted tion reports to determine the extent to which they a methodology for direct verification of termi- address the objectives and outcomes set forth in nal evaluations. Findings of direct verification the project document, to evaluate their internal will be included in future APRs. consistency, and to verify that ratings are properly substantiated. z A review of the policies and guidelines of the GEF Agencies in addressing the carbon foot- The reports are reviewed by Evaluation Office print of their operations (chapter 8). For the staff using a set of detailed guidelines to ensure first time, the Evaluation Office has reviewed that uniform criteria are applied (see annex A for the policies and guidelines of the GEF Agencies these guidelines). When deemed appropriate, a addressing the carbon footprint of their opera- reviewer may propose to upgrade or downgrade tions. The review involved a desk study of these project ratings in a terminal evaluation report. policies and guidelines and interviews with rel- evant staff members of the respective Agencies. The reviews are also examined by a peer reviewer with substantial experience in reviewing termi- z A presentation of findings on management nal evaluations. The peer reviewer provides feed- action records (chapter 9). This assessment, back on the review, which is incorporated by the which is presented annually, reviews and fol- primary reviewer in subsequent versions of the lows up on the implementation status of evalua- review. tion recommendations that have been accepted by management and/or the GEF Council. When a project is downgraded below moderately z A presentation of the performance matrix satisfactory (for outcomes) or below moderately (chapter 10). This assessment, which is being likely (for sustainability), a senior evaluation offi- presented for the first time in this APR, will cer in the GEF Evaluation Office also examines be presented annually. It summarizes the per- the review to ensure that the new ratings are justi- formance of the GEF Implementing Agencies fied. The reviews are then shared with the Agen- and the GEF Secretariat on key performance cies, and, after their feedback is taken into con- parameters. sideration, the reviews are finalized. If a terminal evaluation report provides insuffi- 2.2 Methodology cient information to make an assessment or verify Ensuring Reliability and Timeliness of the Agency ratings on outcomes, sustainability, Terminal Evaluation Reports or quality of project M&E systems, the Evalua- The project terminal evaluation reports submitted tion Office classifies the corresponding project as by the GEF Agencies to the Evaluation Office form unable to assess and excludes it from any further the core information source for much of the APR, analysis on the respective dimension. 2. Scope and Methodology 13 The most pervasive limitation of this review pro- projects. The UNDP Evaluation Office has yet to cess is that it is ultimately based on the information begin reviewing project performance in terms of provided by the terminal evaluation reports. Pilot- outcomes, sustainability of outcomes, and imple- ing of direct verification of the reports for a ran- mentation of M&E. The GEF Evaluation Office domly selected sample of completed projects was will continue its dialogue with the GEF Agencies, aimed at developing a methodology to address this while reviewing their terminal evaluation reports limitation in future. This year, verifications were and verifying their ratings. conducted for three completed full-size projects The GEF Evaluation Office has been tracking the for which terminal evaluation reports had been time between project completion and submission submitted to the Evaluation Office during FY 2007. of terminal evaluation reports and between report Beginning with next year's APR, field verification completion and submission. Although the time will be incorporated in the terminal evaluation lag between completion and submission has been review process and these findings presented. declining, many reports are still being submitted Another way to address the reliability concerns after more than the two months that is the nego- pertaining to terminal evaluation reports is to tiated norm for this task. The World Bank has a work with the GEF Agencies to more fully engage system for automatic submission of an electronic their evaluation units in the process and, when version of implementation completion reports for necessary, to strengthen their independence. its GEF-supported projects when these reports Presently, the World Bank's terminal evalua- are completed. The other GEF Agencies have not tion process meets most of the concerns of the yet developed such systems. The Office will work GEF Evaluation Office. The Bank's Independent with the Agencies so that timely submission of Evaluation Group (IEG) conducts desk reviews terminal evaluation reports can be ensured. of all the implementation completion reports produced by management for FSPs and conducts Data Limitations direct verification of these reports for a sample The Evaluation Office used the F-test and chi of projects. square test to assess differences among groups of projects, and the findings reported here are sig- Beginning in FY 2006, the UNEP Evaluation Office nificant at the 90 percent or higher confidence provided ratings and commentary on the quality level. Regression analysis was used to assess the of the terminal evaluation reports for its com- magnitude and direction of change associated pleted GEF projects. This fiscal year, it increased with different variables. This fiscal year, there was the scope of its commentaries by also assessing improvement in the quality of information pro- project outcomes, sustainability of outcomes, and vided in terminal evaluation reports on param- implementation of M&E based on the evidence eters such as sustainability and project M&E, provided in the terminal evaluation reports. How- which in the past have been identified as areas ever, since only a few assessments have been com- where the terminal evaluations are deficient in pleted so far, it is still too early to determine the providing adequate information. However, there overall reliability of the UNEP ratings. Similarly, has been little improvement in the quality of infor- the UNDP Evaluation Office this fiscal year began mation provided on materialization of cofinanc- to provide commentary on the quality of termi- ing. If sufficient information on a performance nal evaluations for some of its completed GEF parameter for a project has not been provided in 14 GEF Annual Performance Report 2007 its terminal evaluation report, that project has not and are relevant to the GEF Agencies and entities. been included in the portfolio-level assessment Using these criteria, the policies and guidelines of for that parameter. the GEF Agencies have been assessed. Since data are now available for a greater number MARs Assessment of completed projects and over a longer period of years, the GEF Evaluation Office is better able to Management action records facilitate review and predict trends and assess differences in perfor- follow-up on the implementation status of evalu- mance. However, assessing the performance of ation recommendations that have been accepted completed projects in terms of their outcomes, by management (that is, the GEF Secretariat and/ sustainability of outcomes, and implementation of or the GEF Agencies) and/or the GEF Council. M&E reflects actions that are now long past, limit- For each MAR, the Evaluation Office completes ing the extent to which information gathered from the columns pertaining to recommendations, analysis of these data is useful in making real-time management responses, and Council decisions. corrections in operations. Notwithstanding this Management is invited to provide a self-rating limitation, this assessment provides a long-term of the level of adoption of Council decisions on perspective on the extent to which GEF projects recommendations and add any comments neces- are achieving their objectives. sary. After management's response is included in a MAR, the Evaluation Office verifies actual adop- Assessment of Capacity Development tion and provides its own ratings, with comments, Activities in time for presentation to the Council. In this APR, the Evaluation Office presents an assessment of the extent to which capacity devel- Performance Matrix opment activities in GEF projects are relevant, This APR marks the first appearance of the per- effective, and efficient, and the results and sus- formance matrix, which provides a summary of tainability of the results of these activities. Capac- the performance of the three GEF Implement- ity-building activities were assessed at the indi- ing Agencies and the GEF Secretariat on relevant vidual, institutional, and systemic levels. Detailed parameters. Performance on most of these param- country case studies were conducted in Vietnam eters, such as project outcomes, implementation and the Philippines. The 42 terminal evaluation completion delays, materialization of cofinanc- reports, including 41 submitted to the Office dur- ing, quality of M&E during project implementa- ing FY 2007, were also reviewed. This allowed tion, and quality of project terminal evaluations, the Office to assess the extent to which findings is already being assessed annually by the GEF based on the analysis of the data from these two Evaluation Office. Performance on other param- approaches were consistent. eters, such as quality of supervision and adap- tive management, realism of risk assessment, and Review of Carbon Footprint­Related quality of project M&E arrangements at entry, is Policies and Guidelines being assessed every two or three years. For these This APR presents a set of criteria to track progress latter parameters, performance will be updated by the GEF Agencies in addressing their carbon every few years through special appraisals. The footprint. The criteria are based on emerging insti- information presented in the performance matrix tutional approaches to reducing GHG emissions of this APR (see table 10.1) is from the APRs for 2. Scope and Methodology 15 FYs 2005 and 2006. For assessing performance on indicators and tracking tools will be addressed in project preparation elapsed time, assessments will the future. Given the highly specialized and tech- be presented based on the GEF Project Manage- nical nature of this assessment, the GEF Evalua- ment and Information System database. At pres- tion Office will undertake it as part of the program ent, the information provided by the database on studies conducted for the Fourth Overall Perfor- this parameter is not reliable, so such assessments mance Study (OPS4) being prepared in FY 2008 will be carried out when there is a substantial and will enlist assistance from appropriate techni- improvement in the relevant information. cal experts. Four parameters that have been included in the performance matrix will require the develop- Review of Findings ment of new methodologies and approaches. The preliminary findings of this report were pre- This APR reports on progress in developing such sented to and discussed with the GEF Secretariat methodologies and approaches for assessing and GEF Agencies during an interagency meeting three of these parameters: independence of ter- held in Washington, D.C., in March 2008. Individ- minal evaluations, independence of GEF partner ual reviews of project terminal evaluation reports Agency evaluation units, and quality of learn- were shared with the Agencies and GEF Secretar- ing. Assessment of robustness of program result iat for comment. 16 GEF Annual Performance Report 2007 3. Project Outcomes and Sustainability This chapter discusses verified ratings on outcomes in the FY 2006 cohort (65 percent) and just under and sustainability for 41 projects (20 full-size and half of those in the FY 2005 cohort (49 percent) 21 medium-size projects) for which terminal eval- (see figure 3.2). Similarly, 51 percent of FY 2007 uation reports were submitted in FY 2007. This is projects were rated both moderately satisfactory the third time the Evaluation Office has rated out- and above in their outcome ratings and moder- comes and sustainability of GEF projects. ately likely or above in their sustainability ratings; the comparable percentages from prior years were Seventy-three percent of the projects assessed 61 percent in FY 2006 and 44 percent in FY 2005. this fiscal year were rated moderately satisfactory Because of fluctuations in cohort sustainability or above with regard to attainment of project out- ratings, it is too early to conclude whether the comes. Although the percentage of projects rated lower percentages for this year's cohort constitute in the satisfactory range is lower than for FYs 2005 a trend. (82 percent) and 2006 (84 percent), this difference is not statistically significant (see figure 3.1). Figure 3.2 More than half (58 percent) of the projects assessed Percentage of Projects with Sustainability Rated in FY 2007 were rated moderately likely or above Moderately Likely or Above in terms of the sustainability of their outcomes; Percentage of projects this was true for almost two-thirds of the projects 100 80 65% 59% Figure 3.1 60 49% 40 Percentage of Projects with Outcomes Rated 20 Moderately Satisfactory or Above 0 2005 2006 2007 (N = 39) (N = 54) (N = 39) Percentage of projects 100 82% 84% FY of terminal evaluation report submission 73% 80 60 40 20 3.1 Rating Approach 0 2005 2006 2007 The Evaluation Office rated project outcomes of (N = 39) (N = 64) (N = 40) FY of terminal evaluation report submission the FY 2007 cohort based on level of achievement of project objectives and expected outcomes on a 17 six-point scale. The criteria used to assess level of Of the 41 terminal evaluation reports submitted, achievement included assessment of ex ante out- 18 (44 percent) were for UNDP projects and 16 come relevance, actual effectiveness in achieve- (39 percent) were for World Bank projects (see ment of outcomes, and efficiency in achievement figure 3.3a). In the FY 2006 cohort, by contrast, of outcomes. Relevance and effectiveness were the majority of terminal evaluation reports were considered to be critical criteria; the overall rating for World Bank projects (64 percent), with UNDP on achievement of outcomes could not be higher comprising only 23 percent of the total. This shift in than the lower rating attained on either or both of project mix has been taken into account in assess- the critical criteria. The Office used the same rating ing the differences in ratings across fiscal years. approach for the FY 2006 cohort; a different rating approach had been used for FY 2005. To make the Figure 3.3 ratings comparable, the Office reassessed the 2005 Number of Terminal Evaluation Reports by cohort using the above-described approach; these GEF Agency and Focal Area ratings are referenced in the present analysis. a. By Agency b. By focal area The GEF Evaluation Office rated sustainability Multifocal of outcomes based on an assessment of four key Land 3 POPs, 1 degrad., 1 risk dimensions: financial, sociopolitical, institu- Int'l tional framework and governance, and environ- World waters, 2 Bank UNDP Bio- mental. Based on the evidence presented in the 16 18 diversity Climate terminal evaluation reports, risks to sustainabil- 21 change 13 ity of outcomes were assessed on each of these UNEP 7 dimensions. All risk dimensions were regarded as critical; overall ratings may not be higher than N = 41 projects the lower rating on any of these dimensions. Since the FY 2006 cohort was also assessed using this In terms of focal area representation, the FY 2007 approach, its ratings are directly comparable to cohort is similar to those for FYs 2005 and 2006. that for the 2007 cohort. Again, the original rat- For all these cohorts, a majority of projects were ings for the FY 2005 cohort were based on a dif- from the biodiversity focal area (see figure 3.3b for ferent approach. To make the ratings comparable the FY 2007 cohort). The Evaluation Office was not to those for 2006 and 2007, the Office reassessed able to rate the level of achievement of outcomes the FY 2005 cohort using the new approach for for one project and the level of sustainability of out- sustainability ratings; these ratings are referenced comes for two projects because insufficient infor- in this analysis. mation was provided in the respective terminal Continuing the analysis that was first presented in evaluation reports. Reports that did not provide suf- the FY 2006 APR, the Evaluation Office assessed ficient information on a particular dimension were the extent to which outcomes of projects that were excluded from further analysis of that dimension. rated moderately satisfactory or above in terms of their achievement of outcomes were also rated 3.2 Project Outcomes moderately likely or above in terms of the sustain- Of the 40 projects in the FY 2007 cohort whose ability of their outcomes. outcomes were rated by the Evaluation Office, 18 GEF Annual Performance Report 2007 73 percent were found to be moderately satisfac- Figure 3.4 tory or above (see figure 3.4a and table 3.1). Simi- Project Outcome Ratings by Number of Projects larly, of the total investment in the rated projects and Project Funding ($198 million), 69 percent was allocated to proj- a. By number of projects b. By GEF investment ects that were rated as having moderately satisfac- 3% 3% 5% tory or above outcomes (see figure 3.4b). Unlike 18% 13% 5% FY 2006 where the outcome ratings for FSPs and 19% MSPs were similar, for FY 2007 MSPs had higher 35% ratings than FSPs. For example, in FY 2007, the 8% 14% outcomes of 85 percent of MSPs were rated as 35% 48% moderately satisfactory or above by the Evaluation Office compared to 60 percent for FSPs. The dif- ferences in performance by Agency and focal area N = 40 projects; total GEF funding = $199 million were not statistically significant. Highly satisfactory Moderately unsatisfactory By outcome criterion, all of the rated projects Satisfactory Unsatisfactory were found to be moderately satisfactory or above Moderately satisfactory Highly unsatisfactory in ex ante outcome relevance, 73 percent were rated moderately satisfactory or above on effec- tiveness in achievement of outcomes, and 72 per- total investment in rated projects of $198 mil- cent were so rated for efficiency of achievement of lion, $137 million (69 percent) was invested in outcomes (see figure 3.5). projects whose outcomes were rated moderately The GEF investment in individual projects var- satisfactory or above (see table 3.2). By Agency, ied from $0.5 million to $25.0 million. Of the the GEF investment in projects rated moderately Table 3.1 Project Outcomes by GEF Agency and Focal Area Number of projects GEF Agency Focal area Rating UNDP UNEP WB BD CC IW LD MF POPs Total Highly satisfactory 1 0 0 0 1 0 0 0 0 1 Satisfactory 5 1 8 7 5 1 0 1 0 14 Moderately satisfactory 5 5 4 8 4 1 0 0 1 14 Subtotal 11 6 12 15 10 2 0 1 1 29 Moderately unsatisfactory 2 0 1 3 0 0 0 0 0 3 Unsatisfactory 5 1 1 3 2 0 1 1 0 7 Highly unsatisfactory 0 0 1 0 1 0 0 0 0 1 Subtotal 7 1 3 6 3 0 1 1 0 11 Unable to assess 0 0 1 0 0 0 0 1 0 1 Total 18 7 16 21 13 2 1 3 1 41 Note: bD = biodiversity; cc = climate change; IW = international waters; LD = land degradation; MF = multifocal; PoPs = persistent organic pol- lutants; Wb = World bank. 3. Project Outcomes and Sustainability 19 Figure 3.5 The outcomes for one project, UNDP's Barrier Percentage of Projects with Outcome Dimensions Removal for the Widespread Commercialization of Rated Moderately Satisfactory or Above Energy-Efficient CFC-Free Refrigerators in China (GEF ID 445), were rated as highly satisfactory, Percentage of projects 100% because it had fully achieved and exceeded its 100 expected outcomes. This project was instrumen- 80 73% 72% tal in reducing the average energy intensity of new 60 refrigerators sold by nearly 30 percent between 40 project inception in 1999 and 2005 and had cost- 20 effectively contributed to reductions in GHG emissions. (Note that the outcomes of this project 0 Relevance Effectiveness Efficiency were rated by the Evaluation Office as moderately (N = 41) (N = 40) (N = 39) likely to sustain.) Outcome achievement in 14 other projects was rated as satisfactory. Among the projects that were rated as having sat- satisfactory or above was 78 percent for World isfactory or above outcomes, 85 percent were rated Bank projects, 66 percent for UNEP projects, moderately likely or above on sustainability of out- and 60 percent for UNDP projects. By focal area, comes and 90 percent were rated moderately satis- 74 percent of the GEF investment for climate factory or above on implementation of M&E. change, and 62 percent for biodiversity, went toward projects whose outcomes were rated mod- The only project the Evaluation Office rated as erately satisfactory or above. highly unsatisfactory in terms of its outcomes Table 3.2 Outcome Ratings for FY 2007 by GEF Investment Million $ GEF Agency Focal area Rating UNDP UNEP WB BD CC IW LD MF POPs Total Highly satisfactory 9.9 0.0 0.0 0.0 9.9 0.0 0.0 0.0 0.0 9.9 Satisfactory 20.7 0.8 14.6 5.4 13.6 16.2 0.0 0.8 0.0 36.9 Moderately satisfactory 20.8 10.1 60.0 69.4 19.9 0.6 0.0 0.0 1.0 90.9 Subtotal 51.4 10.9 74.6 74.8 43.4 16.8 0.0 0.8 1.0 136.9 Moderately unsatisfactory 17.0 0.0 10.2 27.1 0.0 0.0 0.0 0.0 0.0 27.1 Unsatisfactory 18.0 5.6 0.8 17.7 5.4 0.0 0.5 0.8 0.0 24.3 Highly unsatisfactory 0.0 0.0 10.0 0.0 10.0 0.0 0.0 0.0 0.0 10.0 Subtotal 34.9 5.6 20.9 44.8 15.4 0.0 0.5 0.8 0.0 61.5 Unable to assess 0.0 0.0 1.0 0.0 0.0 0.0 0.0 1.0 0.0 1.0 Total 86.3 16.5 96.6 119.7 58.8 16.8 0.5 2.6 1.0 199.4 Note: bD = biodiversity; cc = climate change; IW = international waters; LD = land degradation; MF = multifocal; PoPs = persistent organic pol- lutants; Wb = World bank. 20 GEF Annual Performance Report 2007 was the World Bank­implemented Solar Devel- 3.3 Project Outcomes Sustainability opment Group global project (GEF ID 595), which aimed at providing electrical services Of the 41 terminal evaluation reports submitted to the poor. The GEF had approved $10.0 mil- in FY 2007, 2 (5 percent) did not provide sufficient lion for this project, which became effective in information to allow assessment of sustainability FY 2001. Due to a lack of progress, the project of outcomes. In previous fiscal year cohorts, the was canceled in 2004; at that time, $1.3 million corresponding percentages of reports with insuf- of the approved GEF grant had been spent. For- ficient information for assessment of sustainabil- mulation of this project was driven more by the ity were 18 percent in FY 2006 (12 of 66 projects) priorities of the project sponsors than by real- and 12 percent in FY 2005 (5 of 41 projects). Thus, istic assessment of the market opportunities the extent to which terminal evaluation reports in the participating developing countries. This provide information on sustainability of outcomes deficiency, coupled with an overly complicated appears to be improving substantially. implementation structure, led to the project per- Of the 39 projects that were rated on sustain- forming well below expectations. The Evalua- ability of outcomes, 23 (59 percent) were rated as tion Office rated outcome achievement for seven moderately likely or above (see table 3.3 and fig- other projects as unsatisfactory. ure 3.6a). In terms of GEF investment, 55 percent Among the projects that were rated unsatisfac- was invested in projects that were rated as mod- tory or below in outcomes, only 20 percent were erately likely or above in the sustainability of their rated as moderately likely or above to sustain the outcomes (see table 3.4 and figure 3.6b). The sus- achieved outcomes, and 43 percent were rated tainability ratings of MSPs were higher than those moderately satisfactory or above on implementa- for FSPs: 75 percent of the former were rated tion of M&E. moderately likely or above on their sustainability Table 3.3 Sustainability of Project Outcomes by GEF Agency and Focal Area Number of projects GEF Agency Focal area Rating UNDP UNEP WB BD CC IW LD MF POPs Total Likely 1 0 2 0 2 0 0 1 0 3 Moderately likely 8 4 8 14 5 1 0 0 0 20 Subtotal 9 4 10 14 7 1 0 1 0 23 Moderately unlikely 6 3 3 5 4 0 1 1 1 12 Unlikely 2 0 2 2 2 0 0 0 0 4 Subtotal 8 3 5 7 6 0 1 1 1 16 Unable to assess 1 0 1 0 0 1 0 1 0 2 Total 18 7 16 21 13 2 1 3 1 41 Note: bD = biodiversity; cc = climate change; IW = international waters; LD = land degradation; MF = multifocal; PoPs = persistent organic pol- lutants; Wb = World bank. 3. Project Outcomes and Sustainability 21 Figure 3.6 rated projects faced at least significant financial Sustainability Ratings by Number of Projects and risks. Twenty-eight percent of the rated projects Project Funding faced significant institutional and governance- related risks (see figure 3.7). a. By number of projects b. By GEF investment 10% 13% 1% 8% Figure 3.7 Perceived Risks Underlying Projects Receiving Low Sustainability Ratings 53% 51% Percentage of projects 50 31% 33% Moderately unlikely 40 Unlikely N = 39 projects; total GEF funding = $183 million 30 30% Likely Moderately unlikely 20 Moderately likely Unlikely 23% 10 20% 11% 10% 3% 0 of outcomes compared to only 42 percent of the Financial Socio- Institu- Environ- (N = 37) political tional mental latter. This finding differs from that for FY 2006, (N = 40) (N = 39) (N = 30) when ratings for FSP and MSP sustainability of outcomes were similar. The reasons for this varia- tion are not clear. Sustainability of outcomes was rated unlikely for Among the risk dimensions that were assessed four (11 percent) projects because of high finan- to rate sustainability of outcomes, financial risks cial risks. The causes of the high level of financial were the most pervasive--about 41 percent of the risks for these projects were as follows: Table 3.4 Sustainability of Outcome Ratings for FY 2007 by GEF Investment Million $ GEF Agency Focal area Rating UNDP UNEP WB BD CC IW LD MF POPs Total Likely 0.7 0.0 1.8 0.0 1.7 0.0 0.0 0.8 0.0 2.5 Moderately likely 36.3 7.8 52.8 75.1 21.2 0.6 0.0 0.0 0.0 96.9 Subtotal 37.0 7.8 54.7 75.1 22.9 0.6 0.0 0.8 0.0 99.4 Moderately unlikely 29.8 8.7 20.8 31.9 25.1 0.0 0.5 0.8 1.0 59.2 Unlikely 3.3 0.0 20.2 12.7 10.8 0.0 0.0 0.0 0.0 23.5 Subtotal 33.1 8.7 41.0 44.6 35.9 0.0 0.5 0.8 1.0 82.8 Unable to assess 16.2 0.0 1.0 0.0 0.0 16.2 0.0 1.0 0.0 17.2 Total 86.3 16.5 96.7 119.7 58.8 16.8 0.5 2.6 1.0 199.4 Note: bD = biodiversity; cc = climate change; IW = international waters; LD = land degradation; MF = multifocal; PoPs = persistent organic pol- lutants; Wb = World bank. 22 GEF Annual Performance Report 2007 z Lower outcome achievement by the project projects of the FY 2007 cohort, 51 percent of dissuaded potential donors from investing in the projects--and 44 percent of the GEF project the planned follow-up activities (UNDP's Pilot investment--were rated moderately satisfactory or Production and Commercial Dissemination of above in outcomes and moderately likely or above Solar Cooker project in South Africa, GEF ID in sustainability (see figure 3.8). Compared to this, 1311). 61 percent of the projects and 56 percent of the investment for the FY 2006 cohort had been rated z The sources of income for sustaining the fol- moderately satisfactory or above in outcomes and low-up activities were not as significant as had moderately likely and above in sustainability of been expected (UNDP's Conserving Mountain outcomes. For the FY 2005 cohort, 44 percent of Biodiversity in Southern Lesotho project, GEF projects and 60 percent of investments had been ID 245). so rated. The difference between the ratings by z The market for the products promoted by the cohort is not statistically significant. project were not as significant as had been expected (World Bank's Solar Development Figure 3.8 Group project). Projects with Outcomes/Sustainability Rated as z The financial support received from the gov- Moderately Satisfactory/Moderately Likely or Above ernment for continuation of project follow- up activities was lower than expected (World a. By number of projects Bank's Central Asia Transboundary Biodiver- Percentage of projects 100 sity Project, GEF ID 110). 80 73% 59% 60 51% 3.4 Outcomes and Outcomes 40 Sustainability 20 0 In the FY 2006 APR, the Evaluation Office for the b. By GEF investment first time presented an assessment of the extent to 100 which projects that are rated moderately satisfac- 80 69% 60 55% tory or above in achievement of their outcomes are 44% 40 also rated moderately likely or above in the sustain- 20 ability of their outcomes. For FY 2007, 39 reports 0 Outcomes Sustainability Outcomes and (95 percent) provided sufficient information to (N = 40; (N = 39; sustainability $193.8 million) $182.1 million) (N = 39; allow assessment of both project outcomes and $182.1 million) sustainability of project outcomes. For the rated 3. Project Outcomes and Sustainability 23 4. Factors Affecting Attainment of Project Results 4.1 Materialization of Cofinancing 6 of these (15 percent), this amount was less than half of that promised. The figures for materializa- Of the 41 projects for which terminal evaluation tion of cofinancing are skewed due to the inclu- reports were submitted in FY 2007, information sion of the World Bank's Energy Efficiency Project on materialization of cofinancing was provided in Brazil (GEF ID 128) in the cohort. This project for 33. The Evaluation Office followed up with the alone accounts for about half of the total short- respective GEF Agencies and thereby obtained fall in cofinancing materialization. The GEF had information on an additional six projects. approved a grant of $15.0 million for this project, of which $11.9 million was actually used. Of the For the 39 projects for which information on cofi- $105.5 million promised in cofinancing for this nancing was reported, an average of $2.60 was project, only $38.1 million materialized. promised per $1.00 of approved GEF grant, and an average of $1.90 (75 percent) was reported to In FY 2006, a lower amount of cofinancing ($2.10) have materialized (see table 4.1 and figure 4.1). had been promised per GEF dollar than in FY 2007 For 21 projects (54 percent), the materialized cofi- but a higher amount was reported to have materi- nancing was lower than the promised amount; for alized ($2.40). Table 4.1 Materialization of Cofinancing FY of report submission Up to All Factor 2002 2003 2004 2005 2006 2007 2006 years Number of projects with cofinancing data available 11 7 30 23 47 39 118 157 Data obtained from terminal evaluation reports 11 7 30 23 47 33 118 151 Data otherwise obtained 0 0 0 0 0 6 0 6 Approved GeF grant per project (million $) 6.2 5.5 5.9 6.4 3.5 4.8 5.0 5.0 Actual GeF grant per project (million $) 6.1 4.6 4.6 6.3 3.3 4.4 4.6 4.5 Promised cofinancing per project (million $) 29.5 8.4 43.2 9.5 7.2 12.8 19.0 17.4 Promised cofinancing per $1.00 of approved GeF grant 4.70 1.50 7.40 1.50 2.10 2.60 3.80 3.5 reported materialized cofinancing per project (million $) 29.2 7.1 38.9 10.0 8.2 9.2 18.3 16.0 reported materialized cofinancing per $1.00 of approved 4.70 1.30 6.60 1.60 2.40 1.90 3.60 3.2 GeF grant Materialized cofinancing per $1.00 of promised cofinancing (%) 99 85 90 106 114 72 96 92 24 Figure 4.1 Reported Materialization of Cofinancing per Dollar of Approved GEF Grant 8 7.4 Promised co nancing Actual co nancing 7 6.6 6 5 4.7 4.7 4 3.5 3.2 3 2.6 2.4 2.1 1.9 2 1.5 1.5 1.6 1.3 1 0 2002 2003 2004 2005 2006 2007 All years (N = 11) (N = 7) (N = 30) (N = 23) (N = 47) (N = 39) (N = 118) FY of terminal evaluation report submission Note: Data for 2004 include the World bank's multimillion-dollar china Sichuan Gas transmission and Distribution rehabilitation Project (GeF ID 75). For the FY 2007 cohort, a higher level of cofinanc- Following up on the discussion presented in the ing was both promised and reported to have mate- FY 2006 APR, the Evaluation Office reviewed rialized for World Bank projects ($4.10 and $3.80, the evidence presented in the terminal evalua- respectively, per GEF dollar) than for those of tion reports submitted during FY 2007 to explore the other Implementing Agencies (see table 4.2). the causal relationships between cofinancing and However, there are substantial differences among project processes and results. Both causes that the Agencies in terms of (1) the nature of the are internal and/or external to the project could projects undertaken, some of which may be eas- be responsible for lower than expected levels of ier to raise contributions for than others; and (2) cofinancing materialization. The most common the type of cofinancing raised--loans to the host internal cause reported for this year's cohort was country (which are relatively easy to raise) versus inability of projects to show progress in ear- grants from other donors (which take more effort lier stages of their implementation, resulting in to obtain and do not require repayment). These withdrawal of support by non-GEF donors. For differences introduce complexities in the analy- example, the non-GEF donors did not fulfill their sis of materialization of cofinancing and will be commitments to as the World Bank's Renewable explored further in future APRs so that a more Energy and Forest Conservation project in Nica- well-grounded comparison of the performance of ragua (GEF ID 847), UNDP's Pilot Production the IAs in raising cofinancing is possible. and Commercial Dissemination of Solar Cooker 4. Factors Affecting Attainment of Project Results 25 Table 4.2 Materialization of Cofinancing by GEF Agency, 2002­07 World Multi- Factor UNDP UNEP Bank Agency Total Number of projects with cofinancing data available 45 21 90 1 157 Data obtained from terminal evaluation reports 41 20 89 1 151 Data otherwise obtained 4 1 1 0 118 Approved GeF grant per project (million $) 3.3 2.7 6.2 20.2 5.0 Actual GeF grant per project (million $) 3.2 2.7 5.5 20.2 4.5 Promised cofinancing per project (million $) 5.0 4.4 25.4 134.8 17.4 Promised cofinancing per $1.00 of approved GeF grant 1.50 1.60 4.10 6.70 3.50 reported materialized cofinancing per project (million $) 4.2 4.1 23.7 103.6 16.0 reported materialized cofinancing per $1.00 of approved GeF grant 1.30 1.50 3.80 5.10 3.2 Materialized cofinancing per $1.00 of promised cofinancing (%) 84 94 94 77 92 project in South Africa, and the World Bank's Some completed projects reported greater than global Solar Development Group project because expected cofinancing to have materialized. Rea- these projects had not made adequate progress sons for better than expected cofinancing during their early stages of implementation. An materialization include, among others, donors endogenous cause that affected materialization contributing higher than expected amounts of cofinancing was savings made during proj- for a particular project component. In some ect implementation. For example, for the World instances, fluctuations in international currency Bank's Renewable Energy for Agriculture proj- markets also lead to higher contributions from ect in Mexico (GEF ID 643), savings were made donors that contribute in nondollar denomina- during implementation of project components. tions; this was the case for UNEP's Community- Although these savings were passed on to other based Management of On-Farm Plant Genetic donors, they were not reflected in lower usage of Resources in Arid and Semi-Arid Areas of Sub- GEF resources. Saharan Africa project (GEF ID 981). An important exogenous cause for lower realiza- The terminal evaluation reports still do not pro- tion of cofinancing is financial crisis in the host vide sufficient evidence to allow for an assessment country. For example, for projects such as the of the extent to which cofinancing was essential World Bank's Energy Efficiency Project, Sustain- in a project's achieving its global environmental ability of the National System of Protected Areas results. This prohibits assessment of the extent project in Bolivia (GEF ID 620), and Landfill Meth- to which the reported materialized cofinancing is ane Recovery Demonstration Project in Uruguay consistent with the manner in which it is defined (GEF ID 766), the host governments did not meet by the GEF. their commitments as a result of a financial crisis faced by their country. In other instances where materialization of cofinancing was lower than 4.2 Delays in Project Completion expected, the causal variables were not discussed The Evaluation Office measured the time differ- in the terminal evaluation report. ence between expected closing at project start and 26 GEF Annual Performance Report 2007 actual closing. The average project completion completed after a delay of 19 months; the average delay for the FY 2007 cohort was 13 months. This delay for medium-size projects was 9 months. was the same figure as for the FY 2006 cohort; the Following up on the discussion presented in the average delay for FY 2005 was 17 months.1 FY 2006 APR, the Evaluation Office reviewed When considering all 140 projects from the the evidence presented in the terminal evalua- FYs 2007, 2006, and 2005 cohorts for which data tion reports submitted in FY 2007 to explore the on completion delays are available, GEF projects causal relationships of completion delays and average a 15-month delay in completion. UNDP project processes and results. Review of the causes projects have significantly longer delays than those for delay shows that while contextual factors are for the other Agencies, averaging 21 months from important, delays are in most cases primarily scheduled to actual completion (see figure 4.2a). due to internal factors such as overly optimistic Climate change projects, with an average delay project design, communication problems among of 22 months between planned and actual com- project partners, delays in transfer of money to pletion, experience significantly longer delays the partners, cumbersome implementation pro- compared to those in the other focal areas (see cedures, and delays in hiring key staff. figure 4.2b). Full-size projects were, on average, Delayed transfer of funds to project partners was reported as a cause of delay for four projects. Figure 4.2 For the Renewable Energy and Forest Conserva- Average Delay in Project Completion by GEF tion Project in Nicaragua, this occurred because Agency and Focal Area, FYs 2005­07 the recipient organization was not well versed in World Bank disbursement procedures. The a. By Agency UNEP­United Nations Industrial Development UNEP FSPs (N = 22) All projects MSPs Organization global project Fostering Active and Effective Civil Society Participation in Preparations World Bank (N = 69) for Implementation of the Stockholm Convention (GEF ID 2067) experienced delays in funds trans- All Agencies (N = 140) fer because of communication problems between the GEF Agencies and the executing agency. For UNDP (N = 49) the other two projects--the World Bank's Mongo- lian Dynamics of Biodiversity Loss and Permafrost b. By focal area Melt in Lake Hovsgol National Park Project (GEF Biodiversity ID 984) and its global Coral Reef Monitoring Net- (N = 77) work Project (GEF ID 814)--the terminal evalu- All focal areas ation reports did not explain the reasons behind (N = 140) the delay in transfer of funds. Int'l waters (N = 17) Delays in accomplishment of critical activities Climate change may stall progress in accomplishment of other (N = 34) activities. In many instances, projects experi- 0 5 10 15 20 25 30 Average delay (in months) ence completion delays because of their inability to hire key staff in time, thus potentially delaying 4. Factors Affecting Attainment of Project Results 27 the project activities contingent on this hiring. may actually lead to improved performance and UNDP's Conservation of Globally Significant For- results. For example, due to efficient management est Ecosystems in Suriname's Guyana Shield Proj- of its Biodiversity Conservation and Integration ect (GEF ID 661) faced delays for this reason; thus, of Traditional Knowledge on Medicinal Plants the management team skills required to imple- in National Primary Health Care Policy (GEF ID ment important components of the project were 1410) regional project, UNEP was able to realize not available in a timely manner. In the case of the savings. Extension of the project duration allowed World Bank's Lake Hovsgol National Park Project, it to use the savings to undertake other relevant an inhospitable climate in the project area made it activities geared toward achieving global envi- difficult to undertake project activities during the ronmental benefits. Similarly, although persistent winter, necessitating a delay in accomplishing the conflicts had made the executing agency unable to activities scheduled for the summer months. undertake planned activities in the municipal area initially selected for implementation of UNDP's The design of some of the reviewed projects inad- Economic and Cost-Effective Use of Wood Waste equately addressed the complexities involved in for Municipal Heating Systems Project in Latvia implementation. Often, projects underestimated (GEF ID 914), increasing the project's duration the time it would take to consult with stakehold- allowed it to use resources to extend implementa- ers, communicate with government agencies and tion of project activities to other communities that local communities, and resolve local conflicts. For were equally suitable as a target for the project-- example, the World Bank's Central Asia Trans- and that exhibited no conflicts. boundary Biodiversity Project experienced a two- year completion delay while the project struggled External causes of project delay include unan- to begin operations in three different countries, ticipated political instability, changes in project each with its own set of complex and cumbersome processing requirements on the part of the host implementation procedures. The project design of government, and financial crisis in the host UNDP's Conservation of Wetland and Coastal Eco- country. For example, insurgency in the proj- systems in the Mediterranean Region Project (GEF ect area made timely implementation of UNDP's ID 410) was reported to have poor problem analy- Upper Mustang Biodiversity Project in Nepal sis and an outdated logical framework, leading to (GEF ID 30) difficult. A completion delay in the difficulties in project implementation. World Bank's Developing the Legal and Regula- tory Framework for Wind Power in Russia (GEF ID Delays during project implementation may have 2194) was occasioned by a reshuffling of staff and negative consequences such as overall inefficiency reorganization of Russian agencies, which made due to the allocation of a greater proportion of it difficult for the project team to negotiate and resources to administrative costs rather than proj- collaborate with key government staff. The World ect activities, an inability to accomplish some of Bank's Conservation of Pu Luong-Cuc Phuong the project deliverables, a reduction in project Limestone Landscape Project in Vietnam (GEF ID scope, lower accumulation of project benefits, and 1477) experienced an 18-month delay primarily decreased participation by project partners. due to the new project processing requirements However, taking additional time for project of the host country, which made synchronization implementation need not imply poorer perfor- of GEF-supported activities with those supported mance. In some instances, additional time taken by other donors difficult. 28 GEF Annual Performance Report 2007 Financial crisis may prevent host governments Project, one of the host governments (Zimba- from meeting their cofinancing commitments in a bwe), which was facing a foreign exchange crisis, timely manner, leading to delays in project compo- insisted on applying an unfavorable exchange rate nent start-up. Depending on the severity and extent for converting the GEF grant into local currency. of the crisis, projects may also be restructured It took time for the project management to negoti- because some components and activities may no ate with the Zimbabwe government and convince longer be viable; such restructuring may add to the it to agree to a more favorable rate. project's total duration. The World Bank's Energy Efficiency Project and its Landfill Methane Recov- Note ery Demonstration Project faced delays due to the 1. The figures reported here differ slightly from those financial crises that affected their respective host reported in the 2005 and 2006 APRs. The differ- ence derives from the inclusion of projects that countries. In UNEP's Community-Based Man- were completed ahead of schedule in the analysis agement of On-Farm Plant Genetic Resources in for this APR. Such projects are here treated as hav- Arid and Semi-Arid Areas of Sub-Saharan Africa ing been completed without any delay (on time). 4. Factors Affecting Attainment of Project Results 29 5. Evaluation of Capacity Development In FY 2007, the GEF Evaluation Office began across projects so that an aggregation of impacts work on the evaluation of GEF capacity develop- can be achieved. ment activities. To this end, the Office has com- pleted country case studies of the Philippines and The effectiveness of capacity development Vietnam, and has reviewed terminal evaluation activities has varied. In Vietnam, some activi- reports submitted during FY 2007. The country ties were effective in providing new skills and case studies examined the nature and results of institutional capacities that showed direct and national, regional, and global interventions and immediate results in the targeted sector; in other related these to capacity development targets cases, the activities had less immediate results, at the policy, institutional, and individual levels although benefits may develop in the longer term. in each country. These case studies have been In many cases in the Philippines, institutions have published on the GEF Evaluation Office Web site been unable to provide appropriate incentives for (www.gefeo.org). The review of the terminal evalu- trained staff, and opportunities to use new skills ation reports assessed the extent to which capacity have proved limited. Concerning efficiency, GEF development activities in GEF projects are relevant, capacity development activities have usually effective, and efficient and looks at the results and met their immediate output and outcome tar- sustainability of the results of these activities based gets, although a few projects have suffered unusual on evidence provided in the reports. delays in implementation. The cost-effectiveness of capacity development activities was difficult 5.1 Findings from the Country Case to assess, since the activities rarely constitute a Studies defined budget heading during project implemen- tation or monitoring. The country case studies found that the GEF portfolios in the Philippines and Vietnam include Although there are many improvements to capac- considerable capacity development activities. The ity at the individual, institutional, and systemic results are generally positive and, in some areas, levels, doubts exist about the sustainability of significant. GEF capacity development support a number of capacity development outcomes. has been relevant in terms of being in line with In the Philippines, several project designs lacked national policy priorities and has engendered a clarity about how the improved capacity will good level of national ownership. However, in be used, and there are limited incentives avail- the Philippines, there is no system that effectively able within the government system to retain and integrates the objectives of capacity development reward motivated and trained staff. 30 In Vietnam, the positive results reflect the substan- not yet available in the region. In the countries tial efforts put into individual-level capacity devel- reviewed, the international waters program has opment activities in many projects. The changing been most effective in utilizing and developing institutional environment in Vietnam also plays a regional training capacity. role in these results, since the need for new skills Based on the country case studies, the GEF Evalua- and approaches is recognized, and improvements tion Office delineated further work to help explain to individual capabilities are often accompanied by the impact of capacity development activities changes to institutional structures and procedures across the GEF portfolio. One component of this which mean these capabilities can be utilized. work is a meta-evaluation of capacity development The GEF Evaluation Office found that, both in findings based on a review of terminal evaluations the Philippines and Vietnam, there has been no and previous Evaluation Office reports. The team systematic monitoring or evaluation of overall has also held discussions with the GEF-UNDP- capacity development performance at the coun- UNEP Support Program for Capacity Develop- try level which could promote improvements in ment on indicators for climate change and has coverage or approach. The case studies revealed made a presentation on capacity development in an underlying weakness in the training pro- climate change at a United Nations Framework grams undertaken by GEF projects, namely a ten- Convention on Climate Change workshop. These dency to plan and execute training as a "one-shot" additional activities will contribute to the develop- solution. Extensive stakeholder consultations dur- ment of a set of tools that will enable future APRs, ing the evaluation showed the importance of pro- GEF country portfolio evaluations, and the OPS4 gression and repetition in training. Progression is to evaluate the achievements of capacity develop- needed to allow successful trainees who have made ment activities on a broader scale. use of their new knowledge and skills to undertake more advanced courses and reach higher levels of 5.2 Findings from the Terminal expertise, thereby further strengthening institu- Evaluations tional performance. As for repetition, staff turn- Forty-two terminal evaluations, of which 41 were over in many government institutions is high, so submitted in FY 2007, were reviewed. Of these, 26 there is a need to repeat training regularly. are for national projects, 10 for regional projects, and 6 for global projects. The terminal evalua- One alternative to the one-shot approach would be tion review identified planned and implemented to tap the existing training resources in a country capacity development activities described in the or region, such as universities or specialist public terminal evaluations and related project docu- or private sector training institutions. These can mentation. The review analyzed the nature and be supported to adapt their existing programs or results of interventions in relation to targets set at create new ones to address the key environment- the policy, institutional, and individual levels. related skills identified as necessary during proj- ect preparation. In some cases, it may be effective Figure 5.1 shows types of capacity development to develop new specialist training entities in a activities described in the project documents and region. One-shot training inputs by international terminal evaluations. All 42 projects reviewed consultants should be a strategy of last resort, included some kind of training activities, whether when it is evident that the required expertise is these were for local executing agencies, other 5. Evaluation of Capacity Development 31 Figure 5.1 been relevant to national development goals, Types of Capacity Development Activities in with the exception of some specific training exer- Projects cises in five projects. The efficiency of capacity development activities has usually been satis- Development of laws, policies, national strategies 57% factory in meeting output and outcome targets. However, 12 projects--or almost 30 percent of the Training executing organization sta 79% total number reviewed--suffered implementation Training local delays, while 13 were overly ambitious in their 50% communities goals and targets. Here again, the cost-effective- Training other ness of capacity development activities was dif- organizations 81% ficult to assess, mainly because budget informa- Learning by doing; on-the- 67% tion was only reported for capacity development job skills development activities when it was a specific component. Awareness-raising/ 98% education activities The effectiveness of capacity development Improved resource/enterprise 45% activities has varied. Of the 24 projects with management models capacity development activities at the systemic Improved M&E/informa- level, 11 projects were successful in improv- 76% tion systems ing policies and legal frameworks. The remain- Pilot/demonstration activities for wider replication 57% ing 13 projects were overly ambitious and made little progress or were only able to achieve their Establishment of local com- 36% munity/stakeholder groups aims partially. Five projects created new institu- Improved production or tions with varying degrees of outcomes. Concern- 31% marketing systems ing training and institutional strengthening, the Improved organizational co- 67% number of trainings conducted, both formal and ordination & communication on the job, was substantial. Terminal evaluations 0 20 40 60 80 100 reported the numbers and types of trainings held and the number of trainees. However, only five, or less than 12 percent, of the terminal evaluations organizations involved in project implementation, reported on the effectiveness and outcomes of or local communities or comprised on-the-job training activities. training for individuals and institutions. All but one Although many improvements to capacity at project included some kind of awareness-raising the individual, institutional, and systemic levels or educational activity. Thirty-two projects aimed were reported in the terminal evaluations, there to improve M&E and information systems. More are concerns about the sustainability of some than half of the projects (24) sought to enhance or of the results. A number of changes to policies, develop laws, policies, strategies, and standards. legal frameworks, and strategies seem to have The terminal evaluation review findings sup- a strong level of commitment. Where terminal ported those of the country case studies. The evaluations reported on the project implementa- results of capacity development activities have tion unit model, it resulted in little or no capacity generally been positive. Support has overall enhancement of government staff or institutions. 32 GEF Annual Performance Report 2007 The design of these projects did not include a viable 5.3 Summary project exit strategy. The review of terminal evalu- ations revealed that the majority of projects consid- The country case studies and the review of termi- ered training as a one-shot exercise. Some projects nal evaluations provide the following lessons on (seven) established long-term training arrange- capacity development activities: ments, mainly through university-level courses. z Capacity development at the systemic level The review of terminal evaluations found that there must be realistic. Overly ambitious goals to is inadequate reporting on the performance of change policies or laws may, when not attained, capacity development activities implemented by adversely affect implementation of other proj- projects. Few projects included institutional devel- ect components. opment baselines or assessments, so the intended z Institutional strengthening requires baselines use of new capacity was unclear. The sustainability and assessments to determine how new capac- of capacity improvements from training remains ity will be absorbed. low because few national or regional long-term training programs were developed. The weak z Long-term training programs and capaci- evidence base of capacity development activities ties should be based on a national or regional makes it difficult to learn lessons or track account- context rather than a dependence on one-shot ability of project investments. training. 5. Evaluation of Capacity Development 33 6. Quality of Project Monitoring Of the 41 terminal evaluation reports reviewed z the information provided by the M&E system for FY 2007, 33 (80 percent) provided sufficient was used for project management; information to allow the GEF Evaluation Office z the parties responsible for M&E activities were to rate quality of project monitoring. Of the rated properly trained to ensure that correct proce- projects, 61 percent were assessed as moderately dures are followed and quality is maintained in satisfactory or above for quality of project moni- data collection. toring. The analysis also confirmed the associa- tion between quality at entry of M&E arrange- ments and quality of project monitoring during 6.2 Overall Findings implementation. As shown in figures 6.1 and 6.2, the propor- tion of projects rated moderately satisfactory or above on quality of monitoring was 61 percent for 6.1 Rating Approach FY 2007. The difference in ratings between the Data from 190 terminal evaluation reports--42 FY 2007 and other cohorts is not statistically sig- from 2004, 41 from 2005, 66 from 2006, and 41 nificant. Within the FY 2007 cohort, 76 percent of from 2007--submitted by the GEF Agencies to the medium-size projects had moderately satisfactory Evaluation Office were analyzed. Of these reports, or above ratings, compared to only 44 percent of 141 provided sufficient information to rate quality of monitoring during project implementation. The Evaluation Office rates quality of project Figure 6.1 monitoring based on how well a project's M&E Percentage of Projects Rated Moderately system was implemented. To do so, it assesses Satisfactory or Above for Quality of M&E whether Percentage of projects 100 z an M&E system was in place and facilitated 80 78% 66% 61% timely tracking of results and progress toward 60 55% project objectives by collecting information on 40 20 chosen indicators continually throughout the 0 project implementation period; 2004 2005 2006 2007 (N = 30) (N = 32) (N = 46) (N = 33) z annual project reports were complete and accu- FY of terminal evaluation report submission rate, with well-justified ratings; 34 Figure 6.2 6.3 Factors Affecting Quality of Quality of M&E during Implementation Monitoring Highly satisfactory Beginning in FY 2006, the GEF Evaluation Office 3% started assessing quality at project entry of M&E Unsatis- factory arrangements for completed projects and ade- 9% quacy of funding for M&E activities. This analysis has been developed further and is presented in Moderately this APR for FY 2007. Forty projects were rated on unsatisfactory Satisfactory 30% 42% the quality of their M&E arrangements at entry. Of these, 27 (68 percent) were rated moderately Moder- satisfactory or above in quality of project moni- ately toring during implementation. satisfactory 15% Thirty-two projects were rated both on quality at N = 33 projects entry of M&E arrangements and on their quality of monitoring during project implementation. Of these, 24 projects were rated moderately satisfac- tory or above for quality at entry of M&E arrange- full-size projects. The reasons behind this differ- ments, and 75 percent of these (18 projects) were ence are not clear; it is also not clear whether this also assessed as being moderately satisfactory or is the beginning of a trend. Table 6.1 summarizes above for quality of project monitoring during ratings on quality of project monitoring by Agency implementation. In contrast, of the remaining and focal area. eight projects that had been rated as moderately Table 6.1 Quality of M&E during Project Implementation by GEF Agency and Focal Area Number of projects GEF Agency Focal area Rating UNDP UNEP WB BD CC IW LD MF POPs Total Highly satisfactory 0 0 1 0 1 0 0 0 0 1 Satisfactory 6 2 6 8 5 0 1 0 0 14 Moderately satisfactory 2 1 2 4 1 0 0 0 0 5 Subtotal 8 3 9 12 7 0 1 0 0 20 Moderately unsatisfactory 7 1 2 6 3 0 0 0 1 10 Unsatisfactory 0 2 1 2 0 0 0 1 0 3 Highly unsatisfactory 0 0 0 0 0 0 0 0 0 0 Subtotal 7 3 3 8 3 0 0 1 1 13 Unable to assess 3 1 4 1 3 2 0 2 0 8 Total 18 7 16 21 13 2 1 3 1 41 Note: bD = biodiversity; cc = climate change; IW = international waters; LD = land degradation; MF = multifocal; PoPs = persistent organic pol- lutants.; Wb = World bank. 6. Quality of Project Monitoring 35 unsatisfactory or below in quality at entry of M&E evaluation reports received in FY 2007, only 20 arrangements, only one was rated as moderately provided enough information to allow an assess- satisfactory or above in quality of project monitor- ment of whether sufficient financial support was ing during implementation. Thus, projects that provided for M&E activities in the project bud- have a weak M&E plan at entry are unlikely get, and only 14 provided sufficient information to be able to make sufficient corrections to for actual support during project implementation. improve their quality of project monitoring Based on the terminal evaluations that provided during implementation. sufficient information, the GEF Evaluation Office It continues to be difficult to determine the ade- found that for 75 percent of the projects, sufficient quacy of financial support for M&E activities funding had been provided for M&E activities both in the project budget and during project in the project budget and for 64 percent during implementation because a significant proportion implementation. The implications of budgetary of terminal evaluations do not provide sufficient support and adequacy of funding on quality of information in this regard. Of the 41 terminal project monitoring are not yet clear. 36 GEF Annual Performance Report 2007 7. Quality of Terminal Evaluations Terminal evaluations that provide an accurate and the 66 submitted last year (see figure 7.1). The Eval- candid assessment of project accomplishments uation Office tracks submission of terminal evalu- and shortcomings are not only essential in form- ation reports based on the expected closing dates ing the building blocks for the APR's assessment of GEF projects. Since many projects face comple- of project outcomes and sustainability but are also tion delays, this makes for a somewhat unreliable a useful learning tool for the GEF partnership. tracking indicator. A better measure would be to The Evaluation Office reviews terminal evalua- track submission based on actual completion of tions to provide verified ratings on project perfor- GEF projects. Unfortunately, such information is mance and on the quality of terminal evaluation not now recorded in the GEF database in a timely reports. So far, the ratings verified by the Office manner. Often, the operational completion of a have been based on desk review of the terminal project becomes known to the GEF only when its evaluation reports, which limits the reliability of terminal evaluation report has been submitted to these reviews. the Evaluation Office. The Office will work with the GEF Secretariat and GEF Agencies so that Field verification enhances the reliability of the timely submission of terminal evaluation reports reviews by allowing the Evaluation Office to more for completed projects can be reliably tracked. rigorously confirm the results of completed proj- ects. This year, the Office piloted a methodology for field verification of terminal evaluations for Figure 7.1 three projects from the FY 2007 cohort. Based on Number of Terminal Evaluation Reports Submitted the experience gained in this pilot, the method- for Review ology will be further refined, and, beginning next Number year, the APR will present the findings of field ver- 70 ification of terminal evaluations. 60 50 To date, 223 terminal evaluation reports have been 40 All projects submitted to the GEF Evaluation Office. This is 30 FSPs the fourth year the Office has rated the quality of 20 MSPs project terminal evaluation reports; 190 have been 10 rated thus far. In FY 2007, 41 terminal evaluation 0 2002 2003 2004 2005 2006 2007 reports were submitted by the GEF Agencies--a FY of terminal evaluation report submission substantial, and thus far, unexplained, drop from 37 Overall, the joint efforts of the Evaluation given a weight of 0.3 each, and the remainder a Office and GEF Agencies in improving the weight of 0.1 each. quality of terminal evaluation reports seem to be yielding results. Of the 41 terminal evalua- 7.2 Findings tion reports submitted this year, the quality of 39 reports (95 percent) was rated to be moderately As noted, of the 41 terminal evaluation reports satisfactory or above. submitted for review in FY 2007, the quality of 95 percent (39 projects) was rated as moder- 7.1 Rating Approach ately satisfactory or above (see table 7.1). By way of comparison, the corresponding proportion of The assessment approach adopted for the termi- reports so rated in previous years was 84 percent nal evaluation reports submitted in FY 2007 in FY 2006, 88 percent in FY 2005, and 69 percent was the same as has been followed since 2005; in FY 2004. For the FY 2007 cohort, the Evalua- those submitted in FY 2004 were assessed using tion Office ratings for quality of terminal evalu- a slightly different but comparable methodology. ation reports were similar for both medium- and The reports submitted since FY 2005 have been full-size projects: for both groups, the quality of assessed based on the following criteria: 95 percent of the reports was rated as moderately z Did the report present an assessment of rel- satisfactory or above. evant outcomes and achievement of project To better see the underlying trends in change in objectives in the context of the focal area pro- quality of terminal evaluation reports, the Evalu- gram indicators, if applicable? ation Office has, since FY 2006, compared ratings z Was the report consistent, the evidence com- for quality of terminal evaluation reports based plete and convincing, and the ratings substanti- on the year of report completion rather than the ated when used? year of report submission. However, using year of completion as a basis for comparison has its own z Did the report present a sound assessment of limitations. Notably, since all reports for some sustainability of outcomes? cohorts--especially the most recent one--have z Were the lessons and recommendations sup- not yet been submitted, ratings for as-yet-unsub- ported by the evidence presented? mitted reports are not known. z Did the report include the actual project costs Figure 7.2 presents the proportion of terminal (total and per activity) and actual cofinancing evaluation reports with quality rated as moder- used? ately satisfactory or above on the basis of year of report completion. The figure shows a clear and z Did the report include an assessment of the sustained trend of quality improvement, with quality of the project M&E system and its use an increasing percentage of terminal evaluation in project management? reports being rated as moderately satisfactory or Performance on each of these criteria is rated on above--rising from 72 percent for reports com- a six-point scale. The overall rating is a weighted pleted in or before FY 2004, the first year in which average of these ratings: the first two criteria are the Office began rating report quality. 38 GEF Annual Performance Report 2007 Table 7.1 Quality of Terminal Evaluation Reports by GEF Agency and Focal Area Number of reports GEF Agency Focal area Rating UNDP UNEP WB BD CC IW LD MF POPs Total Highly satisfactory 1 1 1 1 0 0 1 1 0 3 Satisfactory 9 4 10 16 6 0 0 1 0 23 Moderately satisfactory 7 2 4 4 6 2 0 0 1 13 Subtotal 17 7 15 21 12 2 1 2 1 39 Moderately unsatisfactory 1 0 0 0 1 0 0 0 0 1 Unsatisfactory 0 0 0 0 0 0 0 0 0 0 Highly unsatisfactory 0 0 1 0 0 0 0 1 0 1 Subtotal 1 0 1 0 1 0 0 1 0 2 Unable to assess 0 0 0 0 0 0 0 0 0 0 Total 18 7 16 21 13 2 1 3 1 41 Note: bD = biodiversity; cc = climate change; IW = international waters; LD = land degradation; MF = multifocal; PoPs = persistent organic pol- lutants; Wb = World bank. Performance by Agency APR stressed the need for these Agencies to take For the terminal evaluation reports completed in remedial measures, and the GEF Council at its or before FY 2004, the quality of a substantial per- June 2005 meeting asked UNDP and UNEP to put centage of reports from both UNEP and UNDP in place terminal evaluation review processes for was in the unsatisfactory range (50 and 40 per- GEF projects to improve the quality of their ter- cent, respectively). Consequently, the FY 2004 minal evaluations. UNDP and UNEP have accord- ingly established more rigorous quality control Figure 7.2 processes for terminal evaluation. The GEF Moni- Percentage of Terminal Evaluation Reports with toring and Evaluation Policy (GEF EO 2006), Quality Rated Moderately Satisfactory or Above which was negotiated in the second half of 2005, and the "Guidelines for GEF Agencies in Conduct- Percentage ing Terminal Evaluations" (GEF EO 2007), which 100 had been under discussion since the first half of 80 2006, enunciate GEF expectations of the GEF 60 Agencies in terms of quality control for terminal FSPs 40 evaluations and provide greater clarity on this All projects 20 MSPs issue. These concentrated efforts have led to an overall improvement in quality of terminal evalu- 0 2003 2004 2005 2006 2007 ation reports: almost all of the terminal evaluation (N = 17) (N = 46) (N = 56) (N = 41) (N = 23) reports submitted by UNDP and UNEP that were FY of terminal evaluation completion completed during FYs 2006 and 2007 have ratings in the satisfactory range (see figure 7.3, which also Note: the dotted lines imply that the cohorts for those years are not yet complete. shows improvement by focal area). 7. Quality of Terminal Evaluations 39 Figure 7.3 Figure 7.4 Percentage of Terminal Evaluation Reports with Percentage of Terminal Evaluation Reports with Quality Rated Moderately Satisfactory or Above, Quality Rated Moderately Satisfactory or Above on by GEF Agency and Focal Area Each Quality Dimension a. By Agency UNDP 60% 100 94% (N = 68) 84% 50% 80 UNEP 84% (N = 22) 71% 60 World Bank 85% 89% (N = 90) 88% 40 Reporting on outcomes Consistency & completeness All Agencies 72% Overall Assessment of M&E system 90% 20 (N = 190) 84% Assessment of sustainability Finance information Lessons 0 b. By focal area 2004 2005 2006 2007 79% (N = 22) (N = 56) (N = 41) (N = 25) Biodiversity 93% (N = 104) FY of terminal evaluation completion 88% Climate change 44% 92% (N = 44) 82% Int'l waters 89% (N = 24) 80% 83% causes of variation, and the effect of variation on All focal areas 72% project performance. (N = 190) 94% 84% 0 20 40 60 80 100 Quality of financial reporting is not only contin- Percentage of projects gent on how well terminal evaluations are con- Up to 2004 After 2004 All years ducted but also on the extent to which the finan- cial management systems of the GEF Agencies are able to provide data to the evaluators in a timely manner. Since terminal evaluations are generally Performance by Quality Dimension conducted immediately after a project is opera- Although there has been a significant improvement tionally closed, some of the financial transactions in overall quality of terminal evaluation reports, related to the project may not have been accounted improvements in performance on various qual- for by Agency financial systems. This often makes ity dimensions have been uneven (see figure 7.4). it difficult for the Agencies to provide financial While ratings have improved on dimensions such information to evaluators in a timely manner. as reporting on project outcomes, sustainability of Although this information becomes available outcomes, quality of lessons, and consistency and once a project is financially closed, by that time completeness of reporting, there has been little its terminal evaluation has generally already been improvement in the quality of financial infor- completed. Consequently, financial information is mation. About half of the terminal evaluation often not incorporated in the terminal evaluation reports received continue to provide insufficient report. information on the financial performance of com- pleted projects. This lack of information prevents There are differences among the GEF Agencies the Evaluation Office from assessing the extent to in the quality of financial information provided in which there were differences in actual funds mobi- their terminal evaluation reports. Of the reports lization and utilization vis-à-vis expectations, the submitted to the Office since FY 2005, 70 percent 40 GEF Annual Performance Report 2007 of those from the World Bank provide adequate Of the 41 terminal evaluation reports submitted financial information, compared to only 41 per- to the GEF Evaluation Office in FY 2007, outcome cent from UNEP and 37 percent from UNDP. ratings have been provided by the evaluation This finding is not surprising, in that the Bank is units of the respective Agencies for 12 projects--6 a financial institution which thus accords greater World Bank projects and 6 UNEP projects. Com- attention to financial reporting. pared to the outcome ratings given by the GEF Evaluation Office, those given by the Agency eval- The Evaluation Office will work with the GEF uation units are, on average, 0.25 points higher on Secretariat and GEF Agencies to identify ways in a 6-point scale. This difference is not statistically which financial reporting in the terminal evalua- significant. Moreover, on the binary scale, the rat- tion reports can be improved. ings accorded by the GEF Evaluation Office and Agency evaluation units were identical. The find- 7.3 Agency versus Evaluation ings for the FY 2006 cohort were similar to those Office Ratings for the FY 2007 cohort. For many projects whose terminal evaluation reports have been reviewed by the Evaluation For 13 completed projects in the FY 2007 cohort, Office, ratings on outcomes, sustainability, and ratings on risks to sustainability of outcomes M&E systems are also provided by the terminal were provided by both the GEF Evaluation Office evaluation reports and independent evaluation and the respective Agency evaluation unit. On a units of the respective GEF Agency. Although the binary scale, these ratings deviated for five proj- methodologies followed by the Agency evaluation ects. Further, the Agency evaluation units rated units may not be identical to those followed by sustainability of outcomes to be moderately likely the GEF Evaluation Office, the criteria on which or above for 7 of the 13 projects, while the GEF performance is rated are often the same. The GEF Evaluation Office accorded this rating to only 6 Evaluation Office compares its ratings with those projects. The high level of disagreement in ratings provided by Agency evaluation units and terminal on sustainability underscores the differences in evaluation reports using both an expanded scale approaches adopted by the Office and the Agency (six- or four-point) and a binary scale to deter- evaluation units. The Office will work with the mine the extent to which ratings are consistent. Agency evaluation units to facilitate convergence in the approaches used to provide ratings on this The Independent Evaluation Group of the World parameter. Bank has been providing ratings on outcomes, sustainability, M&E, and quality of terminal evalu- The Agency evaluation units rated the quality of ation reports for some World Bank GEF projects. 28 of the submitted terminal evaluation reports in The UNEP independent Evaluation Office, which the FY 2007 cohort. On a binary scale, both the provided ratings only on quality of terminal evalu- GEF Evaluation Office and the Agency units rated ation reports for FY 2006, began providing ratings 27 of these in the satisfactory range (moderately for other parameters as well during the past fis- satisfactory and above). Ratings differed for two cal year. Similarly, the UNDP Evaluation Office projects. Given the number of observations, this has begun providing ratings on quality of termi- signifies a substantial convergence in the quality of nal evaluation reports for some reports submitted terminal evaluation report ratings provided by the during FY 2007. Evaluation Office and Agency evaluation units. 7. Quality of Terminal Evaluations 41 8. Assessment of GEF Agency Climate Policies and Guidelines The GEF Evaluation Office commissioned a sur- z Reduction targets and metrics to track progress vey of the policies and guidelines that GEF Agen- z Governance structure that makes climate a pri- cies are undertaking to address the GHG emis- ority issue and integrates it into core Agency sions generated by their internal operations. The operations objective was to establish a broad picture of the manner in which the GEF Agencies are managing z Employee engagement program their GHG footprint. z Centralized GHG data management process 8.1 Assessment Approach z Public disclosure This assessment was primarily based on a review Agency policies and guidelines were assessed of publicly available information on the GHG using the above-listed criteria (see table 8.1). footprint policies and guidelines of the 10 GEF Since the GEF Secretariat and the GEF Evalua- Agencies. Additional supplementary information tion Office fall under the administrative frame- was gathered by requesting some Agencies to pro- work of the World Bank Group, they were not vide relevant documents and via phone conversa- independently reviewed for this assessment. For tions. Because several of the Agencies reviewed the most part, the GEF follows World Bank poli- are in the process of defining their approach to cies regarding GHG emissions. The only differ- addressing operational GHG emissions, some of ence is regarding travel--while the World Bank the information presented here may be outdated requires staff to use least-cost options (which or may contain gaps. often imply indirect flights), GEF staff are allowed to use direct flights to limit the GHG impact of A set of climate evaluation criteria was developed their travel, even though the financial costs may in order to review the Agencies' GHG reduction be higher. approaches. Elements of a robust climate program to reduce the environmental impact of internal Information was also gathered for each Agency operations were identified and included in these that is engaged in activities to assess and mitigate criteria. The assessment criteria are as follows: the climate impacts of their projects, investments, and loans. Although these efforts are not a focus z Agency-wide climate policy of this assessment, they certainly can provide sig- z Comprehensive energy-efficiency program nificant benefits to the climate and are therefore z Annual GHG inventory worth noting. 42 Table 8.1 Assessment of GEF Agencies' Climate Policies data management Reduction targets Energy efficiency Public disclosure Centralized GHG climate policy Agency-wide engagement Annual GHG and metrics Governing Employee inventory program strategy GEF Agency African Development bank Asian Development bank european bank for reconstruction and Development Food and Agriculture organization of the United Nations Inter-American Development bank International Fund for Agricultural Development UNDP UNeP United Nations Industrial Development organization World bank Total 7 5 8 5 8 4 1 3 8.2 Findings for assessment. Overall, the banks have made more progress in addressing the climate impacts Table 8.1 summarizes the existence of climate of their internal operations than have the other policies and guidelines for the reviewed Agencies. Agencies. A checkmark for a particular assessment criterion indicates that the Agency has begun to address Eight out of the 10 reviewed Agencies are engaged this criterion and demonstrated commitment to in activities to assess and mitigate the climate furthering work in this area. impacts of their projects, investments, and loans. The findings show that the GEF Agencies are on Neither the Inter-American Development Bank track in addressing the GHG emissions of their nor the International Fund for Agricultural Devel- internal operations, but that most are in an early opment has initiated such activities thus far. stage. Some Agencies do have commitments All Agencies need to advance their efforts to to further address their internal impacts in the address their internal climate change impacts and future. For example, Agencies under the United further develop processes and systems to track Nations umbrella will be working toward reducing and manage their internal GHG inventory data. their GHG emissions as part of a UN-wide climate- The Agencies should first focus their efforts on neutral target that they plan to meet in the future. completing a robust GHG inventory. Once they Among the Agencies reviewed, the European Bank have assessed their climate risk through such an for Reconstruction and Development, the Inter- inventory, they will be well positioned to develop a American Development Bank, and UNEP have cost-effective and disciplined approach to reduc- begun to address most of the criteria considered ing their impacts. 8. Assessment of GEF Agency Climate Policies and Initiatives 43 9. Management Action Record Management action records are used to track z N.A.--not applicable the level of adoption of GEF Council decisions z Not possible to verify yet--verification will taken on the basis of evaluation findings and have to wait until more data are available or recommendations. MARs also aim to increase proposals have been further developed the accountability of GEF management regard- ing Council decisions on M&E issues. The GEF This year's MAR presents ratings of GEF manage- Council approved the format and procedures for ment and the verification of these ratings by the the GEF MAR at its November 2005 meeting and Evaluation Office. They track management actions requested that the GEF Evaluation Office prepare on Council decisions based on 12 GEF Evaluation an updated MAR to be presented to the Council Office documents, 7 of which, listed below, were for review and follow-up on an annual basis. included in last year's MAR: z "Annual Performance Report 2004" (GEF/ME/ 9.1 Rating Approach C.25/1, May 2005) The rating categories for the progress of adop- z "Role of Local Benefits in Global Environmental tion of Council decisions were agreed upon in a Programs" (GEF/ME/C.27/4, October 2005) consultative process between the GEF Evaluation Office and the GEF Secretariat and Agencies and z "Annual Performance Report 2005" (GEF/ME/ are as follows: C.28/2/Rev.1, May 2006) z "GEF Country Portfolio Evaluation: Costa Rica" z High--fully adopted and fully incorporated (GEF/ME/C.28/5, May 2006) into policy, strategy, or operations z "Evaluation of Incremental Cost Assessment" z Substantial--largely adopted but not fully (GEF/ME/C.30/2, November 2006) incorporated into policy, strategy, or operations as yet z "Evaluation of the GEF Activity Cycle and Modalities" (GEF/ME/C.30/6, November z Medium--adopted in some operational and 2006) policy work, but not to a significant degree in key areas z "Evaluation of the GEF Support to Biosafety" (GEF/ME/C.28/Inf.1, May 2006) z Negligible--no evidence or plan for adoption, or plans and actions for adoption are in a very This year's MAR also includes the following five preliminary stage documents included for the first time: 44 z "Annual Performance Report 2006" (GEF/ related to project M&E standards and to the qual- ME/C.31/1, May 2007) ity of supervision. The Evaluation Office will carry out thematic assessments in the future to assess z "Country Portfolio Evaluation: Philippines adoption of these Council decisions. (1992­2007)" (GEF/ME/C.31/3, May 2007) z "GEF Country Portfolio Evaluation: Samoa The Office rated 18 (44 percent) of the 41 verified (1992­2007)" (GEF/ME/C.31/5, May 2007) decisions as having been adopted by management at high or substantial levels. This represents an z "Joint Evaluation of the Small Grants Pro- improvement from last year when the percentage gramme--ExecutiveVersion"(GEF/ME/C.32/2, of decisions so rated was 33. October 2007) The Evaluation Office and GEF management z "GEF Annual Report on Impact 2007--Execu- agreed on the rating on progress of adoption for tive Version" (GEF/ME/C.32/4, October 2007) 51 percent of the Council decisions (21 of 41); for 46 percent of the decisions (19 of 41), the Evalua- 9.2 Findings tion Office gave a lower rating on adoption prog- So far, 12 Council decisions have been fully ress than did management. As shown in table 9.1, adopted and thus graduated from the MAR. Six of most disagreement between management and these were adopted during FY 2006, all pertaining Evaluation Office ratings are for decisions where to the "Evaluation of the Experience of Executing management has rated the accomplished adop- tion to be at a high level. Many of the lower rat- Agencies under Expanded Opportunities in the ings given by the Evaluation Office reflect the fact GEF" (GEF/ME/C.30/4, November 2006). The that even though the GEF has taken measures to other six, based on recommendations from several address these decisions, these measures cannot be evaluations, were fully adopted during FY 2007. considered to demonstrate a high level of achieve- The current MAR tracks management actions on ment yet. They also reflect the fact that some pro- 46 Council decisions. The GEF Evaluation Office posals to the Council are yet to be approved by it. was able to verify the adoption of 41 of these. The If and when the Council does approve these pro- decisions that could not yet be verified are mainly posals, substantial adoption may have occurred. Table 9.1 Ratings of GEF Progress in Adopting Council Decisions GEF Evaluation Office rating Sum of Not possible management Management rating High Substantial Medium Negligible to verify yet ratings High 5 8 8 0 3 24 Substantial 1 4 3 0 1 9 Medium 0 0 12 0 0 12 Negligible 0 0 0 0 1 1 Not possible to verify yet 0 0 0 0 0 0 Sum of Office ratings 6 12 23 0 5 46 Note: Highlighted fields show agreement between the ratings of management and the GeF evaluation office; fields to the right of the diagonal represent higher ratings by management than by the evaluation office (except in the case of not possible to verify yet). 9. Management Action Records 45 Thirty Council decisions from seven GEF Evalua- z The portal does not include information on tion Office documents were included in both last whether (and when) a proposal has been sub- year's and this year's MARs. The Office was able mitted to the GEF Secretariat and is awaiting to follow the progress of adoption of 27 of these approval. decisions; 3 were rated as not possible to verify z It is not clear what status should be used if the yet. Of these 27, 52 percent (14) showed progress project identification form (PIF)/project is not in the level of adoption, for 33 percent there was approved (rejected, deferred, or so on). no progress, and for 15 percent initial gains had been lost (see figure 9.1). Eight out of the nine Council decisions that showed no change in progress of adoption from Noticeable progress over last year is related to last year's MAR were rated by the GEF Evaluation Council decisions requesting increased trans- Office in both years as having a medium level of parency in the GEF project approval process adoption, but this does not necessarily mean that through an improved management information the GEF has not addressed these issues in the past system. In this regard, the Secretariat has recently year. These include five Council recommenda- launched a new Project Management Information tions deriving from the biosafety evaluation for System which includes a country portal to provide which the GEF has prepared the "Program Docu- information related to GEF projects at the country ment for GEF Support to Biosafety during GEF-4," level. While the Evaluation Office acknowledges which was presented to the Council in April 2008. this progress, it notes that several issues remain Although GEF management rates the adoption to be addressed before adoption of these decisions level of these five decisions as high, the Office rat- can be rated as high. These include the following: ing of medium will be changed if and when the z There is some uncertainty as to the accuracy Council approves this document. of the information included in the system (for As in last year's MAR, the Evaluation Office main- example, updating the status of closed projects). tains that the Council decision to integrate local benefits in a more systematic way at all stages of the Figure 9.1 GEF project cycle has not yet been fully adopted. Change in Ratings on Adoption of Council GEF management rates adoption of this decision Decisions since APR 2006 Assessment as high, but the Office concludes that, while the measures listed by GEF management are encour- Ratings aging, the adoption rate for this decision is still decreased medium. This assessment is based on the fact that 15% the Office believes that the GEF Secretariat cannot verify the quality of this aspect in project design Ratings No change increased and implementation because it still has no system 33% 52% in place to involve specialist social development expertise in project review processes. Relatedly, adoption of the Council decision requesting that UNEP and UNDP involve social and institutional N = 27 decisions expertise in their project supervision was rated as negligible by GEF management. 46 GEF Annual Performance Report 2007 Ratings for the progress of adoption of 4 out The GEF Evaluation Office is broadly in agree- of 27 Council decisions (15 percent) have been ment with GEF management ratings on the level downgraded by the GEF Evaluation Office this of adoption of Council decisions made in 2007. year. The first of these decisions requests the GEF While full adoption of some Council decisions to ensure adequate involvement of expertise on is expected to be achieved in the short term, social and institutional issues at all levels of the others--such as those related to improving portfolio. Downgrading of the adoption level of project M&E ("GEF Annual Report on Impact this decision is based on the fact that the GEF has 2007"), or improving supervision during project yet to implement a system to address this issue. implementation ("Annual Performance Report 2006")--are likely to take more time to be fully A second downgraded decision relates to the rec- adopted. ommendation that further consideration be given to ways to enhance the contribution of GEF Scien- The Small Grants Programme management has tific and Technical Advisory Panel reviews in the started to address various Council decisions project review guidelines. The Office rating was from the SGP joint evaluation through the GEF downgraded from substantial to medium because SGP Steering Committee. However, because annex 11 of the "GEF Operations Manual" draft the country coordinators are not represented (available on February 8, 2008) does not mention on the Steering Committee, there is a risk that, the panel's role in project M&E. inadvertently, new proposals will not consider The third decision is related to the Council request country operational issues and perspectives suf- for new operational guidelines to implement the ficiently. The inclusion of some senior country recommendations of the incremental cost assess- coordinators on the Steering Committee would ment evaluation. Although the GEF Secretariat allow for the country program perspective to be is in the process of developing a new operations heard in the discussions that lead to SGP pro- manual, both GEF management and the GEF Eval- posals addressing GEF Council decisions. uation Office downgraded the rating of the adop- Although all involved parties agree that this moni- tion of this decision to medium in accordance with toring of the levels of adoption of Council deci- the established rating criteria. A substantial level of sions is important and relevant, the Evaluation adoption is expected to be achieved once the man- Office believes that this process is becoming more ual is approved by the GEF Council. cumbersome each year. Options such as setting The last downgraded decision has to do with the fact timelines for graduation of Council decisions, that, although representing a reduction in require- breaking down GEF Evaluation Office ratings by ments compared to the previous cycle, there is still actor (rather than the rating being an average of room for further simplification of the PIF to keep compliance by all actors related to a decision), and its criteria simple and unambiguous. In this regard, presenting clusters of Council decisions by issue UNDP has reported that it has to answer an increas- (rather than by document as is currently done) ing number of questions, from both GEF Secretariat will be discussed in the future in order to make and Council members, in relation to the PIF. this tool more user friendly. 9. Management Action Records 47 10. Performance Matrix The performance matrix provides a summary of The figures listed in this year's matrix, however, the performance of the GEF Implementing Agen- may not be four-year averages as data are not cies and GEF Secretariat on a variety of parameters available for all these parameters for the pre- (see table 10.1). Although several of these param- ceding three years. eters are assessed by the Evaluation Office on an z The figures reported for performance on qual- annual basis, to mitigate against fluctuations in ity of project terminal evaluation are two- performance ratings due to differences in project year running averages, as meaningful changes mix or other idiosyncratic factors, the values pre- can be attained in the short run. sented in the matrix are, depending on the param- eter, running averages of two to four years. Of the z Changes in performance are also likely to be 13 parameters included in the performance matrix, gradual for a second set of parameters: quality information has been provided for 9. Note, how- of supervision and adaptive management, ever, that the information provided for parameter realism of risk assessment and risk manage- 13, improvement in performance, addresses only ment, and quality of project M&E arrange- 1 of that parameter's 12 dimensions. In the future, ments. Moreover, assessment of performance as data for more years become available, it will be on these parameters requires intensive thematic possible to track improvement in performance on appraisals. For the sake of efficiency, the Evalu- a greater number of dimensions. See annex D for ation Office will take up such appraisals as part methodological notes for the performance matrix of the APR after a two- to three-year interval. data reported for this APR. Several items cannot yet be reported on. 10.1 Rating Approach z For assessing performance on project prepa- ration elapsed time, information will be ana- Reporting methodology varies by parameter: lyzed from the GEF database. So far, the qual- z Four performance parameters--project out- ity of information available from the database comes, implementation completion delays, is not sufficiently reliable to facilitate such an materialization of cofinancing, and quality analysis. When the recent efforts undertaken of monitoring and evaluation during project by the GEF Secretariat to improve the quality implementation--are being reported as four- of information in the GEF database fructify, year running averages, as improvements on such analyses will be taken up and findings pre- these parameters are expected to be gradual. sented in the matrix. 48 Table 10.1 GEF Agencies and Entities Performance Matrix World GEF Overall GEF Parameter UNDP UNEP Bank Secretariat Performance Results 1. Project outcomes: percentage of completed projects with 78 67 87 __ 81 outcomes rated moderately satisfactory or above Processes affecting results 2. Quality of supervision and adaptive management: percent- 88 36 87 __ 81 age rated moderately satisfactory or above Efficiency 3. Project preparation elapsed time: average number of __ __ __ __ __ months required to prepare projects 4. Implementation completion delays: average delay in 21 10 12 __ 15 completion of projects in months 5. Materialization of cofinancing a. reported materialization of cofinancing per dollar of 1.3 1.5 3.9 3.2 3.2 approved GeF financing b. reported materialization of cofinancing as percentage of 84 94 93 91 91 promised cofinancing Quality of M&E 6. Independence of Agency central evaluation units __ __ __ __ __ 7. Independence of terminal evaluations or independent __ __ __ __ __ review of terminal evaluations 8. realism of risk assessment (robustness of project-at-risk 75 29 80 71 71 systems): percentage of projects rated moderately satisfactory or above in candor and realism in supervision reporting 9. robustness of program result indicators and tracking tools __ __ __ __ 10. Quality assurance of project M&e arrangements at entry 68 40 50 58 58 11. Quality of project M&e during implementation 52 73 82 __ 71 12. Quality of project terminal evaluation 91 81 88 __ 88 Quality of Learning 13. Improvement in performance a. Project outcomes __ __ __ __ __ b. Improvement in quality of terminal evaluations: on a 4 4 3 __ 3 scale of 1 (low performance) to 4 (high performance) For three parameters, the Evaluation Office is in of the process followed for conducting terminal the process of developing reporting approaches. evaluations when terminal evaluation verifica- z Independence of terminal evaluations. The tions are carried out, and it will be based on the Office will assess the extent to which sys- interviews of relevant staff and consultants of tems in the partner Agencies are conducive the partner Agencies. The following criteria to to unbiased and candid terminal evaluations. assess independence of terminal evaluations This will be appraised through the assessment were presented and discussed with the GEF 10. Performance Matrix 49 Agencies and Secretariat at the March 2008 costs, and cofinancing by component) and interagency meeting: the consistency of the text of the report and the ratings ­ Extent to which the drafting of the terms of reference was independent from project z The independence of evaluation units of implementation and included the correct GEF partner Agencies. The GEF Evaluation technical input; particularly important in Office has begun a consultation process with this regard will be to consider the degree the evaluation units of the GEF Agencies so as of separation of the drafting of the terms of to define an appropriate way forward on assess- reference from those responsible for project ing their independence. An initial proposal is implementation to look to devise a way to assess the extent to ­ Extent to which the recruitment of the eval- which the existing institutional arrangements uator was independent from direct project pose risks to the independence of the evaluation implementation; particularly important in units. To assess this, the Office would draw on this regard will be to consider the degree of self-reports and peer reviews carried out in the separation from those responsible for direct context of the Evaluation Cooperation Group project implementation of the Banks and the United Nations Evaluation ­ Extent to which the Agency recruited the Group. The charter and mandate of the various appropriate evaluator for the project evaluation units will also provide evidence of their degree of independence. ­ Extent to which the evaluator had adequate resources (budget and time) to carry out the z Quality of learning. The matrix will report on evaluation; in consultation with the GEF quality of learning through an assessment of Agencies, the Office will develop bench- improvement demonstrated by the GEF Agen- marks to assess budgets and time for termi- cies and entities on the other 12 parameters. nal evaluations for different types of projects This section of the matrix will be accompanied considering factors such as number of coun- by a narrative that explains the areas in which tries, project sites, diversity of project com- learning has taken place and will identify the ponents, and overall project cost specific changes or factors that have contrib- ­ Extent to which the M&E system provides uted to improved performance. This year, rat- access to timely and reliable information ings on performance improvement have been provided for changes in quality of terminal ­ Extent to which there were any undue man- evaluation reports. agement pressures on the evaluators regard- ing the evaluation process (site selection, The assessment of robustness of program result selection of informants, confidentiality dur- indicators and tracking tools will not be reported ing interviews, information disclosure, rat- on in this APR. Given the highly specialized and ings, and so on) technical nature of this assessment, the Evalua- ­ Extent to which the evaluation was sub- tion Office will look at this parameter as part of jected to an independent review process the program studies conducted for the OPS4 in that assessed completeness (including the FY 2008, enlisting the assistance of appropriate coverage of all relevant outcomes, risks to technical experts. The Office has withdrawn two sustainability, M&E system, actual project review criteria--project sustainability ratings and 50 GEF Annual Performance Report 2007 the MAR--which had been proposed earlier, as changes in performance on these parameters the former is not a performance criterion but a are tracked, the findings on these parameters characteristic of a project, and objective compara- will also become important. tive assessment is difficult for the latter. Based on the review of terminal evaluation reports submitted to the Evaluation Office from FY 2005 10.2 Findings to FY 2007, the Office rated outcome achieve- ment (parameter 1) in 81 percent of the projects As noted, information has been provided on 9 of to be moderately satisfactory and above. Although the 13 parameters included in the performance the rated projects do not fall under the provisions matrix. However, only two findings, which are of the fourth GEF replenishment, the percent- more important than the others, are highlighted age is higher than the 75 percent target specified here. for GEF-4. Among the Implementing Agencies, z The first, which pertains to project outcomes, the percentage of World Bank projects with rat- is a key measure of the performance of the GEF ings in the satisfactory range has been significantly Agencies in implementing projects that gen- higher than the target (87 percent). erate global environmental benefits. The GEF For parameter 13, improvement in performance, Council, as noted elsewhere, has established a the Evaluation Office this year assessed perfor- 75 percent satisfactory projects benchmark so mance improvement in quality of terminal evalu- the actual performance against this measure ations. As noted in conclusion 5 of this APR, there may be noted. has been a substantial improvement in perfor- z The second finding pertains to improvement mance on this dimension. In particular, UNEP and in performance on quality of terminal evalu- UNDP demonstrated considerable improvement, ation reports. The performance ratings on and the quality of a high percentage of terminal other parameters pertain to the baseline figures evaluation reports submitted by the World Bank for that parameter. In subsequent years when continues to be in the satisfactory range. 10. Performance Matrix 51 Annex A. Terminal Evaluation Report Review Guidelines The assessments in the terminal evaluation a. Relevance. Were project outcomes consistent reviews will be based largely on the information with the focal area/operational program strate- presented in the terminal evaluation report. If gies and country priorities? Explain. insufficient information is presented in a terminal b. Effectiveness. Are project outcomes commen- evaluation report to assess a specific issue such surate with the expected outcomes (as described as, for example, quality of the project's monitor- in the project document) and the problems the ing and evaluation system or a specific aspect of project was intended to address (that is, the orig- sustainability, then the preparer of the terminal inal or modified project objectives3)? evaluation reviews will briefly indicate so in that c. Efficiency. Include an assessment of outcomes section and elaborate more if appropriate in the and impacts in relation to inputs, costs, and section of the review that addresses quality of implementation times based on the following report. If the review's preparer possesses other questions: Was the project cost-effective? How first-hand information such as, for example, from does the project's cost/time versus outcomes a field visit to the project, and this information is equation compare to that of similar projects? relevant to the terminal evaluation reviews, then Was the project implementation delayed due it should be included in the reviews only under to any bureaucratic, administrative, or politi- the heading "Additional independent informa- cal problems and did that affect cost-effective- tion available to the reviewer." The preparer of the ness? terminal evaluation review will take into account all the independent relevant information when An overall rating will be provided according to verifying ratings. the achievement and shortcomings in the three criteria ranging from highly satisfactory, satisfac- A.1 Criteria for Outcome Ratings tory, moderately satisfactory, moderately unsatis- Based on the information provided in the terminal factory, unsatisfactory, highly unsatisfactory, and evaluation report, the terminal evaluation review unable to assess. will make an assessment of the extent to which the The reviewer of the terminal evaluation will pro- project's major relevant objectives were achieved or vide a rating under each of the three criteria (rel- are expected to be achieved, relevance of the proj- evance, effectiveness, and efficiency) as follows: ect results, and the project's cost-effectiveness.1 The ratings on the outcomes of the project will be z Highly satisfactory. The project had no short- based on performance on the following criteria:2 comings. 53 z Satisfactory. The project had minor shortcom- Impacts will be understood to include positive ings. and negative, primary and secondary long-term z Moderately satisfactory. The project had effects produced by a development intervention. moderate shortcomings. They could be produced directly or indirectly and could be intended or unintended. The terminal z Moderately unsatisfactory. The project had evaluation review's preparer will take note of any significant shortcomings. mention of impacts, especially global environ- z Unsatisfactory. The project had major short- mental benefits, in the terminal evaluation report, comings. including the likelihood that the project outcomes z Highly unsatisfactory. The project had severe will contribute to their achievement. Negative shortcomings. impacts mentioned in the terminal evaluation report should be noted and recorded in section 2 z Unable to assess. The reviewer was unable to of the terminal evaluation review template in the assess outcomes on this dimension. subsection on "Issues that require follow-up." The calculation of the overall outcomes score of Although project impacts will be described, they projects will consider all three criteria, of which will not be rated. relevance and effectiveness are critical. The over- all score will be determined satisfying two different A.3 Criteria for Sustainability conditions: (1) the overall score on project outcomes Ratings will have a constraint that it cannot be higher than Sustainability will be understood as the likelihood the lower score among the critical criteria of rel- of continuation of project benefits after completion evance and effectiveness; and (2) the overall score of project implementation (GEF 2000). To assess cannot be higher than the average score of all three sustainability, the terminal evaluation reviewer criteria calculated using the following formula: will identify and assess the key risks that could Outcomes = (a + b + c) ÷ 3 undermine continuation of benefits at the time of the evaluation. Some of these risks might include In case the average score calculated using this the absence of or inadequate financial resources, formula is lower than or equal to the lower score enabling legal framework, commitment from key among the critical criteria of relevance and effec- stakeholders, and enabling economy. The follow- tiveness, then the average score will be the overall ing four types of risk factors will be assessed by the score. In case the average score is higher than the terminal evaluation reviewer to rate the likelihood lower score among the critical criteria of relevance of sustainability of project outcomes: financial, and effectiveness, then the lower score among the sociopolitical, institutional framework and gover- critical criteria will be the overall rating. If the rat- nance, and environmental. ing for the critical dimensions is unable to assess, then the overall rating for outcomes will also be The following questions provide guidance to unable to assess. assess if the factors are met: z Financial resources. What is the likelihood A.2 Impacts that financial resources will be available to con- Has the project achieved impacts, or is it likely tinue the activities that result in the continua- that outcomes will lead to the expected impacts? tion of benefits (income-generating activities) 54 GEF Annual Performance Report 2007 and trends that may indicate that it is likely z Unlikely. There are severe risks affecting that that in future there will be adequate financial criterion of sustainability. resources for sustaining project outcomes? z Unable to assess. The reviewer was unable to z Sociopolitical. Are there any social or political assess risk on this dimension. risks that can undermine the longevity of proj- z Not applicable. This dimension is not appli- ect outcomes? What is the risk that the level of cable to the project. stakeholder ownership is insufficient to allow for project outcomes/benefits to be sustained? A number rating of 1­4 will be provided in each Do the various key stakeholders see it as in category according to the achievement and short- their interest that the project benefits continue comings, with likely = 4, moderately likely = 3, to flow? Is there sufficient public/stakeholder moderately unlikely = 2, unlikely = 1, and not awareness in support of the long-term objec- applicable = 0. A rating of unable to assess will be tives of the project? used if the reviewer is unable to assess any aspect of sustainability. In such instances, it may not be z Institutional framework and governance. Do possible to assess the overall sustainability. the legal frameworks, policies, and governance structures and processes pose any threat to the All the risk dimensions of sustainability are criti- continuation of project benefits? While assess- cal. Therefore, the overall rating will not be higher ing this parameter, consider if the required sys- than the rating of the lowest rated dimension. tems for accountability and transparency, and For example, if the project has an unlikely rating the required technical know-how, are in place. in either of the dimensions, then its overall rat- ing cannot be higher than unlikely, regardless of z Environmental. Are there any environmen- whether higher ratings in other dimensions of tal risks that can undermine the future flow of sustainability produce a higher average. project environmental benefits? The terminal evaluation should assess whether certain activi- ties in the project area will pose a threat to the A.4 Criteria for Assessment of sustainability of project outcomes. For example, Quality of Project M&E Systems construction of a dam in a protected area could GEF projects are required to develop an M&E inundate a sizable area and thereby neutralize the plan by the time of work program inclusion, to biodiversity-related gains made by the project. appropriately budget the M&E plan, and to fully carry out the M&E plan during implementation. The reviewer will provide a rating under each of Project managers are also expected to use the the four criteria (financial resources, sociopoliti- information generated by the M&E system dur- cal, institutional, and environmental) as follows: ing project implementation to improve and adapt z Likely. There are no risks affecting that crite- the project to changing situations. Given the long- rion of sustainability. term nature of many GEF projects, projects are also encouraged to include long-term monitoring z Moderately likely. There are moderate risks plans that measure results (such as environmental that affect that criterion of sustainability. results) after project completion. Terminal evalu- z Moderately unlikely. There are significant ation reviews will include an assessment of the risks that affect that criterion of sustainability. achievement and shortcomings of M&E systems. Annex A. Terminal Evaluation Report Review Guidelines 55 a. M&E design. Projects should have a sound ­ Was sufficient funding provided for M&E M&E plan to monitor results and track prog- in the budget included in the project ress in achieving project objectives. An M&E document? plan should include a baseline (including data, ­ Was sufficient and timely funding provided methodology, and so on), SMART (specific, for M&E during project implementation? measurable, achievable, realistic, and timely) ­ Can the project M&E system be considered indicators and data analysis systems, and evalu- a good practice? ation studies at specific times to assess results. The time frame for various M&E activities A number rating of 1­6 will be provided for and standards for outputs should have been each criterion according to the achievement and specified. Questions to guide this assessment shortcomings, with highly satisfactory = 6, satis- include: In retrospect, was the M&E plan at factory = 5, moderately satisfactory = 4, moder- entry practicable and sufficient (sufficient and ately unsatisfactory = 3, unsatisfactory = 2, highly practical indicators identified; timely baseline; unsatisfactory = 1, and unable to assess = no rat- targets created; effective use of data collection; ing. The reviewer of the terminal evaluation will analysis systems including studies and reports; provide a rating under each of the three criteria practical organization and logistics in terms of (M&E design, M&E plan implementation, and what, who, and when for M&E activities)? M&E properly budgeted and funded) as follows: b. M&E plan implementation. The M&E system was in place and allowed the timely tracking of z Highly satisfactory. There were no shortcom- results and progress toward project objectives ings in that criterion of the project M&E system. throughout the project. Annual project reports z Satisfactory. There were minor shortcomings were complete, accurate, and with well-justified in that criterion of the project M&E system. ratings. The information provided by the M&E system was used to improve and adapt project z Moderately satisfactory. There were moder- performance. An M&E system should be in place ate shortcomings in that criterion of the project with proper training for parties responsible for M&E system. M&E activities to ensure that data will continue z Moderately unsatisfactory. There were sig- to be collected and used after project closure. nificant shortcomings in that criterion of the Questions to guide this assessment include: Did project M&E system. the project M&E system operate throughout the z Unsatisfactory. There were major shortcomings project? How was M&E information used dur- in that criterion of the project M&E system. ing the project? Did it allow for tracking of prog- ress toward project objectives? Did the project z Highly unsatisfactory. There was no project provide proper training for parties responsible M&E system. for M&E activities to ensure data will continue to be collected and used after project closure? The rating for M&E during implementation will be the overall rating of the M&E system: c. Other questions. These include questions on funding and whether the M&E system was a Rating on the Quality of the Project good practice. M&E System = b 56 GEF Annual Performance Report 2007 A.5 Criteria for Assessment of z Satisfactory. There were minor shortcomings Quality of Terminal Evaluation in the terminal evaluation report on this crite- rion. Reports z Moderately satisfactory. There were moder- The ratings on quality of terminal evaluation ate shortcomings in the terminal evaluation reports will be assessed using the following report on this criterion. criteria: z Moderately unsatisfactory. There were signif- a. The report presents an assessment of all rel- icant shortcomings in the terminal evaluation evant outcomes and achievement of project report on this criterion. objectives in the context of the focal area pro- z Unsatisfactory. There were major shortcom- gram indicators if applicable. ings in the terminal evaluation report on this b. The report was consistent, the evidence pre- criterion. sented was complete and convincing, and rat- z Highly unsatisfactory. There were severe ings were well substantiated. shortcomings in the terminal evaluation report c. The report presented a sound assessment of on this criterion. sustainability of outcomes. The first two criteria (of all relevant outcomes and d. The lessons and recommendations are sup- achievement of project objectives and report con- ported by the evidence presented and are rel- sistency and substantiation of claims with proper evant to the portfolio and future projects. evidence) are more important and have therefore e. The report included the actual project costs been assigned a greater weight. The quality of the (totals, per activity, and per source) and actual terminal evaluation reports will be calculated by cofinancing used. the following formula: f. The report included an assessment of the qual- Quality of the Terminal Evaluation Report = ity of the M&E plan at entry, the M&E system 0.3 × (a + b) + 0.1 × (c + d + e + f) used during implementation, and whether the The total number will be rounded and con- information generated by the M&E system was verted to the scale of highly satisfactory to highly used for project management. unsatisfactory. A number rating of 1­6 will be provided for each criterion, with highly satisfactory = 6, satisfactory A.6 Assessment of Processes = 5, moderately satisfactory = 4, moderately unsat- Affecting Attainment of Project isfactory = 3, unsatisfactory = 2, highly unsatisfac- Outcomes and Sustainability tory = 1, and unable to assess = no rating. This section of the terminal evaluation review Each criterion to assess the quality of the terminal will summarize the factors or processes related to evaluation report will be rated as follows: implementation delays and cofinancing that may have affected attainment of project results. This z Highly satisfactory. There were no shortcom- section will summarize the description in the ter- ings in the terminal evaluation report on this minal evaluation on key causal linkages of these criterion. factors: Annex A. Terminal Evaluation Report Review Guidelines 57 z Cofinancing and project outcomes and sus- opment results to which a project or program is tainability. If there was a difference in the level expected to contribute (OECD DAC 2002). of expected cofinancing and actual cofinancing, 2. Outcomes are the likely or achieved short-term what were the reasons for it? To what extent did and medium-term effects of an intervention's materialization of cofinancing affect project outputs. Outputs are the products, capital goods, and services that result from a development outcomes and/or sustainability? What were the intervention; these may also include changes causal linkages of these effects? resulting from the intervention that are relevant to the achievement of outcomes (OECD DAC z Delays and project outcomes and sustain- 2002). For the GEF, environmental outcomes are ability. If there were delays, what were the the main focus. reasons for them? To what extent did the delay 3. The GEF Secretariat and IAs are currently seek- affect project outcomes and/or sustainability? ing to better align the focal area program indica- What were the causal linkages of these effects? tors and tracking tools with focal area strategic priorities and project objectives. This will enable the aggregation of outcomes and impacts for each Notes focal area to annually measure progress toward 1. Objectives are the intended physical, financial, targets in the program indicators and strategic institutional, social, environmental, or other devel- priorities. 58 GEF Annual Performance Report 2007 Annex B. Terminal Evaluations Reviewed during FY 2007 Approved GEF Focal Project GEF GEF grant ID # Project name Region area size Agency (million $) 2 Philippines - Samar Island biodiversity Project: conservation Asia bD FSP UNDP 5.76 and Sustainable Use of the biodiversity of a Forested Protected Area 4 Vietnam - Hon Mun Marine Protected Area Pilot Project Asia bD MSP Wb 0.97 30 Nepal - Upper Mustang biodiversity Project Asia bD MSP UNDP 0.75 110 regional - central Asia transboundary biodiversity Project ecA bD FSP Wb 10.15 128 brazil - energy efficiency Project LAc cc FSP Wb 15.00 245 Lesotho - conserving Mountain biodiversity in Southern Africa bD FSP UNDP 2.51 Lesotho 264 Syria - Supply-Side efficiency and energy conservation and Asia cc FSP UNDP 4.61 Planning 284 romania - capacity building for GHG emission reduction ecA cc FSP UNDP 2.27 through energy efficiency 410 Global - conservation of Wetland and coastal ecosystems in the Global bD FSP UNDP 13.42 Mediterranean region 445 china - barrier removal for the Widespread commercialization Asia cc FSP UNDP 9.86 of energy-efficient cFc-Free refrigerators in china 505 Pakistan - Mountain Areas conservancy Project Asia bD FSP UNDP 10.10 573 Kenya - removal of barriers to energy conservation and energy Africa cc FSP UNDP 3.19 efficiency in Small and Medium Scale enterprises 595 Global - Solar Development Group Global cc FSP Wb 10.00 597 regional - building Partnerships for the environmental Protec- Asia IW FSP UNDP 16.22 tion and Management of the east Asian Seas 618 bangladesh - Aquatic biodiversity conservation Asia bD FSP Wb 5.00 620 bolivia - Sustainability of the National System of Protected LAc bD FSP Wb 15.00 Areas 643 Mexico - renewable energy for Agriculture LAc cc FSP Wb 8.90 661 Suriname - conservation of Globally Significant Forest ecosys- LAc bD FSP UNDP 9.54 tems in Suriname's Guayana Shield 766 Uruguay - Landfill Methane recovery Demonstration Project LAc cc MSP Wb 0.98 770 Global - Millennium ecosystem Assessment Global bD FSP UNeP 6.96 59 Approved GEF Focal Project GEF GEF grant ID # Project name Region area size Agency (million $) 801 Slovak republic - central european Grasslands - conservation ecA bD MSP Wb 0.75 and Sustainable Use 814 regional - coral reef Monitoring Network in Member States of Africa bD MSP Wb 0.74 the Indian ocean commission, within the Global reef Monitor- ing Network 836 Global - critical ecosystems Partnership Fund Global bD FSP Wb 25.00 847 Nicaragua - renewable energy and Forest conservation: Sus- LAc MF MSP Wb 0.75 tainable Harvest and Processing of coffee and Allspice 913 Philippines - biodiversity conservation and Management of the Asia bD MSP UNDP 0.74 bohol Islands Marine triangle 914 Latvia - economic and cost-effective Use of Wood Waste for ecA cc MSP UNDP 0.75 Municipal Heating Systems 981 regional - community-based Management of on-farm Plant Africa bD MSP UNeP 0.75 Genetic resources in Arid and Semi-arid Areas of Sub-Saharan Africa 984 Mongolia - Dynamics of biodiversity Loss and Permafrost Melt Asia MF MSP Wb 0.83 in Lake Hovsgol National Park 1058 regional - Pacific Islands renewable energy Programme Asia cc MSP UNDP 0.70 1261 Papua New Guinea - community-based coastal and Marine Asia bD FSP UNDP 3.55 conservation in the Milne bay Province 1311 South Africa - Pilot Production and commercial Dissemination Africa cc MSP UNDP 0.80 of Solar cookers 1325 regional - Institutional Strengthening and resource Mobiliza- Africa MF MSP Wb 1.00 tion for Mainstreaming Integrated Land and Water Manage- ment Approaches into Development Programs in Africa 1384 Global - biodiversity Indicators for National Use Global bD MSP UNeP 0.82 1410 regional - biodiversity conservation and Integration of tra- LAc bD MSP UNeP 0.75 ditional Knowledge on Medicinal Plants in National Primary Health care Policy in central America and caribbean 1477 Vietnam - conservation of Pu Luong-cuc Phuong Limestone Asia bD MSP Wb 0.75 Landscape 1646 russian Federation - cost effective energy efficiency Measures ecA cc MSP UNDP 1.00 in the russian educational Sector 1851 regional - Protection of the North West Sahara Aquifer System Africa IW MSP UNeP 0.60 and related Humid Zones and ecosystems 2067 Global - Fostering Active and effective civil Society Participa- Global PoP MSP UNeP 1.00 tion in Preparations for Implementation of the Stockholm convention 2194 russian Federation - Developing the Legal and regulatory ecA cc MSP Wb 0.73 Framework for Wind Power in russia 2344 regional - Desert Margins Programme tranche 2 - Africa Africa bD FSP UNeP 5.62 3036 regional - Supporting capacity building for the third National Asia LD MSP UNDP 0.51 reporting to crIc-5/coP-8 (Asia Pacific) Note: bD = biodiversity; cc = climate change; ecA = europe and central Asia; IW = international waters; LAc = Latin America and the caribbean; MF = multifocal; PoP = persistent organic pollutant; Wb = World bank. 60 GEF Annual Performance Report 2007 Annex C. Greenhouse Gas Footprint Policies and Guidelines Assessment C.1 Criteria ­ Regularly scheduled energy audits to find inefficiencies and areas for improvement The criteria for greenhouse gas footprint policies with focus on largest facilities and guidelines assessment include the following. ­ Building benchmarking to compare effi- Agency-Wide Climate Policy ciency of buildings to others within the orga- nization and available industry standards z Does the Agency's climate policy contain and communicate the Agency's mission and core ­ Performance incentives for energy and facility values with respect to climate change? managers to meet energy efficiency goals z Does it contain and communicate commit- ­ Capability to assess cost-effectiveness of ments to control and improve its carbon impact energy efficiency investments and environmental performance with respect ­ A commitment to invest in all no-cost or to significant climate aspects of the organiza- low-cost efficiency measures tion's products, activities, and/or services? z Does it contain and communicate how the Annual GHG Inventory Agency plans to address climate impacts of z Is the Agency's GHG inventory consistent with internal operations? the GHG Protocol developed by the World z Does the Agency have a policy as to whether to Resources Institute and the World Business evaluate its lending portfolio and investments Council for Sustainable Development? based on climate criteria? z Does it include an inventory management plan to make inventory processes repeatable and Comprehensive Energy Efficiency Program transparent? z Does the Agency have an energy efficiency pro- z Does it include at a minimum, all scope 1, scope gram that includes improvements in buildings, 2, business travel emissions (scope 3), and emis- equipment, and lighting? The program should sions from the hosting of conferences (scope 3)? include all forms of energy use at the Agency, including purchased electricity, on-site fuel z Does it include all employees and on-site con- use, and use of vehicles. sultants? z Does the Agency have the following elements z Does it include headquarters, regional offices, of an energy efficiency program: and field offices? 61 z Are results made public through an annual them as to what they can do at work and at report, Web site, or other venue? home to reduce their impact? z Is the inventory signed off on by an Agency z Are tools available to collect and share best senior executive? practices in areas such as business travel, equip- ment management and support, food services, Reduction Targets and Metrics to Track mail and transportation services, and supplier Progress management? z Has the Agency set an absolute GHG reduction z Are incentive programs in place to encourage target and energy efficiency targets over a 5- to behaviors that reduce GHG impacts at home 10-year time period? and in the office? z Has the Agency set normalized targets for energy z Does the Agency provide employee commuter efficiency and GHG reductions such as emis- benefits? sions/square foot and emissions per employee? Centralized GHG Data Management Governance structure That Makes climate Process a Priority Issue and Integrates It into Core Agency Operations z Are corporate-level processes in place for col- lecting and calculating GHG information to z Is the Agency's board of directors actively ensure consistency and completeness? engaged in climate change policy and has it assigned oversight responsibility to a board z Are tools available for analyzing GHG data to member, board committee, or full board? create prioritized lists of actionable reduction initiatives? z Does the Agency president assume a leader- ship role in articulating and executing climate Public Disclosure change policy? z Does the Agency publicly disclose material risks z Are top executives and/or executive com- and opportunities posed by climate change? mittees assigned to manage climate change z Do the communications to the public offer response strategies? comprehensive, transparent presentation of z Does the climate management team review response measures? each year the Agency's climate change program to ensure its continuing suitability, adequacy, C.2 Findings on Individual and effectiveness? Agencies ­ Is executive officers' performance evaluation The following table summarizes the status of linked to attainment of climate strategy goals GHG footprint policies and guidelines for each of and GHG targets? the 10 GEF Agencies. A checkmark for a particu- lar assessment criterion indicates that the Agency Employee Engagement Program has begun to address this criterion and has dem- z Are there training programs in place to educate onstrated commitment to furthering work in this staff about climate change and to empower area. 62 GEF Annual Performance Report 2007 Table C.1 Status of Agency GHG Policies and Guidelines African Development Bank (AfDB) Agency-wide climate policy Formulation of a comprehensive corporate environmental strategy that includes GHG impacts is currently under development. energy efficiency program As part of AFDb's corporate environmental strategy, a comprehensive energy efficiency program was developed. Annual GHG inventory A comprehensive GHG inventory was recently completed. reduction targets and No information could be found about AfDb reduction targets or metrics for climate metrics change. Governing strategy AfDb is creating an environmental Working Group that will consist of several experts and board members who will review and assess implementation of the environmental strategy as practiced throughout AfDb. GHG emissions are a component of the strategy. employee engagement No information could be found about AfDb's employee engagement program. program centralized GHG data man- No information could be found about a centralized GHG data management process. agement process Public disclosure the AfDb Web site does not contain climate program information, and this information was not found through any other public source. Initiatives to assess and AFDb is currently developing a strategy aimed at integrating climate risk management and mitigate impacts of loans, adaptation into its operations. the goal of the strategy is to safeguard and enhance the projects, and investments effectiveness of AfDb investments in a changing climate, and to improve its support to its regional member countries in the area of climate risk management and adaptation. Information sources www.afdb.org; AfDb corporate environmental Initiatives Analysis and Strategy Framework document, January 2008; bank Group climate risk Management and Adaptation Strategy­ Approach Paper, February 2008 Asian Development Bank (ADB) Agency-wide climate policy ADb's climate change brochure, reAcH program, and carbon Market Initiative detail its climate change initiatives. ADb has an energy policy created in 1995 that includes its cli- mate change program, but not a separate policy for climate change initiatives. ADb has an energy strategy under development, which will guide its future operations in the energy sector. the issues the strategy will address are energy security, global warming/climate change, sector policy reform and governance, and energy efficiency. ADb has a robust environmental policy, but it provides little information on how the bank addresses climate change within that policy. the environmental policy deals with five major challenges at ADb; within the third challenge, the Need to Maintain Global and regional Life-Support Systems, there is some mention of the need to address climate change. energy efficiency program ADb promotes energy efficiency in how its buildings are cooled, heated, ventilated, and lit. Its headquarters building in Manila is fitted with the most efficient technologies available today (www.adb.org/documents/brochures/ADb-energy-efficiency.pdf). Annual GHG inventory the AfDb paper on its environmental strategy references the results of an ADb GHG inventory. reduction targets and No information could be found about ADb reduction targets and metrics for climate metrics change. Governing strategy ADb's president, Haruhiko Kuroda, is actively engaged in the climate change issue. His remarks are contained in the foreword of the ADb's climate change brochure developed in 2007. employee engagement there is no information available about ADb's employee engagement efforts. program Annex C. Greenhouse Gas Footprint Policies and Initiatives Assessment 63 Asian Development Bank (ADB) centralized GHG data man- No information could be found about the ADb GHG data management process. agement process Public disclosure the ADb Web site links to all its initiatives and programs to mitigate climate change. Initiatives to assess and In 2005, ADb established the energy efficiency Initiative with the aim of compiling and mitigate impacts of loans, analyzing existing knowledge and experience on energy efficiency policies and formulat- projects, and investments ing a clean energy investment strategy with an indicative lending target of $1 billion over the next few years. the carbon Market Initiative, approved by the ADb board in November 2006, provides financial and technical support to developers and sponsors of projects with GHG mitigation benefits that can qualiFY as an eligible clean development mechanism project under the Kyoto Protocol. Information sources www.adb.org/Documents/Policies/environment/env020100.aspwww.adb.org/Documents/ books/sustainability-report/default.asp; www.adb.org/rSDD/default.asp; www.adb.org/ clean-energy/default.asp; www.adb.org/Documents/brochures/climate-change/ climate-change-brochure.pdf; AfDb corporate environmental Initiatives Analysis and Strategy Framework, January 2008 European Bank for Reconstruction and Development (EBRD) Agency-wide climate policy ebrD does not have a specific climate policy, but it does include climate initiatives in its 2005 energy policy. there is also a brief mention of following Intergovernmental Panel on climate change and United Nations Framework convention on climate change mecha- nisms for climate change in its environmental policy. energy efficiency program ebrD has a number of energy efficiency initiatives under way. It completed an indepen- dent study on its water and energy consumption in 2005. ebrD headquarters is currently being refurbished with installation of new lighting and an energy management system to reduce its impact on the environment. the ebrD energy policy guidelines are published online and include several initiatives for energy efficiency (www.mtvsz.hu/dynamic/ ebrdeen.pdf). Annual GHG inventory ebrD conducted a preliminary GHG assessment in 2005. It has published guidelines for others to follow its methodology (www.ebrd.com/enviro/tools/ghg.pdf). reduction targets and the energy policy of 2005 set renewable energy targets for ebrD including a target to metrics meet at least 10 percent of its electricity needs by renewable sources. other reduction goals and achievements include (www.ebrd.com/pubs/general/sus05.pdf): · 2005: reduced paper use to 78.3 tons from 96 tons · recycles light bulbs, paper, glass, printer and toner cartridges · employ a refrigerant recovery unit to collect/reuse refrigerants · Use video conferencing for meetings when possible Governing strategy ebrD created a multidepartmental working group to consider a range of energy and environment issues, specifically concerning energy efficiency, increasing recycling, and reducing environmental impact. the working group brings together the administration department, environmental department, communications, and staff council members. employee engagement Many organizations throughout erbD support energy efficiency and climate initiatives. program ebrD encourages staff at its London headquarters to reduce car use by providing interest- free loans for season tickets for public transport. It also provides secure storage space for people who cycle to work (www.ebrd.com/pubs/general/sus05.pdf). centralized GHG data man- No information could be found about an ebrD centralized GHG data management process. agement process Public disclosure ebrD discloses its climate change information on its Web site and in its annual sustain- ability reports. It includes information on internal and external climate projects. ebrD embraces the Global reporting Initiative reporting elements; these provide guidelines for sustainability reporting initiatives for voluntary use by organizations to report their activi- ties concerning economic, social, and environmental actions from their activities, products, and services (www.ebrd.com/pubs/general/sus05.pdf). 64 GEF Annual Performance Report 2007 European Bank for Reconstruction and Development (EBRD) Initiatives to assess and the energy policy of 2005 sets a target that 50 percent of all lending in the power sector mitigate impacts of loans, should be earmarked for increasing energy efficiency. ebrD analyzes its increase in carbon projects, and investments emissions from projects financed. this information was reported in the ebrD Sustainability report 2005. the direct investments signed by ebrD in 2005 are estimated to lead to a net increase in GHG emissions of 5.6 Mtpa. Information sources www.ebrd.com/enviro/index.htm Food and Agricultural Organization of the United Nations (FAO) Agency-wide climate policy FAo does not have a stand-alone climate policy, but many of its programs address climate change. It contributes to the climate change debate by assessing the available scientific evidence, participating in observing and monitoring systems, collecting unique global data sets, and providing a neutral forum for negotiations and technical discussions on climate change and agriculture. energy efficiency program No information could be found about FAo's energy efficiency activities. Annual GHG inventory No information could be found about FAo's annual GHG inventory. reduction targets and No information could be found about FAo's reduction targets and metrics for climate metrics change. Governing strategy It appears that the issue of climate change is integrated into core operations, but it is unclear if there is oversight from a board or top management. employee engagement No information could be found about FAo's employee engagement program. program centralized GHG data man- No information could be found about FAo's GHG data management. agement process Public disclosure FAo has a good deal of climate information readily available on its Web site, but this does not cover any activities it is implementing to reduce the impact of its internal operations. Initiatives to assess and In 2001, an integrated climate change program based on current activities was approved. mitigate impacts of loans, this includes the promotion of practices for climate change mitigation, the adaptation projects, and investments of agricultural systems to climate change, the reduction of emissions from the agricul- tural sector as far as it is carefully considered within the major objective of ensuring food security, the development of practices aimed at increasing the resilience of agricultural production systems to the vagaries of weather and climate change, national and regional observing systems, as well as data and information collection and dissemination. through the newly established environment, climate change and bioenergy Division, FAo will strengthen its information and advocacy activities to encourage investment in prepared- ness; promotion of adaptation of agricultural, forestry and fishery systems to climate change to ensure that efforts to protect the land and aquatic base for sustainable agricul- tural production are integral elements of climate change adaptation measures. Information and analysis will be critical and so will capacity building. Information sources www.fao.org/clim/; www.fao.org/clim/role_en.htm Inter-American Development Bank (IDB) Agency-wide climate policy IDb has a carbon Neutral Initiative and is investing in projects in Latin America and the carib- bean, which is based on four lines of action: calculating emissions, reducing emissions, offset- ting emissions, and setting an example in Latin America and the caribbean. IDb headquarters has been carbon neutral since 2007; in 2008, IDb country offices will also be carbon neutral. the IDb commitment to going carbon neutral supports the objectives of the environment and Safeguards compliance Policy and the Sustainable energy and climate change Initiative, both of which encourage the reduction and control of GHG emissions. As part of its com- mitment, IDb has become a partner of the U.S. environmental Protection Agency (ePA) for three programs: ePA climate Leaders, ePA Green Power Partnership, and ePA energy Star. Full details of IDb's partnerships with ePA will be available online in April 2008 (http://idbdocs. iadb.org/wsdocs/getdocument.aspx?docnum=932547). Annex C. Greenhouse Gas Footprint Policies and Initiatives Assessment 65 Inter-American Development Bank (IDB) energy efficiency program As an ePA energy Star partner, IDb has made a voluntary commitment to measure, track, and benchmark energy performance. In 2007, the IDb main headquarters buildings received an energy Star rating of 84 out of 100. In addition, IDb has committed to develop- ing and implementing a plan to improve energy performance. to that end, it is continually identifying ways to reduce energy and gas usage and conducting research into improved energy performance and ozone protection. In 2007, this included the purchase of renew- able energy certificates equivalent to 100 percent of its energy consumption in its four U.S. headquarters buildings, a move recognized by IDb acceptance into the ePA Green Power Partnership. In 2008, IDb intends to begin energy audit of its buildings (www.iadb.org/sds/ reports/sustainability2006/commitments.cfm?language=en). Annual GHG inventory IDb conducts an annual GHG inventory for its headquarters emissions. this included all direct emissions associated with the operation of IDb offices, such as heating and electric- ity, chiller emissions; and indirect emissions from IDb travel (air travel and hotel nights). In 2007, these emissions were approximately 22,000 tco2eq. In addition, in 2007 IDb began to conduct individual inventories for each of its 26 country offices and 2 nonregional offices. the calculation of these is in its final stage. results are expected in April 2008 and will be made public. In conducting its GHG inventory, IDb follows the ePA's climate Leaders Greenhouse Gas Inventory Protocol, Design Principals, and the GHG Protocol. IDb also cal- culates emissions associated with its annual meeting of the board of Governors. Full details of the bank's GHG inventory will be available online in April 2008 (http://idbdocs.iadb.org/ wsdocs/getdocument.aspx?docnum=932547). IDb is currently developing an inventory management plan, which sets out a clear approach to managing and monitoring GHG data. A decision has not yet been taken as to whether this will be a public document. reduction targets and to meet a goal of carbon neutrality, IDb annually offsets carbon and other emissions that can- metrics not be reduced by greening and efficiency measures and programs. It purchases verified emis- sion reductions (Vers) from leading carbon brokerage firms, ensuring credibility and delivery guarantee; these are sourced from offset projects in Latin America and the caribbean, focusing on energy efficiency, renewable energy, and fuel switch projects which provide additional environmental and social benefits for local communities. In 2007, this included projects in Peru, Guatemala, and colombia. In 2008, carbon offsets will be purchased to ensure that coun- try offices are also carbon neutral. IDb will continue to purchase renewable energy certificates equivalent to 100 percent of its energy consumption in its four headquarters buildings in the United States, and seek energy efficiency measures both at headquarters and in its country offices. As part of its commitment to ePA climate leaders, IDb is in the process of discussing and setting a GHG reduction goal to be achieved over the next 5 to 10 years. Governing strategy In 2006 IDb President Moreno tasked the IDb's sustainability and facilities functions with forming a Greening the bank task Force to address issues of internal sustainability includ- ing energy efficiency. In discussing this issue in 2007, he underlined: "We are committed to doing all we can to address the issue of climate change in the region and provide sustain- able development opportunities to local communities. We look forward to applying the [carbon neutral] concept to other bank activities." employee engagement In 2008, IDb will be focusing on improving information available internally and externally program on the way in which it tackles the challenges of climate change internally. this includes the launch of a new Sustainability Portal, as well as planned workshops to engage staff in the carbon neutral initiative. the current focus is on discussing the initiative with country office staff who are providing inventory data. centralized GHG data man- IDb is documenting its inventory process through an inventory management plan. It is agement process unclear if the data collection process has been centralized. Public disclosure IDb has numerous links to carbon footprint information on its Web site. A new Sustainabil- ity Portal will be launched in April 2008 with updated information on IDb's greening and carbon neutral initiatives, as well as the 2007 Sustainability review. Information sources www.iadb.org/sds/carbon_neutral/; www.iadb.org/sds/reports/sustainability2006/highlights. cfm?language=english 66 GEF Annual Performance Report 2007 International Fund for Agricultural Development (IFAD) Agency-wide climate policy Many of IFAD's programs take climate initiatives into consideration. IFAD understands the issues surrounding climate change and its mission. For IFAD, climate change has a special significance. Its mission is to enable poor rural people to overcome poverty. Agriculture is the main source of livelihood for most poor rural people, and it is also the human activity most directly affected by climate change. In 1988, an interdepartmental Group on climate was established by FAo to closely monitor the state of climate change science, develop the position of the organization, and participate in the international discussions, particularly after the adoption of the United Nations Framework convention on climate change (1992) and the Kyoto Protocol (1997). energy efficiency program No evidence of an internal energy efficiency program was found for IFAD. Annual GHG inventory No information could be found about IFAD's annual GHG inventory program. reduction targets and No information could be found about IFAD's reduction targets or metrics for climate metrics change. Governing strategy Lennart båge, president of IFAD, published, "climate change: A Growing challenge for Development and Poverty reduction," indicating that top management is actively engaged on climate initiatives, but there is no evidence the issue is integrated into core operations. employee engagement there is no public information available about what IFAD does to engage employees on program the issue of climate change. centralized GHG data man- No information could be found about IFAD's GHG data management process. agement process Public disclosure IFAD has a good deal of climate change science information readily available on its Web site, although it does not disclose any internal climate activity information. Information sources www.ifad.org/climate/index.htm United Nations Development Program (UNDP) Agency-wide climate policy UNDP does not have a climate policy to direct climate activities. even without a policy, it has implemented many climate initiatives (www.energyandenvironment.undp.org/undp/ indexAction.cfm?module=Library&action=GetFile&DocumentAttachmentID=1019). energy efficiency program No information available on an UNDP internal energy efficiency program. Annual GHG inventory UNDP's 2006 GHG inventory estimate data was included in a recently completed United Nations GHG inventory. UNDP has an ongoing process to improve its inventory data and eventually complete a comprehensive GHG inventory. reduction targets and UNDP supports the UN Secretariat's goal for all UN agencies to be climate neutral, although metrics the Agency has not yet committed to this goal. Governing strategy UNDP is working across the world to help developing countries build the capacity needed both to adapt to the impacts of climate change and dramatically expand the reach of affordable, improved energy services to the 2 billion people who currently go without. "As global citizens we cannot afford to wait to address these urgent issues," noted UNDP Administrator Mark Malloch brown. two bureau heads are in charge of the Agency's green- ing initiatives including its climate-neutral effort. employee engagement No information was found on internal UNDP employee engagement programs for climate program change. centralized GHG data man- No information was found on a UNDP centralized GHG data management process. agement process Public disclosure UNDP has information on its climate change initiatives on its Web site. Little information was found on its internal climate activities. Annex C. Greenhouse Gas Footprint Policies and Initiatives Assessment 67 United Nations Development Program (UNDP) Initiatives to assess and to mobilize the benefits of carbon financing for the developing world, UNDP established mitigate impacts of loans, the MDG carbon Facility, an innovative mechanism for the development and commercial- projects, and investments ization of emission reduction projects. the core objectives of the facility are (1) broadening access to carbon finance by enabling a wider range of developing countries to participate, particularly those countries that are presently underrepresented; and (2) promoting emis- sion reduction projects that contribute to the UN Millennium Development Goals, yielding additional sustainable development and poverty reduction benefits. the UNDP Initiative for Sustainable energy provides a policy framework for UNDP support to countries seeking to integrate sustainable energy activities into their national plans for social, economic, and environmental development. Its focus is on encouraging countries to establish appropriate legal, regulatory, and institutional frameworks for increasing access to energy services and financing mechanisms. In 2000, 66 percent of UNDP's country offices reported working with governments on sustainable energy projects involving energy efficiency, renewable energy, and energy planning. Information sources http://hdr.undp.org/en/media/hdr_20072008_summary_english.pdf; www.undp.org/climat- echange/; phone conversation with UNDP environment staffer United Nations Environment Program (UNEP) Agency-wide climate policy A UNeP climate neutral strategy has been developed with the specific objective of making UNeP climate neutral as of January 1, 2008. the strategy has been endorsed by UNeP's senior management team in September 2007. energy efficiency program No information is available about UNeP's energy efficiency program. Annual GHG inventory An initial inventory of UNeP GHG emissions was prepared in 2007 highlighting air travel and electricity consumption as two major sources of GHG. A Preliminary comprehensive GHG emissions Inventory is currently under preparation. It will be finalized by the end of March 2008. reduction targets and UNeP has a goal of becoming climate neutral as of January 1, 2008. A detailed Action Plan metrics for climate Neutral UNeP is in development based on the analysis of energy use patterns and emissions inventories from the Preliminary comprehensive GHG emissions Inventory. Governing strategy the UNeP/GeF document "Investing in our Planet" notes the following: "on climate change, UNeP's work builds on its expertise in assessment, standard setting, methodology development, and demonstration. UNeP draws upon its own programs, its risoe centre and networks such as the International energy Agency, Global Network for energy and Sustainable Development, reN21, reeeP, the IPcc, Global change observing Systems, WMo and the International Solar energy Society. Much of UNeP's work takes on a market sector approach responding to environmental drivers. UNeP works with National cleaner Production centers to address energy and chemical use. UNeP has developed disaster preparedness methods through its APeLL program, important to climate change adapta- tion vulnerability through its assessment work." (http://dgef.unep.org/intranet/strategy/ UNeP-2pager-18April07.doc). employee engagement A training program on green practices in the office, including activities to reduce GHG program emissions, is being prepared in partnership with the United Nations office in Nairobi. the program will also cover green practices that staff members may implement in their private life outside the office. centralized GHG data man- A new position has been created within UNeP to coordinate and monitor the implementa- agement process tion of the climate Neutral Strategy. the functions will include (1) design and implement systems to collect and manage GHG data for UNeP headquarters operations and provide advice to UNeP outpost offices on how to design and implement such systems; (2) coordi- nate with UNeP outpost offices on the collection of GHG data; and (3) develop the UNeP global GHG inventory using the World resources Institute-World business council for Sus- tainable Development GHG reporting protocol and calculate the UNeP climate footprint, consistent with the UN systemwide climate Neutral Strategy. 68 GEF Annual Performance Report 2007 United Nations Environment Program (UNEP) Public disclosure UNeP has information on its external climate change initiatives on its Web site. Little infor- mation is available on its internal climate activities. Initiatives to assess and the UNeP risoe center on energy, climate and Sustainable Development supports UNeP mitigate impacts of loans, in its aim to incorporate environmental aspects into energy planning and policy world- projects, and investments wide, with special emphasis on developing countries. the center supports research by local institutions, coordinates projects, disseminates information, and conducts a full in-house research program in close collaboration with other institutions in Denmark and interna- tionally. It pursues its objectives through initiation of and participation in UNeP-sponsored energy-environment projects at the national or regional level, research and methodologi- cal development on energy-environment issues and climate change mitigation, and technical support to programs on energy and climate change. Information sources www.unep.org/DePI/; www.climateneutral.unep.org/cnn_contentdetail. aspx?m=131&amid=194; www.uneprisoe.org/index.htm United Nations Industrial Development Organization (UNIDO) Agency-wide climate policy UNIDo has recently set up a basic in-house coordination mechanism, its climate team, and is beginning to put the building blocks in place. the climate team will shortly be submit- ting an initial strategy to UNIDo's executive board on how to implement its commitment to climate neutrality. energy efficiency program No information is available on UNIDo's internal energy efficiency program. Annual GHG inventory UNIDo completed a rough estimate of its 2006 GHG inventory as part of the UN envi- ronment Management Group's effort to complete a GHG inventory estimate of all UN agencies. reduction targets and UNIDo is committed to the UN Secretariat's goal for all UN agencies to reach climate neu- metrics trality although a date for meeting this goal has not been set. Governing strategy reducing the impacts of climate change has been integrated into the operations of UNIDo, and UNIDo recently created a climate team that will soon present a climate-neutral strat- egy for approval to the executive board. employee engagement employees at many divisions within UNIDo work on energy efficiency. there is no indica- program tion of an employee engagement program educating staff about how they can reduce their climate impacts. centralized GHG data man- No information could be found about UNIDo's GHG data management process. agement process Public disclosure UNIDo discloses information about its external climate change initiatives to the public. No information is available on its internal climate activities. Initiatives to assess and UNIDo's industrial energy efficiency program helps industries and support institutions mitigate impacts of loans, adopt a systems approach to lower energy consumption, which can lead to huge effi- projects, and investments ciency gains. UNIDo's energy efficiency capacity building and advisory services include (1) provision of train-the-trainer interventions with the objective of transferring to national specialists the knowledge and skills required to enable them improve the efficiencies of industrial energy systems; (2) training on financing aspects of energy-efficiency invest- ments to enable trained experts to commercially market their services to industrial clients; (3) creating awareness of the importance of life-cycle costing in energy-efficiency analy- sis, demonstrating that over their working life motor and steam boiler systems consume energy and fuel costing far more than the initial capital investment for the system; and (4) assistance in promoting the necessary financing and investment for energy efficiency at national and industry levels. Information sources www.unido.org/doc/51262; www.unido.org/en/doc/71847# Annex C. Greenhouse Gas Footprint Policies and Initiatives Assessment 69 World Bank Agency-wide climate policy the World bank does not have a climate change policy at this time, although it does have a stated commitment to address climate change issues both as a development agency (to promote low-carbon development) and an organizational entity (to reduce and offset its footprint). A comprehensive strategic framework for World bank engagement, including support for developing countries' efforts to adapt to climate change and achieve low-car- bon energy growth, is in the consultation phase within the World bank and is expected to be presented at the 2008 annual meetings for endorsement by the board of Governors. energy efficiency program the World bank has worked to increase its building energy efficiency through many measures including building operational efficiency, equipment changes, and engineering attention. Its decentralization goal has decreased the number of staff working in its Wash- ington offices. this, combined with technology to allow long-distance teleconferencing, has reduced unnecessary business travel. Annual GHG inventory the World bank completed a 2006 GHG inventory of its U.S. operations for ePA's climate Leaders program and is beginning to prepare a worldwide 2007 GHG inventory which will become an annual exercise. reduction targets and As part of its ePA climate Leaders participation, the World bank pledges to reduce total U.S. metrics GHG emissions by 7 percent from 2006 to 2011. the International Finance corporation, a World bank agency, is not part of this commitment. In FYs 2005, 2006, and 2007, the World bank met a carbon neutral target. this includes emissions associated with operating its Washington-area buildings, business flights tracked from Washington, as well as transpor- tation and hotel stays for spring and annual meetings for delegates and staff. the World bank has purchased renewable energy certificates to cover 100 percent of the electricity used in its Washington offices. these certificates pay for the incremental cost of using clean energy instead of traditional fossil fuel energy. to offset the remaining carbon emissions in FY 2007, it purchased 59,400 metric tons of carbon dioxide equivalents in verified emis- sions reductions from projects in Moldova and costa rica. Governing strategy No information was available on the World bank's integrating climate change into its core operations, although mainstreaming environment is a key plank of the bank's environment Strategy published in 2002. employee engagement the World bank Group subsidies to employees for the use of public transportation. program centralized GHG data man- No information could be found about the World bank's GHG data management process. agement process Public disclosure the World bank discloses information about its external climate change program to the public. No information on its internal climate activities is available. Initiatives to assess and the World bank administers nine carbon funds and facilities through the carbon Finance mitigate impacts of loans, Unit, leveraging public and private investment for projects to reduce GHG emissions. In projects, and investments FY 2006, 27 emission reductions Purchase Agreements were signed for a total volume of 148 million tons of carbon dioxide equivalent, including two trifluoromethare (HFc-23) projects totaling 129 million tons of carbon dioxide equivalent. In the last seven years, the bank's carbon finance activities have grown from the pioneering $180-million Prototype carbon Fund to funds and facilities that represent almost $1.6 billion, with more than 62 private sector companies and 13 governments as participants. the World bank is also work- ing on a policy for assessing the climate risk of their loans, projects, and investments. Information sources http://go.worldbank.org/75DLocZFG0; http://go.worldbank.org/W13H8ZXSD1; http:// go.worldbank.org/VretHAGHe0; discussions with World bank Group environmental staff 70 GEF Annual Performance Report 2007 Annex D. Methodological Notes on the Performance Matrix This annex briefly describes the considerations months required to prepare projects. The data on taken into account for each of the performance this parameter will be provided by the Agencies matrix's 13 parameters. and the GEF Secretariat database. These figures will be updated biennially. This year, no figures are D.1 Project Outcomes provided for this parameter yet. Although the figures on project outcomes are projected to be four-year moving averages based D.4 Project Implementation on the terminal evaluation reports submitted in Completion Delays the preceding years, including the fiscal year for The information presented in the terminal evalu- which the APR is being presented, the figures pre- ation reports is the primary source for this param- sented in this year's APR are based on the termi- eter. The figures for implementation completion nal evaluation reports submitted during FYs 2005, delays are projected to be four-year averages and 2006, and 2007. This is because project outcomes are based on the information provided in the ter- were not assessed for the FY 2004 cohort. The minal evaluation reports. However, in this year's aggregate figures are weighted averages, with each APR, the figures provided are based on a three- project considered to have equal weight. year period (terminal evaluation reports submit- ted during FYs 2005, 2006, and 2007). D.2 Quality of Supervision and Adaptive Management D.5 Materialization of Cofinancing The figures presented on quality of supervision The figures for materialization of cofinancing per- and adaptive management are based on the find- tain to projects whose terminal evaluation reports ings of the pilot assessment of project supervi- were submitted to the Office during FYs 2004, 2005, sion presented in the FY 2006 APR. The projects 2006, and 2007. The analysis is based on the infor- considered for this assessment were under imple- mation provided by the Agencies in the terminal mentation during FY 2005 and/or FY 2006. The evaluation reports or through other communica- figures will be updated every two to three years tions. These figures have not been verified. through follow-up assessments. D.6 Independence of Agency D.3 Project Preparation Elapsed Time Evaluation Units The figures presented for project preparation This year, the GEF Evaluation Office has begun a elapsed time will indicate average number of consultation process with the evaluation units of 71 the GEF Agencies to define an appropriate way z Extent to which there was any undue pressure forward in assessing their independence. Broadly, from management on the evaluators regarding the assessment provided on this parameter will be the evaluation process (for example, in terms based on self-reporting by the Agencies and peer of site selection, selection of informants, con- reviews carried out in the context of the Evalu- fidentiality during interviews, information dis- ation Cooperation Group of the Banks and the closure, and ratings) United Nations Evaluation Group. The charter z Extent to which the evaluation was subjected to and mandate of the various evaluation units will an independent review process also provide evidence of their degree of indepen- dence. No ratings are provided on this parameter Ratings will be provided on this parameter when this year. field verifications become a regular feature of the APR. D.7 Independence of Terminal Evaluations D.8 Realism of Risk Assessment Independence of terminal evaluations will be The figures for realism of risk assessment are based appraised through the assessment of the process on the findings of the pilot assessment of project followed for conducting terminal evaluations supervision for candor and realism of supervision through field verification and will be based on reporting presented in the 2006 APR. The proj- interviews with relevant staff and consultants of ects considered for this assessment were under the partner Agencies. This will allow the Office implementation during FY 2005 and/or FY 2006. to assess the extent to which systems in the part- The figures will be updated after a two- to three- ner Agencies are conducive to unbiased and can- year period through follow-up assessments. did terminal evaluations. The following dimen- sions will be assessed to provide ratings on this D.9 Robustness of Program Result parameter: Indicators and Tracking Tools z Extent to which the drafting of the terms of ref- The assessment of robustness of program result erence is independent of the project manage- indicators and tracking tools will remain unre- ment team ported in the 2007 APR. Given the highly special- ized and technical nature of this assessment, the z Extent to which the recruitment of the evalu- Evaluation Office will take up this exercise as part ator was independent of the project manage- of the program studies for the OPS4 in FY 2008, ment team enlisting the assistance of the appropriate techni- z Extent to which the Agency recruited the cal experts. appropriate evaluator for the project z Extent to which the evaluator had adequate D.10 Quality Assurance of Project resources (budget and time) to carry out the M&E Arrangements at Entry evaluation The assessment of quality assurance of project z Extent to which the M&E system provides M&E arrangements at entry was carried out in the access to timely and reliable information 2005 APR and was based on a review of the M&E 72 GEF Annual Performance Report 2007 plans of the project appraisal documents that were D.13 Quality of Learning: endorsed by the GEF Chief Executive Officer in Improvement in Performance that fiscal year. The Evaluation Office will update the ratings on this parameter every two to three The performance matrix will also report on an years through follow-up assessments. assessment of the improvement demonstrated by GEF Agencies and entities on the other 12 parameters included in the performance matrix. D.11 Quality of Project M&E during This section of the matrix will be accompanied Implementation by a narrative that explains the areas in which Figures on quality of project M&E during imple- learning has taken place and will identify the spe- mentation are based on review of the terminal cific changes or factors that have contributed to evaluation reports submitted to the Evaluation improved performance. Ratings on improvement Office. The figures need to be four-year run- in performance on individual parameters will be ning averages of the percentage of projects rated provided using the following scale: moderately satisfactory or above in M&E during implementation. However, for this APR, the fig- z 4--significantly improved or maintained excel- ures reported in the matrix are a weighted average, lent performance with each project having an equal weight, of the z 3--marginally improved or maintained good data from the review of the reports submitted dur- performance ing FYs 2006 and 2007. In due course, as data for z 2--marginally deteriorated or maintained subsequent cohorts become available, the figures mediocre performance presented will shift to a four-year running average. z 1--significantly deteriorated or maintained D.12 Quality of Project Terminal poor performance Evaluation In this performance matrix, only the changes in Figures on quality of terminal evaluation reports quality of terminal evaluations (parameter 12) are based on the ratings provided by the Evaluation have been reported on. A two-year running aver- Office after their review. For this parameter, two- age of quality ratings for the terminal evaluation year running averages are used, with each project reports submitted during FYs 2006 and 2007 has having an equal weight. The figures presented in been compared with the baseline for the FYs 2004 the matrix pertain to FYs 2006 and 2007. and 2005 cohort. Annex D. Methodological Notes on the Performance Matrix 73 Annex E. Management Response This annex presents the management response to The Secretariat notes with some concern the the executive version of this report, which was pre- finding of only 75 percent materialization of the sented to the GEF Council in April 2008 as GEF/ promised cofinance for the cohort of projects that ME/C.33/2. Minor editorial corrections have been submitted terminal evaluation reports in FY 2007. made. Note that the paragraph numbers cited refer Although the degree to which cofinancing actually to the executive version of the report. materialized has varied from year to year, and the average of the last four years is still above 90 per- The Secretariat welcomes the GEF Annual Perfor- cent, the Secretariat finds that this issue deserves mance Report 2007, prepared by the GEF Evalu- special attention in the 2008 APR. ation Office. The 2007 APR provides an assess- ment of project outcome ratings, materialization The Secretariat also notes the finding of an aver- of cofinancing, quality of monitoring and evalua- age delay of project completion of about one year. tion, and delays in project completion. This year's The delays seem to be decreasing, and the Secre- APR also for the first time presents a performance tariat would like to see this possible trend further matrix summarizing the performance of GEF analyzed in the 2008 APR. Agencies on a number of parameters, some of which are still being developed. This year's APR The Secretariat welcomes the finding on the posi- furthermore focuses on capacity development tive results of GEF capacity development activities projects and on the GHG emissions of the Agen- and notes the apparent weaknesses in relation to cies' internal operations. sustainability. The work undertaken by the Evalu- ation Office on evaluation of capacity develop- The Secretariat welcomes the report's conclusion ment is very useful to determine the best way to that completed GEF projects remain on target to improve the delivery and sustainability of capac- achieve 75 percent satisfactory outcomes, a target ity building across projects. The capacity devel- that was agreed upon in the GEF-4 replenishment opment strategy approved by the GEF Council is agreement. The report notes that the number of now being implemented, and the comments and terminal evaluation reports submitted in FY 2007 suggestions received in this year's APR will con- (42) was lower than the number of reports (66) sub- tribute to making it more effective. mitted in FY 2006, and the Evaluation Office states its intent to investigate this difference. The Secre- The Secretariat also acknowledges the comments tariat notes that the number of reports submitted in on capacity development given in chapter [5] of FY 2005 (41) was almost the same as for FY 2007. the APR and notes the Evaluation Office's views 74 that the GEF portfolios in the Philippines and that there is little or no formal role to play in this Vietnam are generally positive and in some areas area for the Secretariat. significant. It is further noted that the effectiveness has been varied in terms of results. As the evalua- The Secretariat welcomes the progress toward a tion reports highlight, there are limited incentives performance matrix that summarizes the APR find- in the government to retain and reward motivated ings in a set of quantitative indicators. The Secre- staff, and capacity development is also closely tariat notes that this tool is still under development linked to institutional structures and procedures as and that the scores indicated for the Secretariat so well as progression and repetition in training rather far are set as the aggregated scores of the Agencies. than "one-shot" solutions. The Secretariat appreci- Additional comments to this year's APR have been ates the insights and recommendations received. received from UNEP, UNDP, and the World Bank. These will be taken into account in future in order UNEP notes the following: to strive to improve the delivery, effectiveness, and efficiency of capacity development. UNEP agrees with the Evaluation Office that some parameters in the Agency Performance Matrix can only The Secretariat welcomes the significant be realistically assessed at 2- or 3-year intervals. This improvement of the quality of the terminal evalu- is the case for parameters 2, 8, and 10 in table [10.1]. ation reports and notes the observed correlation However, given that this APR presents ratings based between the quality at entry of M&E arrange- on the 2005 APR findings and on the quality of project supervision exercise performed in 2006, UNEP wishes ments and the actual quality of monitoring. to stress that such ratings do not correspond to current UNEP performance levels. For the last two years UNEP The Secretariat welcomes the progress of the has implemented several measures to strengthen Agencies toward minimizing the carbon foot- project M&E systems and improve quality of project prints of their internal operations, as documented supervision which have been previously reported to by the assessment undertaken by the GEF Evalu- Council as part of the Management Action Record. ation Office. Of greater interest, however, will be An indicator of improved levels of performance is the the resulting climate change policies and strate- 73 percent rating on quality of project M&E during implementation (table [10.1], parameter 11), which is gies being prepared for adoption by the Agencies based on FY07 terminal evaluation findings. and the resulting impacts that these will have on the Agencies' portfolios. UNDP has noted the following points: In response to the recommendation that the GEF UNDP tracks when its terminal evaluation reports Secretariat, in consultation with the GEF Agen- [TERS] are completed and submitted to the GEF Sec- retariat. For example, a total of 33 TERs will be com- cies, conduct an in-depth assessment of the GEF pleted in UNDP's FY 2007 (year end) and submitted Agencies' efforts to reduce GHG emissions, the to the GEF Secretariat which is a considerably larger GEF Secretariat finds that this task falls beyond number than the 18 UNDP TERs that the GEF Evalu- its mandate and should not be considered a prior- ation Office used to make its conclusions and recom- ity. As noted by the APR, the Agencies are mak- mendations in this APR for GEF FY 2007. ing progress toward understanding and managing UNDP regards the Performance Matrix as potentially their carbon footprints and are cooperating with useful for fair and transparent benchmarking provided each other and the UN Environment Manage- the criteria for measuring each parameter are objec- ment Group in this area. This finding indicates tively defined, the parameters remain constant from Annex E. Management Response 75 year to year, and the time periods for reporting are Para 19 and 20. No evidence is presented on why there clear--we note for example that the current version should be doubts about the sustainability of capacity mixes current and historical data. building. To say that sustainability is low because few national or long-term training programs were devel- The World Bank has noted the following points: oped focuses on a continuation of the provision of training rather than on whether the training provided On a general note, with the exception of the Institutional by the project enabled better institutional or individ- Performance Matrix, the report presents aggregated ual performance. Moreover, capacity development is findings, conclusions, and recommendations across all broader than training per se. Development of long- the Agencies, with no differentiation. Our preference term training programs also has long-term financial would be to avoid such distinctions among Agencies but implications. it is clear from the Performance Matrix that the World Bank recorded high ratings on all but one of the indica- Para 27. The Bank's Implementation Completion tors (quality at entry of M&E) and the highest among the Report template includes fields for financial report- Agencies in nearly all cases. Generalizing on the results ing and for analysis. Again, the generalization of the especially given the implications for specific Agency findings probably masks different practices among the action masks these differences and provides a one-size- Agencies. fits-all approach. In previous years, Agencies that were particularly at fault in specific areas were identified. For Para 30. At the end of the paragraph, we suggest add- example, conclusion 1 gives the overall outcome ratings ing the following text: "though the financial costs may in the satisfactory range as close to the 75 percent tar- be higher." It would also be useful to know whether get. The World Bank's result was 87 percent, which is there is a cutoff point when the costs under the direct significantly above the satisfactory range. route followed by the GEF Secretariat are so high that it outweighs the GHG benefits and therefore carbon Para 2. In the case of the World Bank, the GEF Evalu- neutrality could be more efficiently obtained from ation Office has previously stated that it accepts the purchasing carbon offsets. findings of IEG's project reviews. Therefore, there is no uncertainty in the verification process. Perhaps the Para 33. This is another example of the generalizations. Evaluation Office needs to more fully involve the cen- There is a difference between those Agencies that tral evaluation units of other Agencies in the process, achieved six or seven of the criteria and those that have but it not only already works closely with IEG but has only achieved one (International Fund for Agricultural modeled its own project reviews on IEG's processes Development) or two (UNDP). We earlier pointed out and formats. Similarly, with regard to the reference in in written comments that the Bank does include a cli- the paragraph to piloting a verification methodology mate policy in its overall strategy, through the broader in the field, IEG's Project Performance Assessment greening program, which for example includes energy Report is used for this purpose and has been previ- efficiency in buildings. ously accepted by the Evaluation Office. It would be Performance Matrix. It is not clear why there is missing useful, therefore, to provide a footnote indicating the information in the case of the Bank for the following. status of independent evaluation in the World Bank. For line 3 the Bank provided information on elapsed Para 6. Is there evidence that information on project time as part of the Project Implementation Review. completion is not available in a timely manner? Is there Line 6, IEG's independence is well established. Line 7, an assessment of the lag between preparation of the IEG does an independent review of the terminal eval- completion report and its receipt by the Evaluation uations. The result for the Bank in Line 13 (11) con- Office? In the Bank's case, all completion reports for veys the wrong impression. The quality of the Bank's FSPs are automatically sent to the Evaluation Office at terminal evaluations has consistently been rated in the same time it is circulated to other units within the the 80 percent range and hence there is far less room Bank. As a result there should be no lag in receipt of for improvement. The rating of 3 conveys the wrong the reports, as well as IEG's review. impression. 76 GEF Annual Performance Report 2007 References The GEF Council documents cited here (indicated GEF EO (Global Environment Facility Evaluation with the designation "GEF/C.xx") are available on Office). 2006. The GEF Monitoring and Evaluation Policy. Evaluation Document No. 1. Washington, the GEF Web site, www.thegef.org, under Docu- DC. ments/Council Documents. GEF Evaluation Office documents can be found on the GEF Evaluation --. 2007. "Guidelines for GEF Agencies in Con- ducting Terminal Evaluations." Draft. Office Web site, www.gefeo.org, under Publications. GEF (Global Environment Facility). 2000. "GEF Project OECD DAC (Organisation for Economic Co-operation Cycle." GEF/C.16/Inf.7. and Development Development Assistance Com- mittee). 2002. Glossary of Key Terms in Evaluation --. 2006. "Summary of Negotiations on the and Results Based Management. www.oecd.org/ Fourth Replenishment of the GEF Trust Fund." dataoecd/29/21/2754804.pdf. GEF/C.29/3. 77 GEF Evaluation Office Publications Number Title Year Evaluation Reports 44 GeF Annual country Portfolio evaluation report 2008 2008 43 GeF country Portfolio evaluation: South Africa (1994­2007) 2008 42 GeF country Portfolio evaluation: Madagascar (1994­2007) 2008 41 GeF country Portfolio evaluation: benin (1991­2007) 2008 40 GeF Annual Performance report 2007 2008 39 Joint evaluation of the GeF Small Grants Programme 2008 38 GeF Annual Performance report 2006 2008 37 GeF country Portfolio evaluation: Samoa (1992­2007) 2008 36 GeF country Portfolio evaluation: the Philippines (1992­2007) 2008 35 evaluation of the experience of executing Agencies under expanded opportunities in the GeF 2007 34 evaluation of Incremental cost Assessment 2007 33 Joint evaluation of the GeF Activity cycle and Modalities 2007 32 GeF country Portfolio evaluation: costa rica (1992­2005) 2007 31 GeF Annual Performance report 2005 2006 30 the role of Local benefits in Global environmental Programs 2006 29 GeF Annual Performance report 2004 2005 28 evaluation of GeF Support for biosafety 2006 third overall Performance Study 2005 GeF Integrated ecosystem Management Program Study 2005 biodiversity Program Study 2004 climate change Program Study 2004 International Waters Program Study 2004 Evaluation Documents eD-3 Guidelines for GeF Agencies in conducting terminal evaluations 2008 eD-2 GeF evaluation office ethical Guidelines 2008 eD-1 the GeF evaluation and Monitoring Policy 2006 Evaluation Office G LOBAL ENVIRONMENT FACILITY Global Environment Facility Evaluation Office 1818 H Street, NW Washington, DC 20433 USA www.gefeo.org