THE WORLD BANK 2005 Annual Report on Operations Evaluation INDEPENDENT EVALUATION GROUP ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Independent Evaluation Group (IEG) reports directly to the Bank’s Board of Executive Directors. IEG assess- es what works, and what does not; how a borrower plans to run and maintain a project; and the lasting contri- bution of the Bank to a country’s overall development. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank’s work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. WORLD BANK INDEPENDENT EVALUATION GROUP 2005 Annual Report on Operations Evaluation 2006 The World Bank http://www.worldbank.org/ieg Washington, D.C. © 2006 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 Telephone 202-473-1000 Internet www.worldbank.org E-mail feedback@worldbank.org All rights reserved Manufactured in the United States of America The findings, interpretations, and conclusions expressed here are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. The World Bank cannot guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in the work do not imply on the part of the World Bank any judgment of the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The World Bank encourages dissemination of its work and will normally grant permission promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400, fax 978-750-4470, www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, World Bank, 1818 H Street NW , Washington, DC 20433, USA, fax 202-522-2422, e-mail pubrights@worldbank.org. Cover photo: Curt Carnemark. ISBN-10: 0-8213-6521-5 ISBN-13: 978-0-8213-6521-2 e-ISBN: 0-8213-6522-3 DOI: 10.1596/978-0-8213-6521-2 e-ISBN 0-8213-6462-6 DOI: 10.1596/978-0-8213-6461-1 Library of Congress Cataloging-in-Publication Data has been applied for. World Bank InfoShop Operations Evaluation Department E-mail: pic@worldbank.org Knowledge Programs and Evaluation Capacity Telephone: 202-458-5454 Development (OEDKE) Facsimile: 202-522-1500 E-mail: eline@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 Printed on Recycled Paper Contents v Acknowledgments vii Foreword xi Acronyms and Abbreviations 3 1 Introduction 3 Relationship of the 2005 AROE to Previous AROEs 3 Importance of a Country Focus 4 Overview of the 2005 AROE 7 2 Measuring and Managing Performance at the Country Level 8 Strategic Planning 11 Performance Measurement 16 Performance Management 18 Conclusions 23 3 Update on Monitoring and Evaluation of Bank Operations 23 Monitoring of Bank Operations 28 Evaluation 31 Conclusions 35 4 Improving IEG’s Effectiveness 35 IEG’s Response to 2004 AROE Recommendations 37 IEG Client Survey Results 38 Recommendation for IEG 39 5 Conclusions and Recommendations 39 Recommendations 41 Appendixes 43 A: Review of CAS Results Matrices 57 B: Country Director Survey Results on the Use of Monitoring and Evaluation Systems and Information at the Country Level 63 C: Monitoring and Evaluation Framework in the Bank 65 D: IEG Client Surveys 75 E: Management Response 81 Endnotes 83 References iii 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N Boxes 7 2.1 Elements of Results-Based Management 9 2.2 What Is a Results-Based CAS? 12 2.3 Harmonization Around Results 13 2.4 Observations from IEG’s Review of the Poverty Reduction Strategy Initiative 14 2.5 IEG Lessons on Harmonization from Evaluation 15 2.6 Developing the Capacity to Manage by Results at the Country Level 17 2.7 How Performance Information Is Used 19 2.8 Incorporating Results in Portfolio Reviews 32 3.1 Lessons on Results-Based Management in the U.S. Federal Government Figures 10 2.1 Distribution of Performance Measures by Evaluation Level 17 2.2 Important Uses of Information iv Acknowledgments This report was prepared by a team comprised Keith Mackay, and Kyle Peters, provided of Janardan Prasad (J.P.) Singh (Task Manager), extensive support and contributions to all phases Lauren Kelly, Howard Smith, and Yoshine of this report. Basil G. Kalvasky and F. Stephen Uchimura. The peer reviewers were Harry O’Brien provided feedback on the final draft. We Hatry, Hartwig Schafer, and Hasan A. Tuluy, who gratefully acknowledge their assistance. provided invaluable advice and guidance in Others who have provided valuable input preparing the report. and assistance include Princess Moore-Lewis. IEG is grateful to the country directors who Lydia Ndebele and Juicy Qureishi-Huq provided participated in the survey conducted for the administrative support. Alex H. Mckenzie, Julius report and who shared their time and experi- Gwyer, Maria J. Mar, and Yang Soo Yoon offered ence. Our thanks also go to Nick Gompper computer and technical support. (Caliber Associates), Vivian R. Jackson, and Ray Document production support was Rist and for their support in survey design and provided by William Hurlbut. Caroline McEuen analysis. edited the report. Many IEG staff, including Soniya Carvalho, The report was prepared under the direction Laurie Effron, Victoria Elliott, Nils Fostevedt, of Klaus Tilmes, Manager, Knowledge Programs Patrick Grasso, Fareed Hasan, John Johnson, and Evaluation Capacity Development. Director-General, Independent Evaluation, World Bank Group: Vinod Thomas Director, Independent Evaluation Group–World Bank: Ajay Chhibber Manager, Knowledge Programs and Evaluation Capacity Development: Klaus Tilmes Task Manager: Janardan Prasad (J.P.) Singh v Foreword T he World Bank launched its Managing for Results initiative in 2002 to enhance the effectiveness of its operations. The focus on results is a wel- come development and reflects an interest within the development com- munity and the Bank to better demonstrate the effectiveness of its assistance, and needs to be encouraged. The 2005 Annual Report on Operations CAS period. A review of CASs conducted for the Evaluation (AROE) focuses on the country 2005 AROE found that most attempted to link because it is the main unit of account for country development goals with CAS outcomes monitoring, managing, and evaluating perform- and Bank interventions. These linkages were ance. It examines Country Assistance Strategies better developed in education, health, and the (CASs) for how well they link country goals, CAS environment than in other disciplines. These outcomes, and Bank programming. It also particular CASs included performance examines how Bank information from M&E measures to track progress toward achieving systems is actually used to manage the perform- goals and outcomes. However, many of these ance of Bank country programs. Finally, the measures lacked baselines, specific targets, or report takes stock of the measures taken since both, limiting their effectiveness for monitor- the 2003 and 2004 AROEs to strengthen the ing, management, and evaluation. results focus in monitoring and evaluation. A greater results focus leads to shared accountability for development results between Measuring and Managing Performance the government and the Bank. A survey of at the Country Level country directors conducted for the 2005 AROE The Results-Based Country Assistance Strate- found that only one-third have developed a gies provide the framework to better link common system with the government to country goals with Bank operations. They develop information and jointly monitor represent a strengthening of the traditional progress toward CAS objectives. The reason CASs, which set broad goals but did not clearly often noted for the absence of joint systems is a articulate the outcomes the Bank or the lack of country capacity or a lack of interest in country wanted to achieve at the end of the performance measurement, a finding confirmed vii 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N by other IEG studies. This indicates that more Closing the Gaps to Increase a Results progress is needed in this area. Focus in Monitoring and Evaluation The Bank is supporting countries in develop- Systems ing performance measurement through The Bank is making progress in establishing Bankwide efforts such as the Statistical Capacity guidelines to strengthen the results focus of its and the Evaluation Capacity Development monitoring and evaluation at the country, Programs and through country-specific activi- sector, and instrument levels. At the project ties as part of its country programs. It is level, monitoring and reporting on results has important that capacity development be been strengthened with the revised Project conducted systematically, in collaboration with Appraisal Document (PAD), which now development partners, to ensure that key outlines who is responsible for producing the constraints are identified and addressed. The required information for monitoring review of the CASs conducted for the 2005 implementation progress. The Project Supervi- AROE found that the CASs include references to sion Report has been replaced by the monitoring and evaluation, but that monitoring Implementation Status and Results Report and evaluation are not being approached in a (ISR), which gives more prominence to results. systematic or strategic manner. The Bank Outcome tracking systems have been should assess the country’s performance established for the Bank’s economic and sector measurement capacity and identify actions work and technical assistance and for trust needed to strengthen it as part of CAS prepara- fund–supported activities, but these systems tion to ensure consistency with priority country are not yet being widely used. needs. At the sector level, the initial Sector Strategy To achieve results, information on perform- Implementation Update (SSIU) replaced ance has to be useful to both government and individual sector strategy updates and Bank country managers. Country directors documented progress in all sectors. Remaining surveyed for this AROE report that improving challenges include developing results country-level management to achieve greater frameworks for each sector and theme, results is the main use of information at the integrating them into the Bank’s operational country level. Within countries there is often a work, and reconciling tensions between perception that monitoring and evaluation country priorities and global sector priorities. information is collected to meet external This reaffirms the 2004 AROE, which identified reporting requirements. sectoral fragmentation and unclear account- The Country Portfolio Performance Reviews abilities in the Bank’s matrix structure as a (CPPRs) could be made into a tool for the major constraint to a stronger results orienta- government and the Bank to jointly manage tion in the Bank. performance and used to develop an At the corporate level, the Millennium understanding of the use of performance Development Goals (MDGs) and the IDA14 information. The 2005 AROE found several performance measures provide an overall instances where the Bank used CPPRs to framework for monitoring results Bankwide. The increase awareness of results and identified Regions select from the MDGs, IDA14 country possible uses of performance information to outcome indicators, and the sectoral indicators improve operational outcomes. The CPPR developed under the Implementation Follow-Up could help clarify and strengthen the links exercise when determining key performance between CAS outcomes and individual Bank indicators for final Regional outcomes in their operations, help identify possible areas of Strategy and Performance Contracts. Also, the synergy, and better align the results of individ- Bank is designing a Results Reporting System to ual Bank operations with overall country capture and disseminate information for results. corporate learning and accountability. viii FOREWORD IEG’s Role in Increasing the Results Recommendations Focus in Monitoring and Evaluation IEG has strengthened its attention to results- For Management oriented monitoring and evaluation in its • Incorporate in the CAS a diagnosis of the coun- evaluations at the country, sector, and project try’s performance measurement capacity and levels, and this work will need to continue and recommendations for action to strengthen deepen. IEG is also working to improve that capacity. The diagnosis and actions should awareness of its products. The results of the be customized to country capacity and focused annual IEG client survey for 2005 show that 45 on critical information needed for the gov- percent of Bank staff who responded to the ernment to implement its program. survey were aware of the evaluation for which • Expand use of the Results-Based CAS as a tool they were surveyed, a marked increase over the to manage country programs and strengthen awareness levels found in previous AROE the link between the Results-Based CAS and the surveys (2004: 27 percent). There is an ongoing country program by providing country teams need for IEG to pursue improvements in the with guidance designed to increase the results communication and dissemination of its orientation of CPPRs. evaluation products within the Bank and among its external clients and partners. The For IEG new Outreach and Communication Strategy • Review the results of pilot initiatives to promote launched by IEG will be tracked by key more effective outreach and communication of performance indicators. Lessons learned IEG products and consider lessons learned in during the pilot phase will inform the full the full implementation of the communica- implementation of the strategy. tion strategy. Nils Fostvedt Acting Director-General, Operations Evaluation ix ACRONYMS AND ABBREVIATIONS AAA Analytical and advisory activities ACS Activity Completion Summary AIS Activity Initiation Summary AUS Activity Update Summary AROE Annual Report on Operations Evaluation ARPP Annual Review of Portfolio Performance BP Business Process (World Bank) CAE Country Assistance Evaluation CAS Country Assistance Strategy CCR Country Assistance Strategy Completion Report CFP Concessional Finance and Global Partnerships CODE Committee on Development Effectiveness CPPR Country Portfolio Performance Review DEC Development Economics/Chief Economist DECDG Development Economics/Chief Economist Development Data Group DFID Department for International Development (U.K.) DGF Development Grant Facility DIME Development Impact Evaluation Initiative DPL Development Policy Loan ECD Evaluation capacity development ED Executive director ESSD Environmentally and Socially Sustainable Development (World Bank network) ESW Economic and sector work GCR Grant Completion Report GPP Global programs and partnerships ICRs Implementation Completion Reports ICM Implementation Completion Memorandum IDA International Development Association IDF Institutional Development Fund IEG Independent Evaluation Group IFU-7 Implementation Follow-Up 7 IPDET International Program for Development Evaluation Training ISR Implementation Status and Results Report KPI Key performance indicators MAR Management Action Record MDGs Millennium Development Goals M&E Monitoring and evaluation OP Operational Process (World Bank) OPCS Operational Policy and Country Services PAD Project Appraisal Document PPAR Project Performance Assessment Review xi 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N PRSC Poverty Reduction Support Credit PRSP Poverty Reduction Strategy Paper PSR Project Status Report QAG Quality Assurance Group RBCAS Results-Based Country Assistance Strategy RMS Results measurement system RRS Results reporting system SAP Systems, Applications, and Products in Data Processing SPC Strategy and Performance Contract SSIU Sector Strategy Implementation Update SSP Sector Strategy Papers TA Technical assistance TF Trust funds TFO Trust Fund Operations VPU Vice presidential unit xii Chapter 1: Evaluation Highlights • The 2005 AROE focuses on the country and examines the use of in- formation by Bank managers to improve results and enhance Bank effectiveness at the country level. • It examines three elements that are critical to a strong results focus: strategic planning, performance measurement, and performance management. • It reviews the actions taken by the Bank to strengthen the results ori- entation in M&E at the country, sector, and product levels. • It evaluates IEG’s effectiveness. 1 Introduction T he Annual Report on Operations Evaluation (AROE) fulfills IEG’s man- date to assess the progress, status, and prospects for monitoring and eval- uating the development effectiveness of World Bank activities. The objective of the 2005 AROE is to assess the use and usefulness of monitoring and evaluation (M&E) information for Bank country program management. results. The Bank’s research on aid effectiveness Relationship of the 2005 AROE and the Monterrey consensus have emphasized to Previous AROEs the importance of developing country ownership The 2003 and 2004 AROEs reviewed the Bank’s of sound policies and governance. Managing for Results initiative and the results Country-level management also is important focus in M&E. The 2003 AROE analyzed M&E at for achieving results. An IEG review of Country the instrument, country, sector, and global Assistance Evaluations (CAEs) found that one- levels and commented on its overall effective- third of evaluated country assistance programs ness. The 2004 AROE examined how results- were unsuccessful (IEG 2004c). While project oriented the Bank’s M&E systems are, the outcomes may have been rated satisfactory, extent to which they contribute to managing they did not always add up to a successful for results in the Bank, and the organizational country program. The review concluded that a changes and incentives needed to move toward more comprehensive view of the outcome of managing for results. the Bank’s assistance programs is possible from the country level. The Quality Assurance Group Importance of a Country Focus has similarly found that “the quality, and hence The 2005 AROE examines the use of information the impact, of the country AAA [analytical and by Bank managers to improve results and advisory activities] programs reviewed are enhance Bank effectiveness at the country level. often less than the sum of their parts” (World The country has supplanted the project as the Bank 2005c). focus of development over the past few decades. The Bank’s results initiative agenda was In the Bank, the country has become the princi- launched in September 2002 in the Develop- pal unit of account for performance M&E. The ment Committee’s paper “Better Measuring, Millennium Development Goals (MDGs) have Monitoring, and Managing for Results.” The highlighted the importance of country-level country program cycle was identified as the 3 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N focal point for formulating, monitoring, and sector, and product levels within the Bank, achieving greater development effectiveness. updating the overview in the 2003 and 2004 In contrast to a project-by-project approach, AROEs. Third, it evaluates IEG’s effectiveness. analysis at the country level captures the It concludes with a set of recommendations for broader context of a country and the political Bank management and IEG. economy of reform. Barriers to development Chapter 2 assesses strategic planning, perform- can be better understood, and interventions ance measurement, and performance manage- designed accordingly. Moreover, the Bank’s ment at the country level and identifies areas that effectiveness will then be judged by the results need strengthening. The analysis done for the the countries achieve with Bank support. assessment used a review of the CASs (Appendix A) and a survey of country directors (Appendix B). Overview of the 2005 AROE Chapter 3 reports on changes made since the The 2005 AROE has three parts. First, it 2003 AROE to enhance M&E and identifies areas examines three elements critical to a strong where it needs further strengthening. Chapter 4 results focus at the country level: strategic examines IEG effectiveness and reports on its planning, performance measurement, and response to recommendations made in the 2004 performance management. It reviews the steps AROE. It also presents highlights of the results of the Bank has taken in these three areas. To the annual IEG client survey (Appendix D). determine current practices, a review was Chapter 5 summarizes conclusions and presents conducted of Country Assistance Strategies recommendations for Bank management and (CASs) submitted to the Board in fiscal years IEG. In addition to the sources already 2004 and 2005. This was supplemented with a mentioned, this report draws on reports from survey of country directors. Second, the AROE Operations Policy and Country Services (OPCS), catalogues the measures taken to strengthen the Quality Assurance Group (QAG), and on IEG the results orientation in M&E at the country, evaluations. 4 Chapter 2: Evaluation Highlights • Results-Based CASs allow a better articulation of the contribution Bank operations make to the achievement of country goals. • While performance measures are included to track progress, im- provement is needed in developing baselines and targeted per- formance measures. • The Bank will need to deepen its work with clients to strengthen their monitoring and evaluation systems. • As the Bank increases its results orientation, the focus of perform- ance management will shift from the Bank to the country and toward shared accountability for results. • Successful performance management requires creating demand for performance information in countries. 2 Measuring and Managing Performance at the Country Level T hree elements are essential for effective results-based management at the country level: strategic planning, performance measurement, and performance management (see box 2.1). Strategic planning is needed to align the To see how these three Results matrices of 24 Bank’s products and services with desired elements fit together in CASs were analyzed to country outcomes to maximize their effective- Bank operations, two ness. The strategy needs to be supported by a reviews were conducted assess their usefulness as system to monitor progress and evaluate as part of this AROE. In a management tool. effectiveness in achieving country outcomes. the first review, the results Finally, the information obtained from M&E matrices of 24 CASs presented to the Board in fiscal needs to be useful to Bank managers for years 2004 and 2005 were analyzed to assess their making operational decisions. usefulness as a management tool and their impact Box 2.1: Elements of Results-Based Management • Identify clear and measurable objectives Strategic Planning • Select indicators that will be used to measure progress toward each objective Measurement Performance • Set explicit targets for each indicator, used to judge performance Performance Management • Develop performance monitoring systems to regularly collect data on actual results • Review, analyze, and report actual results vis-à-vis the targets • Integrate evaluations to complement performance information not readily available from performance monitoring systems • Use performance information for internal management accountability, learning, and decision making processes, and also for external performance reporting to stakeholders and partners Source: Binnendijk 2001. 7 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N on enabling the CAS to become more results- The results-based approach has been taken up based. The review examined the strength of the by CASs other than the seven pilots. Of the 24 linkages between country development goals, CAS CASs reviewed for this report, 13 had adopted or intermediate outcomes, and Bank program- a results-based approach. ming, which together constitute the results The Results-Based CAS allows a better articu- framework. The performance measures were lation of the contribution Bank operations examined to determine whether they were clearly make to the achievement of country goals. The defined, baselined, and targeted and whether they Results Matrix requires the Bank to distinguish provided information that would be useful in and then link country development goals, CAS managing the country program. The method and and intermediate outcomes, and Bank results are presented in Appendix A. operations. Of the 24 CASs reviewed for the For the second review, country directors who AROE, 19 attempted to establish such causal had presented CASs to the Board in fiscal years linkages. 2 The review concluded that the 2004 and 2005 were surveyed to learn how they Results Matrix makes it easier to see the contri- use information from M&E systems.1 The bution of Bank operations to the CAS and to survey responses indicate the degree to which country goals. the Bank’s current M&E Countries that had Results-Based CAS pilots Country directors who systems enable them to or that had gone through a Poverty Reduction had presented CASs to the effectively manage their Strategy Paper (PRSP) exercise were able to Board were surveyed to country operations. The formulate improved linkages. learn how they use complete results of the The understanding of results chains and use survey and the meth- of the matrix could be strengthened, however. information from Bank od are presented in For example, some of the weaker CAS matrices M&E systems. Appendix B. simply listed goals, measures, and programs under pillar headings (the pillars: strengthen- Strategic Planning ing country capacity, enhancing the results The Bank is making progress in strength- focus in operations, and fostering a global ening country-level strategic planning, partnership to manage for results). This yet improvement is needed in developing approach makes it very difficult to understand baselines and targeted performance the assumed interactions between actions and measures. results and to assess the contributions of A management review of Bank CASs (World particular programs to results. Bank, OCPS 2005a) concluded that it was Lack of baselines and targeted performance difficult to use the CAS to assess Bank effective- measures limits the effectiveness of the Results ness. The CAS policy and program matrix that Matrix as a monitoring, management, and underpin the self-evaluation framework set evaluation tool. A qualitative assessment of the broad goals, but did not clearly articulate the performance measures for country, CAS, and outcomes that the Bank or country wanted to Bank intermediate outcomes was done to achieve by the end of the CAS period. Nor did assess how well they were defined, whether it clearly differentiate between long-term baselines were provided, and whether they country goals and goals to which the Bank included targets. The country, CAS, and Bank program is designed to contribute. It also intermediate performance measures for each lacked baseline data and targets to monitor CAS were sorted into four categories, with a progress and assess whether the intended higher-numbered category indicating that the results were achieved. measures were better defined, baselined, or The Bank introduced the Results-Based CAS targeted. For example, performance measures (described in box 2.2) to sharpen the strategic in one CAS may be rated Level 3 for definition if focus of the CAS and strengthen it as a more than 75 percent are well defined, Level 2 planning, management, and evaluation tool. for baselines if more than half but less than 75 8 M E A S U R I N G A N D M A N A G I N G P E R F O R M A N C E AT T H E C O U N T R Y L E V E L Box 2.2: What Is a Results-Based CAS? A Results-Based Country Assistance Strategy is oriented to- • An M&E framework for learning and self-evaluation that uses ward achieving realistic outcomes and includes a results-oriented existing country’s systems (to the extent possible) to steer im- M&E system. Its central innovation is the design of a results plementation toward outcomes. framework that specifies expected linkages between the Bank’s interventions and long-term development goals. A greater focus Results are selected during CAS design, tracked during im- on results, in turn, affects the CAS’s vision, diagnosis, and pro- plementation, and evaluated at the end of the CAS period. The build- gramming. The features of the new CAS framework are: ing blocks of the design include three main components: (a) a Results Matrix that summarizes the results framework, providing • More explicitly defined outcomes that establish clear links be- the links upward from CAS outcomes to the broad development tween the two ends of the results chain: development goals of goals of the CAS, and downward to the proposed Bank program- the country and Bank-supported interventions ming; (b) a CAS Progress Report for mid-point stocktaking and, as • A stronger alignment of the Bank’s program with the country’s vi- necessary, course correction; and (c) self-evaluation of the coun- sion for economic and social development and poverty reduction, try assistance program in a CAS Completion Report. IEG con- and development goals and priorities that derive from this vision ducts a review of the CAS Completion Report. Source: World Bank, OCPS 2005a. percent have baselines, and Level 1 for target- health, and the environment than elsewhere. ing if less than half have clear targets. 3 The That is, the causal linkages between country methods and results of the review are outcomes, Bank out- Lack of baselines and described in Appendix A. Figure 2.1 presents comes, and Bank ac- the combined results of the evaluation for tivities were stronger in targeted performance definition, baselining, and targeting. It shows these sectors, so the measures limit the that the performance measures were generally CASs were able to effectiveness of the Results adequately defined, but they often lacked present a more com- baselines and targets.4 The review concluded pelling, results-oriented Matrix as a monitoring, that 54 percent of the country, CAS, and narrative through the management, and intermediate measures combined were consid- Results Matrix. evaluation tool. ered to be well defined (at Level 3), while only Other studies have 18 percent were considered well targeted, and also found a stronger results focus in edu- 13 percent adequately baselined. At the same cation and health. In reviewing projects under time, 24 percent of the measures were rated implementation, QAG (World Bank 2005d, poor (Level 0) for targeting and 28 percent for Annex 9) noted that projects in health and baselines, but only 15 percent of the measures education, along with those aimed at poverty were considered to be poorly defined. reduction, took a more systemic approach to Results-based approaches and performance monitoring and evaluation, often using broad measures are better developed in some areas national outcome indica- than in others, which may explain some of the tors and relying on the Results chains and disparity. Performance measures generally were monitoring of project performance measures better developed at the country level, where performance indicators were better developed in more established measures (such as the MDGs) under another project or education, health, and are available. Countries with a PRSP had an a government-managed advantage because PRSPs included results evaluation program. the environment. monitoring at the country level. The review also IEG’s CAE Retrospective (IEG 2005a) also noted that results chains and performance found that Bank country programs were more measures were better developed in education, successful in education and health than in 9 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N Figure 2.1: Distribution of Performance Measures by Evaluation Level (percent) 18% Level 3 13% 54% 18% Level 2 15% Evaluation level 19% 40% Level 1 44% 11% 24% Level 0 28% 15% 0 10 20 30 40 50 60 Percentage of cases Targeted Baselined Defined other sectors. A possible explanation is that many CASs, even those with smaller numbers of these sectors have had a stronger focus on measures, lacked adequate baselines and targets. outcomes. For example, educators have been However, larger numbers of measures would concerned about the impact of their programs increase the challenge for measurement and on student learning, and health professionals determination of baselines. are focused on the improvement in the health While the number of measures should of patients. reflect the scope of the Bank’s country The large number of measures may have program, the Bank needs to be selective in exacerbated the problem of baselining and determining which measures to monitor, targeting. The average number of country especially in view of the data constraints performance measures was 22; CAS outcome within the country. A large set of measures measures, 39; and intermediate outcome may be needed for evaluation at the end of measures, 40. One CAS the CAS period, but a smaller set could be The Bank needs to be had 80 outcome meas- used to monitor and manage the country selective in determining ures and 97 intermediate program. Some CASs summarize the key which measures to measures. The extent of performance measures, establishing a priority Bank involvement and among them. A more systematic approach monitor, especially in the size of the program will be to establish the information necessary view of data constraints would drive the number to manage the country program, and ensure in countries. of measures needed to that the measures include baselines and manage the program. It targets. Some of these measures can be is not possible to attribute the lack of baselines qualitative, such as policy or administrative and targets directly to large numbers because milestones. 10 M E A S U R I N G A N D M A N A G I N G P E R F O R M A N C E AT T H E C O U N T R Y L E V E L Country ownership of monitoring data is ciaries. These data collec- A stronger focus on important. The country’s capacity to collect and tion efforts could results will require the analyze the information should be considered become more elaborate, in establishing the performance measures. time consuming, and Bank to work closely with Availability of data, however, should not be a expensive in the future. its clients to collect data main reason for selecting measures. It may be For example, an agency regularly on the outcomes necessary to include indicators for which data may rely on its operating collection systems will need to be developed units to provide informa- of Bank-supported during the CAS period. The link to country tion on outputs such as operations. capacity is discussed in more detail in the kilometers of roads built following section. or training programs conducted, but may need to conduct surveys to Performance Measurement collect information on beneficiary impact. To manage for results, the Bank will need Similarly, the move toward monitoring higher- to deepen its work with clients to level country outcomes shifts the focus of data strengthen their monitoring and evalua- collection from units within agencies to agency- tion systems and take a systematic wide or national systems. While strengthening approach to performance management at the results focus and project-level monitoring and the country level. evaluation systems continues to be important, The Bank has systems to help managers more attention would need to be given to manage their country operations and needs to strengthening performance measurement at the continue to make the information readily sector or agency level and at the national level. available. The Bank’s internal information The Bank will need to work closely with systems allow Bank managers and staff to country and development partners to coordi- monitor budget use, product deliveries, and nate information needs and performance the quality of the portfolio at the country level. measurement at the The Bank will need to In a survey of selected country directors carried country level. Country out for this report (Annex B), 67 percent of data, such as the MDGs, work closely with respondents indicated that they were able to will be used by the country and development obtain the information needed from the Bank’s government, other partners to coordinate information systems. Many (75 percent) donors and develop- packaged the information to better meet their ment partners, and the information needs and needs by consolidating data from different Bank to monitor performance measures at Bank systems or by including outcome progress toward achiev- the country level. information obtained from country sources. ing country strategic Several country directors expressed a need for goals. To be effective, performance measure- more systematic assembly of country informa- ment systems need to be country-owned and tion for program management. The Bank is used, focused on essential information, and working to improve management of country- harmonized between the country and other level information; one proposed solution is a stakeholders, including other donors, multilat- Country Director’s Portal.5 eral institutions, and the Bank. A stronger focus on results will require the A multi-partner evaluation of the Compre- Bank to work closely with its clients to collect data hensive Development Framework (CDF regularly on the outcomes of Bank-supported Evaluation Management Group 2003) noted operations and to adjust the way it monitors that “development performance should be country program implementation. At the project evaluated through measurable, on-the-ground level, the emphasis on outcomes requires the results” and recommended that all donors and implementing agency to collect information on others “strengthen and use country-led M&E the impact of its services and products on benefi- systems, view country development outcomes 11 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N as a joint product, to which donors contribute discuss progress toward CAS objectives, along under country leadership,” and see results with operational matters such as portfolio monitoring as “a shared responsibility in which status and readiness of projects under prepara- donors and governments use the same tion, with their government counterparts. information flows for their individual These discussions are conducted as part of purposes.” regular consultations and program and portfo- The development community supports lio reviews. In low-income countries, PRSP and greater collaboration and country ownership in Poverty Reduction Support Credit reviews are performance measurement. Following the important vehicles for these discussions. Second International Roundtable on Managing But most survey respondents do not yet for Results in Marrakech in 2004, the leaders of have a common system or process to develop the development agencies stressed the need to and share results information with the govern- “align cooperation programs with desired ment. Fewer than one-third (31 percent) of the country results, define the expected contribu- responding country directors indicated that tion of our support programs to country they have a system or process in common with outcomes, and rely on and strengthen countries’ the government to jointly develop and share monitoring and evaluation systems to track information for monitoring progress toward progress and assess outcomes” (African meeting CAS objectives. Most existing systems Development Bank and others 2004). The Paris and processes were centered on the PRSP. Declaration on Aid Effectiveness issued in March Several country directors indicated that they 2005 reaffirmed the commitment of the were currently developing a joint system with countries, donors, and multilateral institutions the government through an Institutional to country ownership and coordination around Development Fund (IDF) grant or other country-led information systems (box 2.3). means. Lack of government capacity or interest The survey of country directors found that was often cited as the reason for not develop- 87 percent of those responding regularly ing a joint monitoring system. Lack of interest Box 2.3: Harmonization Around Results The ministers from developing and developed countries and the and align them with effective partner-country per- heads of multilateral and bilateral development institutions met formance assessment frameworks, refraining from re- in Paris in February-March 2005 to discuss measures to im- questing the introduction of performance indicators prove aid effectiveness. One of the commitments from the meet- that are not consistent with partners’ national devel- ing is to “managing for results.” Specifically: opment strategies. d. Work with partner countries to rely, as a far as possible, on • Partner countries commit to: partner countries’ results-oriented reporting and monitor- a. Strengthen linkages between national development strate- ing frameworks. gies and annual and multi-annual budget processes. e. Harmonize their monitoring and reporting requirements, b. Endeavor to establish results-oriented reporting and as- and—until they can rely more extensively on partner coun- sessment frameworks that monitor progress against key di- tries’ statistical, monitoring, and evaluation systems—part- mensions of the national and sector development strategies, ner with countries to the maximum extent possible on joint and that these frameworks should track a manageable formats for periodic reporting. number of indicators for which data are cost-effectively • Partner countries and donors jointly commit to: available. f. Work together in a participatory approach to strengthen • Donors commit to: country capacities and the demand for results-based c. Link country programming and resources to results management. Source: Paris Declaration on Aid Effectiveness, March 2, 2005. 12 M E A S U R I N G A N D M A N A G I N G P E R F O R M A N C E AT T H E C O U N T R Y L E V E L was more prominent in middle-income can support domestic Most country directors countries where there is no PRSP equivalent to decision making (box surveyed do not have a focus the country, donors, and the Bank on 2.4). results.6 Several country directors mentioned A similar observation system or process to the time and effort required to develop joint was made in an IEG develop and share results systems with the government.7 assessment of evaluation information with the IEG reviews have also identified lack of capacity development capacity for data collection and limited country efforts (IEG 2004b). government. demand for performance information as Following in-depth reviews in two countries with constraints to developing country performance extensive Bank involvement, the report observed measurement systems. The multi-partner that weak demand could result in weak systems evaluation of the Comprehensive Develop- and that greater attention should have been paid ment Framework (CDF Management Group to sector ministries. These ministries are responsi- 2003) noted that most government systems ble for service delivery and could use the informa- were ill-suited to the often “elaborate and tion and become the focus of M&E. complex monitoring approaches” donors IEG has also cautioned that harmonization may required of them. take time and effort. An overview of IEG’s work An IEG review of the Poverty Reduction reconfirmed the importance of harmonization in Strategy Initiative (IEG 2004a) noted that reducing transaction costs and enhancing develop- results orientation was one of the weaker ment effectiveness. IEG areas of the process, although “results orienta- noted that successful Lack of both capacity for tion with a focus on outcomes that benefit the harmonization hinges on data collection and poor” was one of the five underlying princi- country capacity and demand for performance ples of the PRSPs. Establishment of monitor- government commit- ing systems was constrained by lack of ment, especially where information are capacity for M&E in the countries. A results changing “the existing constraints to developing orientation was impeded in some cases by a way of doing business” country performance decision to track a large number of indicators could conflict with differ- for which too little data was available. Efforts ent donors’ and agencies’ measurement systems. to build monitoring capacity and to monitor own interests. It also results were seen by governments in many cautioned that harmonization can have significant countries as a requirement imposed from up-front costs and prove frustratingly slow relative outside. The study noted that countries need to expectations. The lessons are summarized in to select and monitor relevant indicators that box 2.5. Box 2.4: Observations from IEG’s Review of the Poverty Reduction Strategy Initiative Added attention to monitoring, but results focus remains weak. rience. In many cases, monitoring systems are designed to meet The Poverty Reduction Strategy (PRS) process has begun to donor data requirements—for example, through the progress re- orient stakeholders toward a results focus, although the devel- ports—even when they exceed the country’s needs or capacity. opment of country-specific indicators and monitoring systems Since the majority of low-income countries will probably not to track them is still at a preliminary stage in most countries. Avail- be able to meet most of the Millennium Development Goals, PRSPs able information is generally not linked to decision making. must be more effective in enabling countries to select and moni- Some countries have adopted PRSP targets that are unrealis- tor relevant indicators that can support domestic decision mak- tic in light of their initial conditions, resources, and recent expe- ing on a sustained basis. Source: Extract from IEG, 2004a, p. xv. 13 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N The Bank is working with countries to IEG has supported another 17 countries, strengthen their M&E systems. The Develop- primarily through knowledge sharing ment Economics Department’s Development (provision of materials, identification of Data Group (DECDG) experts, and the like) and has prepared and The Bank is working with manages the global Trust widely disseminated case studies, manuals, and countries to strengthen Fund for Statistical other resource materials on ECD. Finally, over their M&E systems. Capacity Building, which 630 participants from 62 countries have helps countries to attended the International Program for improve the collection, processing, analysis, Development Evaluation Training (IPDET), a dissemination, and use of statistics, especially four-week course developed by IEG and to support poverty reduction. DECDG has Carleton University. The IEG self-evaluation of designed a new lending program, the Statistical ECD concluded that “IEG has been highly Capacity Program, to support the development active in ECD, and the ECD strategy has of more effective and efficient statistical systems contributed to a number of ECD results, in developing countries. The program is under outputs and outcomes, over the past five years” way in Burkina Faso and Ukraine, and prepara- (IEG 2004b, p. ii). tions have started in four additional countries.8 Third, the Bank is supporting efforts to DECDG will take the lead in improving statisti- strengthen M&E at the country, sector, and cal capacity in International Development project levels through use of Institutional Association (IDA) countries under IDA14. Development Fund grants, coordination with In addition, IEG has been engaged in evalua- other donors, and AAA. These efforts are led by tion capacity development (ECD), helping the country departments. Examples of some countries to develop their M&E systems recent innovative approaches where the Bank through technical assistance and training (IEG is focusing on country capacity development at 2004b). IEG has provided ECD support to 34 a national level are described in box 2.6. countries since 1999 and has engaged Treatment of performance measurement in intensively in 2 countries, Uganda and Egypt, CASs is relatively weak and inconsistent. Most and to a lesser extent in 15 others, through of the 24 CASs reviewed for this report diagnostics, organization of workshops and included references to M&E, but many did not other training, and support of local evaluation provide much detail. Only a few discussed data associations. The Uganda experience is availability and contained proposals to described in box 2.6. strengthen monitoring and evaluation of the Box 2.5: IEG Lessons on Harmonization from Evaluation IEG’s evaluations have demonstrated that proliferation of un- Poverty Reduction Strategies. coordinated procedures and requirements across aid agencies • Where successful, harmonization has required capacity for is imposing substantial costs on both donors and recipients and and commitment to reform on the part of both donor and re- limits the overall effectiveness of aid. The evaluations have so cipient countries. far found only modest progress in the harmonization of aid • Harmonization can have significant up-front costs, as has processes and programs. Nevertheless, the evaluations support been the case with the Poverty Reduction Strategy Initiative. the following lessons about harmonization: • Harmonization among donors may not always align donor strategies with country development goals. • Harmonization can be improved through mechanisms such • Changing donor procedural requirements has proved to be as Consultative Groups, Sector-Wide Approaches, and slow and difficult. Source: IEG documentation. 14 M E A S U R I N G A N D M A N A G I N G P E R F O R M A N C E AT T H E C O U N T R Y L E V E L Box 2.6: Developing the Capacity to Manage by Results at the Country Level • Evaluation Capacity Development in Uganda: At the request Program is intended to help the government develop a policy of the Bank’s country director and Ugandan authorities, IEG framework for comprehensive program evaluation and con- started providing evaluation capacity development (ECD) duct a series of comprehensive evaluations to pilot the frame- support to Uganda in 2000. The work included a government- work, most of them in Bank-supported programs in social wide diagnosis of M&E and a study of M&E in the education, development, health, education, agrarian development, hous- health, and water sectors and workshops to disseminate ing and urban development, and micro-credit. M&E practices and experiences. The policy framework would establish incentives for line min- Thirty Ugandan officials were given scholarships to attend istries to comprehensively evaluate their programs, put forward the International Program for Development Evaluation Training a methodological framework and guidelines for conducting (IPDET). IEG also provided support to the fledgling Uganda comprehensive evaluations, and specify the institutional roles Evaluation Association. IEG’s work was linked closely to the re- needed to implement a results-based evaluation system within form agenda under the Poverty Reduction Support Credit the federal government. The program is expected to promote (PRSC). a culture of evaluation and complement the Bank’s activities The government decided in 2004 to implement a National to strengthen results orientation in the government through the Integrated Monitoring and Evaluation system based in sub- Results-Based CAS and M&E in individual operations. stantial part on IEG input. IEG partnership agreements with Nor- • Training of Trainers in Morocco: Discussions are under way way and the Netherlands helped fund this work and enabled in Morocco for a program for training of trainers in results- IEG to scale up its support. based project management and M&E, with support from an An IEG review of the experience concluded that, despite in- Institutional Development Fund grant. This follows a results- sufficient attention paid to bureaucratic realities and to sec- based Country Portfolio Performance Review that developed tor ministries, on balance, the impact on evaluation capacity a common understanding of each operation’s relationship to in Uganda had been positive and significant. It also commented country goals, results matrix, performance indicators, and that the most important impact was that “the lip service given M&E requirements. in 2000 to [M&E] has become four years later genuine interest, The training will be done at a local institute that is expected and awareness of the importance of results and of outward ac- to develop expertise and continue such training beyond the pe- countability has percolated through Uganda’s public sector riod of the grant. The expected outcome of the program is a net- and much of society at large.” work of resource persons with strong knowledge of • Brazil Multi-Sectoral Program to Support Evaluation (BRAVA): results-based management, M&E, and fiduciary procedures (es- M&E is challenging in Brazil because the country has neither pecially as they relate to improving project outcomes) who a strong mandate for evaluations nor methodological frame- will be given the responsibility to respond to training needs in works for program assessments. The Bank-supported BRAVA the country in the same areas of focus. Source: IEG 2004b. CAS during implementation.9 While the Bank is country capacity. Bank Preparation of Results- taking measures to strengthen performance CAS guidelines as of measurement at the country level, it is not clear March 2004 mention that Based CASs should take a whether this is being done by identifying priori- “as appropriate, the CAS systemic approach to ties, identifying gaps, and factoring appropriate should discuss efforts to country-level support into the Bank country program. develop monitoring and Preparation of the Results-Based CAS should evaluation capacity in performance take a systemic approach to performance borrowing countries as measurement. measurement at the country level by taking well as efforts at the stock of the M&E information needs; identifying country level to harmonize monitoring and capacity development needs; and building in evaluation approaches and requirements among measures, where appropriate, to develop different partners”10 (World Bank, OCPS 2004b, 15 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N p. 7). These guidelines are being revised to need to report on performance at the country mainstream the experience from the pilot level is expected to increase. Within the Results-Based CASs and propagate good country, there is often a perception that practices. As part of the revision, country teams monitoring and evaluation is driven by external could be required to undertake a diagnosis or concerns (see box 2.4). However, there is a Readiness Assessment to identify areas where growing need to make performance informa- the Bank and the government could work tion part of overall decision making. This will together to ensure that performance measure- occur only if there is demand for performance ment and management are addressed in a timely information within the government. manner. Many country directors who responded to Use of the information for operational the survey carried out for this report see performance enhancement and achievement of learning and management improvement as the agreed outcomes should be the driving force key uses of information.11 They identified behind development of performance measure- learning, business planning, and modifying ment systems. The sources of information, the operations as the most important uses of data to be collected, responsibility for data information (figure 2.2). Accountability was collection and analysis, and capacity develop- considered less important. Several commented ment needs and proposed support could be on the difficulty of determining the extent of summarized in a Monitoring and Evaluation the Bank’s contribution to an outcome as a Action Plan. A good practice note should be reason for giving accountability a lower rating. prepared based on ongoing country efforts to Bankwide reporting on results may increase encourage and assist the country teams with the the pressure on country directors, sector Readiness Assessments and action planning. directors, and managers to focus on perform- ance reporting for accountability for results. Performance Management The Strategy and Performance Contracts that As the Bank increases its results orienta- are being introduced in fiscal 2006 will provide tion, the focus of performance manage- the Regions with a list of indicators from which ment will shift from the Bank to the to select key performance indicators for final country and toward shared accountabil- outcomes.12 While the contracts will be at the ity for results. Establishing the use of, and Regional level, the actual results are achieved creating demand for, performance in the countries. Some links will need to be information will be critical for successful established between the Regional indicators performance management. and the countries, most likely through the Performance information is meaningless if it Results-Based CASs. The experience with PRSPs is not used. An OECD/DAC report identified has highlighted some of the challenges: balanc- two categories of use for performance informa- ing country-specific information needs and tion: (a) management improvement (managing Bankwide monitoring and reporting, ensuring for results) and (b) performance reporting that the information will be useful for country (accountability for results). These concepts are decision makers, and keeping the level of described in box 2.7. The information to be collected manageable and Performance information former is driven by consistent with country capacity. needs to be part of internal organizational As the Bank increases its results orientation, concerns, the latter by the focus on performance management will overall decision making. external stakeholders. shift from the Bank to the country and toward This will occur only if Within the Bank, shared accountability for results. The Bank has there is demand for management improve- a responsibility to deliver loans and credits and ment is seen as the main to provide analysis and advice through studies performance information use of information at the and technical assistance, as agreed with the within the government. country level, but the government. The government, in turn, will 16 M E A S U R I N G A N D M A N A G I N G P E R F O R M A N C E AT T H E C O U N T R Y L E V E L Box 2.7: How Performance Information Is Used An OECD/DAC review of results-based management in multi- The OECD/DAC review noted a potential for conflict between lateral and bilateral aid agencies distinguished two uses of per- the two uses for performance information. As the focus moves up formance information: the results chain (from inputs, to outputs, to outcomes), problems of aggregation and attribution emerge. That is, it becomes more • Management improvement (managing for results): Perfor- difficult to “add up” the results from lower levels (projects to mance information: (a) promotes continuous management country) and attribute changes to Bank operations (clearly demon- learning about what results are being achieved and why and strate that the change was due to a Bank intervention). (b) facilitates operational decision making (implementation, The review suggested resolving this conflict by moving from a strategy and policy formulation, resource allocation, and the like). “command and control” view of accountability, holding managers • Performance reporting (accountability for results): Perfor- accountable for achieving results, to shared accountability, where mance information is used to report performance and ful- managers are held accountable for “working with partners to fill accountability to external stakeholders and internal achieve higher-order results, for learning from failures, and for con- management. tinually using performance information in their decision making.” Source: Binnendijk 2001. need to use these inputs from the Bank ance. Traditional CPPRs As the Bank increases its effectively and provide infrastructure, goods, have been Bank-led, facilities, and services to their beneficiaries. It focused solely on the results orientation, the is essential that the Bank generate demand for ongoing project portfo- focus on performance performance information and continue to lio, addressed primarily management will shift foster the interest, political will, and capacity to implementation prob- from the Bank to the manage for results within the government. lems, and were often a The Country Portfolio Performance Review review of problem country and toward shared (CPPR) can be made into a tool for the govern- projects. The CPPR can accountability for results. ment and the Bank to jointly manage perform- be made more effective Figure 2.2: Important Uses of Information 4.50 4.00 (1 = Not at all—5 = Very much) 3.50 3.00 2.50 2.00 1.50 1.00 Learning Business Modifying Communications Accountability planning operations 17 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N by providing the country with a leadership role, lar reviews of the overall Bank program, rather expanding beyond ongoing projects to include than projects under implementation, and to AAA, and focusing the discussion around assess program effectiveness in achieving CAS progress toward agreed results (country-level outcomes, and thereby generate demand for outcomes and CAS-level outcomes), rather than performance information. solely around implementation issues. Project administration issues can then be discussed in Conclusions the context of overall progress toward agreed The move to a results approach shifts the results and foster joint accountability. focus on performance measurement and An expanded CPPR management from within the Bank to the The CPPR can be would help clarify and country. The shift to outcomes will require developed into a vehicle strengthen the links the Bank to work with the government to put to conduct regular between CAS outcomes in place systems to collect and analyze the and the outcomes of necessary information. This should not reviews of the overall individual Bank instru- become a burden for the government. The Bank program. ments (such as projects Bank will need to: (a) use government and AAA), help identify systems and help build capacity to develop possible areas of synergy across operations, and maintain such systems; (b) help to and begin to address the disconnect between generate demand for, and use of, the informa- overall country and individual instrument tion; (c) simplify and emphasize essential results. It would complement the CAS Progress information; and (d) harmonize information Reports and Completion Reports by generating systems across Bank operations, as well as information needed for country program with other development agencies. management and by incorporating results into The Bank should take a more systematic the discussion of the program. Finally, the approach to performance measurement during anticipated increase in lending comes with the CAS preparation. The Results-Based CASs risk that the quality of operations may drop. A (RBCASs) provide the strategic framework for regular review of program performance and its managing for results at the country level, implications for overall country results could although the lack of baselines and targets for ameliorate some of this concern. performance indicators makes them less Some countries are piloting this new approach effective as a monitoring, management, and (see box 2.8). For example, the Government of evaluation tool. While the RBCAS is new and is Malawi and the Bank agreed to take a results- only now being mainstreamed, the Bank needs oriented approach to Malawi’s CPPR. While some to focus on strengthening baselines and participants initially focused on implementation targets. More attention needs to be given to issues, the review built an understanding and monitoring performance during CAS imple- awareness of the importance of results. In the mentation. The Bank could work with the Middle East and North Africa Region, the Bank government to: also used the CPPRs to raise awareness of the importance of results. a. Determine the information needed to moni- The CPPR is an established tool, and several tor progress toward country goals and manage country directors who responded to the survey the Bank country program. reported that they use it b. Set targets and establish baselines. An M&E readiness as an occasion to review c. Identify the sources of information. assessment and action progress toward CAS d. Establish responsibility for collection and analy- plan should be made part objectives with the sis of the data. government. The CPPR e. Determine areas where additional capacity of the CAS preparation can be developed into a building and assistance may be required and process. vehicle to conduct regu- addressed through the CAS. 18 M E A S U R I N G A N D M A N A G I N G P E R F O R M A N C E AT T H E C O U N T R Y L E V E L Box 2.8: Incorporating Results in Portfolio Reviews The Malawi Country Program Review and (c) next steps. Steering groups were set up to continue work Preparation of the Country Program Review started in the fall of on the strategy and implementation issues. 2004 as a partnership with government. Both partners wanted to take a results-oriented approach, with the focus on outcomes Results-Focused CPPRs in the Middle East and how the program was (or was not) positioned to support and North Africa Region Malawi in achieving its development goals. Implementation is- In the Bank’s Middle East and North Africa Region, Country sues (such as procurement or M&E) would be looked at to see Portfolio Performance Reviews (CPPRs) in Algeria, Morocco, and how they might be supporting or hindering achievement of the Tunisia focused on enhancing results on the ground by using a goals and priorities. participatory approach in evaluating and strengthening the The review was designed to initiate: (a) a continued dialogue countries’ M&E. The Region moved away from traditional CPPRs on results; (b) a collaborative process to design the next CAS; that focused on inputs and outputs and made outcomes its es- and (c) preparation of the CAS Completion Report. Both parties sential focus. agreed that an interactive, workshop-style format would be ap- The discussions with the government started with development propriate for an open exchange of perspectives between the gov- objectives and included results chains, performance indicators ernment and the Bank and between technical and more senior (measurable, realistic, relevant, precise, with baseline and targets), levels of government. and capacity and risk (readiness of systems in place, who does During the week-long event, the level and depth of discus- what). The Region helped its counterparts achieve a better un- sions around outcomes and results evolved. Initially some partic- derstanding of results-based management and a common under- ipants focused largely on implementation bottlenecks. As the standing of each operation’s results matrix and performance focus on results caught on, the participants engaged in strategic indicators, along with the extent to which objectives are related discussion on outcomes, although some found the change in mind- to the overall country’s goals, on which CASs are based. set challenging. The CPPRs went well beyond diagnosis and reached agree- The event proved an excellent forum for the government to ex- ments on action plans for sound M&E systems. To ensure sus- press some concerns over how to translate the president’s vision tainability of the efforts, the Region is working with the governments into operational programs—while balancing the growth strategy of Algeria and Morocco on building permanent in-country ca- with social policies. Senior and technical government officials pacity in results-based management and in helping the countries and Bank staff exchanged views and offered solutions on (a) the link national strategies to budget process and public expenditure alignment of Bank support to the government’s development pri- management through the use of Institutional Development Fund orities; (b) identification of key institutional implementation issues; grants (box 2.6). Source: World Bank documentation. An M&E readiness assessment and action ment, but the information has to be useful for plan should be made part of the CAS prepara- both Bank and government managers for tion process. learning and decision making, as well as for The government and the Bank will be jointly reporting. accountable for results. Bank managers are The Bank should becoming increasingly focused on managing for expand the CPPR to The Bank should expand results, yet external pressure for the Bank to include AAA and place the CPPR to include AAA report on its results would most likely lead to greater emphasis on and place greater increasing emphasis on accountability for project and AAA outcomes results. While Bank managers will continue to be and their linkages to CAS emphasis on project and accountable for managing resources and deliver- and country development AAA outcomes and their ing quality products to the country, the govern- goals, in addition to linkages to CAS and ment and the Bank will be jointly accountable implementation issues. for outcomes. Performance information is This would allow the Bank country development essential for effective results-based manage- and the government to goals. 19 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N focus discussion on the performance of the Bank’s tool for the government and the Bank to jointly overall program and contribution to achieving manage performance and would operationalize the country development and Bank CAS objectives. An links between the CAS outcomes and project or expanded results-focused CPPR would become the sector-level outputs and outcomes. 20 Chapter 3: Evaluation Highlights • The Bank continues to integrate a results focus in monitoring and eval- uation systems. • Country-level monitoring for results is gaining momentum. • Monitoring of sector-level results is not as far along, but progress is being made. • New lending guidelines and templates have been issued that require more systematic collection of data for monitoring of lending operations. • Monitoring of analytic and advisory activity remains a challenge. • Corporate-level monitoring is gaining ground. • Country-level evaluation is becoming more developed and progress in sector-level evaluation needs to be deepened. • Trust fund and global program evaluation are not fully effective. 3 Update on Monitoring and Evaluation of Bank Operations T his chapter updates the 2003 and 2004 AROE on the status of the Bank’s work in revising and introducing guidelines and templates for incorporating a results focus in the Bank’s M&E of its products at the country, sector, and instrument levels. It summarizes the key observations and recommendations from the previous reports and highlights steps taken to con- tinue to integrate a results focus in M&E systems in the Bank. Monitoring of Bank Operations between country teams and clients (World Bank, OPCS 2005a) to include a greater focus Country-Level Monitoring for Results Is on results. As discussed in greater detail in Gaining Momentum Chapter 2, performance indicators still need to The CAS presents the Bank’s proposed be better baselined and targeted. The Results- program of lending and nonlending activities Based CAS can be further enhanced by taking a to support the country’s development vision systematic approach to identifying, prioritizing, and includes an M&E framework to gauge the and ensuring collection of country information results of the Bank’s performance. The 2003 as part of the CAS preparation process. AROE noted that most CASs had not specified The 2003 AROE noted that the CPPR was to clear outcomes for the Bank, linked Bank be replaced by the Country Portfolio Review. interventions to expected outcomes, or The intent of the new Country Portfolio Review defined performance indicators. was to facilitate a more dynamic and integrative The Results-Based CASs address some of the process to link the CAS with Bank projects, as concerns raised in the 2003 AROE. These CASs well as selected AAA and grants and guarantees. include results frameworks that link country The Bank did not formally revise its policies on outcomes, CAS outcomes, and Bank interven- CPPRs to incorporate the new approach tions and results matrices that summarize these because it was looking to reduce the number of results frameworks with performance indica- formal policies as part of the simplification tors for M&E. A stocktaking report by OPCS of agenda. the seven Results-Based CAS pilots suggests With decentralization, three-quarters of that these CASs might be changing the content country directors are in the field, and country of conversations within country teams, and managers or portfolio coordinators with portfo- 23 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N lio responsibility are in many country offices, demand to manage for results in client and country portfolio discussions tend to be countries. For the next SSIU, it would be useful more continuous. As discussed in Chapter 2, to report on progress in the collection of data several country directors are already exploring for sectoral indicators and progress in strength- new CPPR approaches to incorporate AAA and ening the linkages between the indicators and are making the results link explicit. It would be outcomes. possible and desirable to build on the experi- Also, tensions between country priorities ence to date, enhance the results focus of the and global sector strategies need to be CPPR, develop demand for performance reconciled. Global sector objectives that are information, and use the CPPR to jointly manage reflected in country-level sector strategies need the Bank program with the government. to bear a relationship to country-level interests and feasibility. Sector-Level Results Monitoring Is Not as Far Along, but Progress Is Being Made Lending Guidelines and Templates to Enhance The 2003 AROE noted that the framework for Monitoring Are Too New for Their Impact To Be monitoring outcomes of the Bank’s sector Determined strategies and programs had been evolving, but The 2003 AROE noted that the framework for had not advanced as far as it had for country monitoring investment projects is well programs and specific instruments. Sector established, and the 2004 AROE pointed out Strategy Papers lacked performance measures that M&E has received the greatest attention in for M&E that included baselines and targets guidelines at this level. The AROEs pointed out over a specified period. two areas of weakness: establishing outcome Since the 2003 AROE, the Implementation indicators and monitoring and reporting on Follow-Up (IFU) exercise has resulted in seven implementation progress. Since then, the Bank sectors with results frameworks and outcome has taken several steps to strengthen the M&E indicators for annual progress monitoring.1 focus of its lending instruments by requiring This pilot approach will now be expanded to more systematic data collection and manage- other sectors. While most of these frameworks ment for monitoring its lending operations and need further development, they provide a tracking implementation. starting point for Bank staff as they develop For investment lending, the Project country results chains that link country Appraisal Document (PAD), which develops development goals with specific Bank the rationale for the proposed investment operations. A key challenge will be adapting a loans, was revised in 2004 to increase the generic framework to country circumstances. focus on results-oriented project design and The Sector Strategy Implementation Update monitoring. Instead of a logframe, the PAD (SSIU) of February 2005 is a new report that now includes an annex on the M&E of replaces individual Sector Strategy Paper outcome results and a results framework that updates. It documents progress in all the includes the project development objectives sectoral (and thematic) strategies. It anticipates and indicators, to measure progress toward that the Results-Based CASs will improve the meeting them. The indicators are accompa- Bank’s internal capacity to collect and report nied by baseline and target values. The PAD on sector-level results and will highlight format requires that arrangements for intermediate and final outcome indicators. monitoring and data collection be noted, The initial SSIU exercise highlighted two along with how local capacity for data collec- challenges: The first is developing and improv- tion will be supported where needed. Task ing results frameworks for each sector and teams now need to be more precise in indicat- theme and integrating these frameworks into ing how and where the performance informa- the Bank’s operational work. The second tion will be collected and who will be challenge is strengthening the capacity and responsible for preparing the data. 24 U P D AT E O N M O N I T O R I N G A N D E VA L U AT I O N O F B A N K O P E R AT I O N S A desk review of 28 randomly selected PADs on results frameworks and whether they have submitted to the Board in the first three been strengthened. quarters of fiscal year 2005 found that monitor- With regard to tracking implementation ing frameworks are being improved with the progress, the 2003 AROE noted that the Project new format. Of the 28, 24 used the new format, Status Report (PSR) was not fully effective in while 4 used the logframe. The majority (21 of reporting on implementation performance the 24) included targets for most of their during supervision. Few line managers used indicators, but baselines continue to be a the PSR as a management tool because it did concern. One-third (8 of the 24) had not report adequately on project outcomes. In inadequate baseline data. Often the baseline 2005, a QAG report pointed out that the data was “to be determined.” At the same time, one underpinning portfolio monitoring indicators of the PADs using the logframe included continued to be hampered by the absence of detailed performance indicators broken down candor and accuracy in project performance by year, with specific amounts. While the reporting (World Bank 2005d). change in the PAD addresses the concerns To provide better-quality reporting on the raised in the previous AROEs, the key will be status of project implementation and ensuring that the frameworks are actually used outcomes, the PSR was replaced in fiscal 2005 during project implementation to improve by an Implementation Status and Results monitoring and implementation. Report (ISR). This tool strengthens and stream- A new policy on development policy lending lines task team reporting to managers on the was issued in August 2004 that emphasizes the implementation performance of projects and results framework in development policy their prospective outcomes. Sector managers lending and tightens the links between are required to approve or disapprove updated program objectives, reform measures, and ISRs. The ISR was introduced in January 2005, outcomes. Information on good practice was and after June 30, 2005, an ISR was requested issued in June 2004 to provide staff with for every active project. guidance in preparing development policy The success of the ISR will depend on the operations. An internal good practice note on degree to which it addresses the challenges results measurement and monitoring in encountered with the PSR, which included development policy was issued in June 2005. weak incentives for its use as a management The program document highlights the country tool. To encourage more focus on, and realism program being supported, the objectives of the in, project supervision, portfolio oversight will operation, and the specific results expected be included in the results agreements of from the policy program being supported by country and sector managers. development policy lending. Measurable indicators are required that will serve as the Monitoring AAA Remains a Challenge basis for monitoring progress during The 2003 AROE noted that the monitoring implementation, as well as for evaluating the framework for nonlending operations is program on completion. relatively weak and that measurable outcomes A desk review of 10 program documents and performance indicators are rarely set. It submitted to the Board during the first three recommended that management strengthen quarters of fiscal 2005 found that the policy the outcome orientation and evaluation of the matrices included the expected results of the Bank’s nonlending services at the aggregate loan credit. Development policy lending seeks and task level. The 2004 AROE reiterated the to change practices within governments, and difficulty in applying results-based M&E and the impact may be difficult to observe, and the Bank’s accountability for results to the therefore challenging to monitor and quantify. majority of its AAA products. More experience with the new policy will be An OPCS review of progress on AAA in 2004 needed to understand the effect of its emphasis led to a simple framework to improve the 25 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N accuracy of tracking the results of economic coherence was rated at 78 percent satisfactory. and sector work (ESW) and technical assistance AAA has greater impact when the tasks have (TA) activities and to link development strategic relevance and are integrated, owned, objectives more closely to actual development and supported by the country and are charac- results. Beginning July 1, 2004, task team terized by dialogue and dissemination. leaders were required to define at the AAA can be better incorporated into the beginning of the ESW and TA activities the management of the Bank’s country programs. desired interim outcomes and to identify The Annual Report on Portfolio Performance monitorable indicators of success. for fiscal 2004 recommended that Regions The task-level tracking is a positive step for consider including AAA in the regular CPPRs monitoring. It provides an important, albeit (World Bank 2005c). It also suggested that AAA partial, perspective on monitoring and impact. reporting be balanced away from the individual The effectiveness of ESW and TA tasks is likely task and toward the country level. A similar to be strengthened when the tasks are coordi- theme is noted by OPCS regarding AAA, with nated to have impact in aggregate and are the suggestion that the CAS Completion Report strategically and operationally linked with examine the contribution of AAA to the success complementary activities. While the impact of of the overall country program, a recommen- an individual task may be quite limited, the dation that has been implemented. These steps combination of strategically structured tasks would help to strengthen country management could be significant. Designing ESW and TA oversight of the AAA program. accordingly and identifying outcomes from strategically focused work can come from Trust Funds. The 2003 AROE noted inadequacies strengthened country management oversight in SAP (Systems, Applications, and Products in of the overall ESW and TA program. It would Data Processing – the Bank’s budget and cost also help to address the challenge of managing monitoring software) reporting on the use of AAA in a matrix environment and allow for trust funds and that VPUs did not have the better coordination across sectors, themes, and ability to properly monitor the use of trust management units. funds and their impact on Bank activities and The 2003 AROE noted QAG’s pilot program country outcomes. The report recommended in the use of a country-level approach to assess- monitoring and reporting on the use of ing AAA quality. In the pilot, which began in performance indicators for trust funds that are 2002, QAG chose a sample of all ESW tasks linked with the country activity they support. during the CAS period in three countries and The Bank has made a number of changes to reviewed the quality of individual tasks and improve the monitoring and reporting on trust their contribution to the CAS objectives and funds. A new grant-monitoring tool requires country program. QAG has mainstreamed the reporting on progress in outputs, implementa- pilot and has conducted 36 country AAA tion, procurement goals, and disbursement reviews, comprising 289 individual tasks. The targets. The Initiating Brief for Trust Funds country AAA assessments have found that while Form was revised in February 2004. Ongoing task-by-task assessments resulted in quality work is needed. An internal review of the trust ratings of 90 percent or fund portfolio for 2004 concluded that little more satisfactory, the progress has been made in developing AAA has greater impact strategic relevance of standard performance indicators to measure when the tasks have the entire program has the results and development impact associated strategic relevance and been judged satisfactory with a trust fund–supported program. in about 86 percent of The 2003 AROE noted that OP/BP 14.40 was are integrated, owned, countries, and the under revision. The content of the revisions is and supported by the quality of follow-up set out in the July 2004 Board presentation on country. activities and program trust funds, and the focus of the revisions is 26 U P D AT E O N M O N I T O R I N G A N D E VA L U AT I O N O F B A N K O P E R AT I O N S noted as primarily addressing nationality project level, four IDA14 established a restrictions on procurement (World Bank indicators will be 2 results-based framework 2004c). Strengthening M&E for trust-funded tracked. Finally, output activities remains an unfinished agenda. data from completed that will be implemented projects in four sectors in part through the Corporate-Level Monitoring Is Gaining Ground (health, education, Results-Based CAS. water supply, and rural Millennium Development Goals. Since the Millen- road transport) will be obtained from project nium Declaration was adopted five years ago, completion reports and aggregated. the international community has been engaged The success of the IDA14 results measure- in ongoing stocktaking of MDG implementa- ment framework will depend on the quality of tion and how progress toward achieving the the data for measuring progress on key indica- MDGs can be accelerated. The urgency of this tors. An earlier review of IDA indicators found task is growing as the 2015 deadline for achiev- that only a small number have a well- ing the MDGs approaches. The Bank is established source of data (World Bank pursuing an agenda that includes sharpening 2004d). 3 At the other end of the spectrum, its emphasis on development results, as noted there are no trend data for other indicators. In in the Global Monitoring Report 2005 (World between, data for a sizable number of indica- Bank 2005e). Two areas of priority for the Bank tors come from surveys that do not provide a and development partners are highlighted in consistent supply of data. In the short term, the report: managing for results and measuring progress is needed to strengthen the data for results. Key themes emphasize building sources necessary to track progress on the capacity in countries to help them manage for IDA14 outcome indicators. For the long term, results through the use of independent and country capacity building, discussed in Chapter self-evaluations. 2, will be required. IDA14. The Results-Based CAS is one element of Strategy and Performance Contracts. Strategy and the IDA14 results measurement framework, Performance Contracts (SPCs) are part of the along with efforts to build results measurement Corporate Budget Reform and are intended to capacity in client countries. The IDA14 results support a new type of dialogue between vice measurement framework has two tiers. The presidential units (VPUs) and senior manage- first tier focuses on monitoring country ment, and between senior management and outcomes. It includes 14 country outcome the Board. The intent is to strengthen account- indicators that are consistent with the MDGs, ability for choices on strategy and resources PRSP priorities, and IDA’s activities in IDA and focus on key performance indicators. The countries. In addition, all new IDA operations SPCs build on the previous VPU Unit Compacts in health, education, water supply, and rural and introduce key performance indicators that road transport will, beginning in fiscal 2006, will be used to monitor implementation. This report on and strengthen country capacity to approach to allocating administrative resources enable monitoring of key sector outcome across Bank units was introduced in fiscal 2005. indicators included in the IDA14 results The SPC raises the question as to its link measurement framework. with the results framework, where the country The second tier of the IDA14 results is the focus. In the SPCs, measurement framework monitors IDA’s the Regions and the The corporate budgeting contribution to country outcomes. At the networks are the focal and resource allocation country level, the introduction of Results- points. It will be im- process can provide Based CASs in IDA countries will be monitored portant to explain and targeted to ensure progress in incorporat- operationally how per- incentives to advance the ing country-level results frameworks. At the formance measurement results agenda. 27 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N in the CASs will be aggregated at the Regional achievements and limitations of the Bank’s and network levels. For IDA countries, the country program, usually covering up to three IDA14 results framework builds a bridge CAS periods, and they offer lessons learned for between individual countries and an aggregate the preparation of the upcoming CAS. Since of countries. Guidance will be needed on how the 2003 AROE, IEG has conducted a self- this process will take place for other types of evaluation of CAEs (IEG 2005a). It noted that countries. A balance also will be needed the CAEs yield a more complete picture of the between standardizing performance measure- outcome of the Bank’s assistance programs ment and accounting for variable country than do individual project evaluations. circumstances. The budgeting and resource In one-third of the country programs allocation process will provide incentives that evaluated, aggregate outcomes from projects can advance the results agenda. As the budget were satisfactory, but the overall country reform process proceeds, it will be important assistance program was unsuccessful. The CAE for these to be aligned to further deepen the Retrospective also concluded that CAEs place operationalization of the results framework. too much emphasis on Bank instruments and formal compliance issues and tended to Results Reporting System. OPCS is preparing a neglect the Bank’s development impact and concept paper on a Results Reporting System the results achieved. A more results-based (RRS) that is intended to reinforce the Bank’s approach is needed that links the Bank’s focus on helping its clients manage for results; objectives, instruments, and the outcomes build on existing systems; and harmonize, achieved. The recommended approach for simplify, and streamline many different forms of future CAEs is that they outline key objectives reporting on results. This is a first step. RRS is and results the Bank has been trying to achieve expected to answer such questions as whether based on the CASs, assess the relevance and countries, sectors, and global programs are efficacy of the instruments used to achieve achieving outcomes and whether the Bank is each objective, and evaluate the Bank’s contri- contributing to these outcomes. The RRS will bution to the objective. In this way, CAEs will draw out linkages with the IDA14 results evaluate Bank impact at the country level. measurement system, the reformulation of the The CAS Completion Report (CCR) assesses SPC, and other recent efforts to sharpen the outcomes achieved relative to expectations in results focus in other key business processes. the four-year CAS time horizon. It addresses The simplification process that the RRS may whether the previous CAS included an M&E engender could be an important counterbalance system and performance indicators and to a proliferation of multiple reporting systems. whether information from the system was properly factored into the discussion on the Evaluation achievement of specific CAS outcomes. The Bank evaluates its operations through self- IEG conducts a review of the CCR. IEG’s evaluation by the units responsible for review is an intermediate product, with less in- operational programs and activities and by QAG depth coverage than a CAE. IEG draws evalua- and OPCS. Independent evaluation is carried tive judgments to independently validate major out by IEG. The 2003 AROE recommended that CCR findings. IEG’s criteria for exemplary CCRs management issue an OP/BP that sets out the include a strong focus on M&E. IEG delivered mandate, framework, and roles and responsibil- 11 CCR reviews in the last 2 months of fiscal ities for self- and independent evaluation. This 2005. The timing of IEG’s delivery depends on OP/BP remains to be issued. the CAS schedule. Country-Level Evaluation Is Developing The Sector Level Is Getting More Attention, but Country Assistance Evaluations provide the Progress Needs to Be Deepened Board with an independent assessment of the The 2003 AROE reported that there is no 28 U P D AT E O N M O N I T O R I N G A N D E VA L U AT I O N O F B A N K O P E R AT I O N S framework for self-evaluation of sector share of actual loan or IEG and management outcomes. In the past, IEG has provided credit disbursements will need to develop a independent evaluations of Bank experience in made in the periods numerous sectors, timed to serve as inputs to a before and after approval formal framework and new or updated sector strategy. IEG and of the revision. These timeline for evaluating management will need to develop a formal guidelines contribute to the outcomes of sector framework and timeline for evaluating the strengthening the self- outcomes of sector strategies. evaluations and independ- strategies. ent evaluations of Bank projects. Efforts Are Under Way to Strengthen Lending Greater focus is being placed on impact Evaluation evaluations. The Development Impact Evalua- The 2003 AROE noted that a constraint to tion Initiative (DIME), a pilot initiative led by the evaluation of lending operations is that the Development Economics Vice Presidency (DEC), design of project completion reporting has not is intended to address gaps in the Bank’s kept pace with the evolution of lending instru- conduct of project impact evaluations and to ments. It was recommended that the special strengthen the develop- Greater focus is being features of Development Policy Loans, ment community’s glo- Adaptable Program Loans, and Learning and bal learning. DIME is placed on impact Innovation Loans be reflected in the guidelines supporting systematic evaluations, which are for their Implementation Completion Reports evaluation of programs expected to provide (ICRs). in a selected number of Since the 2003 AROE, OPCS and IEG have strategic priority themes information on what provided guidance on simplified ICRs for in the Human Develop- works in different programmatic development policy lending ment, Infrastructure, and country contexts. (DPL). This was in response to concerns raised Environmentally and by operational managers and staff who Socially Sustainable Development (ESSD) questioned the requirement to prepare an ICR Networks. within six months of closing programmatic As an example, impact evaluations of land adjustment loans, since individual loans are reform programs in Brazil, Colombia, India, linked, by design, to produce results at the end Mexico, and South Africa will be undertaken to of the series. In consultation with operational determine the effect of the programs on the staff, OPCS and IEG agreed on guidelines for productivity of land use and on the welfare of simplified ICRs that apply to each operation in a affected populations. In addition, DIME will programmatic series, except the last. The support evaluation studies focused on specific simplified ICRs meet the requirements in countrywide economic policies, such as the Operations Policy/Bank Procedure 13.55 on poverty impact of reforms on infrastructure Implementation Completion Reporting. They policy and trade. also reduce possibly duplicative work and meet This initiative may prove useful for support- reasonable information and self-evaluation ing and strengthening M&E in the Bank and requirements. A standard ICR that encompasses can be expected to provide information that the entire programmatic series is required after could be valuable to borrowers and others in the last operation. the development community who want to In January 2005 new joint OPCS/IEG know what works in different country contexts. guidelines were issued for evaluating projects DIME also has the potential to improve the with formally revised objectives. They apply to identification of monitoring indicators of both ICRs and IEG assessments and seek to intermediate outcomes. harmonize portfolio monitoring and rating IEG has completed two impact evaluations methodologies. The guidelines require that in partnership with DFID: the Ghana Basic outcome ratings be weighted in proportion to the Education Program and the Bangladesh 29 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N Maternal and Child Health Program. An impact stronger M&E of development outcomes for evaluation of selected India Rural Poverty trust-funded activities. programs is ongoing. These ex-post impact Grants complement the Bank’s lending and evaluations use advanced quantitative evalua- nonlending operations. Grant Completion tion techniques (based on a quasi-experimen- Reports (GCRs) are required only when a tal design) and demonstrate that impact program exits from DGF funding, a process evaluations can be undertaken even when that can take several years. Most of the evalua- baseline data are not available. tions of grant programs over $300,000 are done by external consultants paid by program Trust Fund and Global Program Evaluation Are management, and are not regarded as truly Not Fully Effective independent. The 2003 AROE noted that the evaluation In response to IEG’s Phase I report on global framework for trust funds was partial, focused partnerships and programs (IEG 2002), the on compliance with Bank policy and donor Bank has implemented organizational and conditionality, and needed to be strengthened. procedural changes in global program manage- It suggested that management set guidelines ment. The DGF has instituted an external peer for independent evaluations of large trust review process for new programs seeking grant funds because this would be consistent with support. In response to IEG’s meta-evaluation the existing practice that requires external of the CGIAR (IEG 2004e), Bank management evaluations of Development Grant Facility has accepted the principle of independent (DGF) grants larger than $300,000. The Trust oversight by assigning oversight of the CGIAR Fund Operations (TFO) and Global Programs to the Bank’s chief economist. These actions and Partnerships (GPP) group are currently in are intended to strengthen the Bank’s global discussions with IEG on ways to advance M&E program portfolio. But further improvements for trust funds and GPPs. are needed to enhance the linkages between An Implementation Completion Memoran- global programs and country operations. dum (ICM) is required for all activities in excess As part of the 2004 IEG Phase 2 review of the of $1 million funded under trust funds. The Bank’s involvement in global programs and ICM is a self-assessment instrument that partnerships, IEG reviewed 26 Bank-supported monitors the degree of success of the trust- global programs that accounted for 90 percent funded activities as measured by outputs, of the Bank’s global program expenditures in outcomes, impact, and Bank and recipient 2002 (IEG 2004d). The review recommended performance. The Trust Fund Quality that global programs be included in IEG’s Assurance and Compliance Unit reviewed 74 standard processes of evaluation and reporting ICMs received during fiscal 2004 and found that to the Board. This would involve working with many reports did not identify the impact of the the Bank’s global partners to develop and apply trust -funded activities, but instead identified international standards for the evaluation of inputs and outputs such as workshops and global programs. It also recommended that IEG reports. It was recommended that the Bank review program-level evaluations conducted by and donors initiate a targeted M&E program for Bank-supported global programs, similar to larger country-based or sector-specific trust IEG reviews of other self-evaluations at the fund programs. It further suggested that OPCS project and country levels. and Concessional Finance & Global Partnership In April 2005, IEG issued guidelines for its (CFP) develop standards that defined when Global Program Reviews (GPRs). These reviews M&E is required for trust fund programs, based of global program evaluations are seen as not only on dollar amounts but also on the parallel to existing review processes for the significance of the trust fund program. A evaluation of investment projects and country targeted M&E program would help address the programs, and will therefore be built on 2003 AROE recommendations that called for established models for ICR Reviews and Project 30 U P D AT E O N M O N I T O R I N G A N D E VA L U AT I O N O F B A N K O P E R AT I O N S Performance Assessment Reports (PPARs) for increasingly being seen IEG issued guidelines for investment projects, and CAS Completion as designed to support Report Reviews and CAEs for country programs. the CAS. New CAS its reviews of evaluations The GPR is based on a review of the external progress reports are of global programs in evaluation of the global program, and in some expected to provide April 2005. cases on fieldwork conducted by IEG. A key just-in-time learning. difference between GPRs and the reviews of The development and refinement of results projects and country programs is that global frameworks for all sectors as part of the SSIU is programs are partnership programs. Instead of a start in providing Bank staff with tools to being the responsibility of a Bank operational establish the links between country and CAS Region or network, global programs are the objectives and to identify intermediate responsibility of the governing body of the outcomes that can be used to monitor program, and the Bank is only one member, progress. though it often acts as the chair. The revision of the PAD, introduction of the In the context of results orientation and ISRs, consolidation of adjustment lending into accountability for results, it should be noted that DPLs with a stronger emphasis on results, and in cases where the Bank‘s role is that of fiscal introduction of simplified ICRs for develop- agent, the Bank is not accountable for program ment policy lending enable the Bank to obtain results. Global program evaluations are the a clearer picture of the results of its operations. property of the global program, which suggests These steps foster the identification and the need for common standards for global evalua- reporting of clear and monitorable indicators tions. Also, global programs, unlike investment and the tracking of implementation progress. projects, are ongoing rather than completed. While policies and procedures are being put IEG will initiate two pilot GPRs in fiscal 2005 in place, it will take some time before the Bank that will be similar in scope to a PPAR. They will is able to effectively manage for results. Experi- include lessons learned rather than recommen- ence from developed countries, where results- dations. Like PPARs, each review will undergo a based management was introduced in the late multi-tiered review within IEG. For the pilot 1980s and early 1990s, indicates that there is a phase, IEG plans to ask the Board’s Committee significant learning period for an organization as on Development Effectiveness (CODE) to managers and staff expand their focus from review the first two pilots in order to provide managing inputs such as budgets and adminis- feedback to IEG regarding the new product. trative matters to managing outputs, and then outcomes (box 3.1). The challenges in develop- Conclusions ing countries will be more pronounced because The Bank is making progress in strengthening of data and capacity constraints. Bank manage- a results orientation in M&E. Preparation of the ment will need to align CAS Completion Reports, the IEG reviews of incentives to manage for While policies and these reports that are now required for all results. It has taken an procedures are being put CASs, and the mainstreaming of the Results- important step in this Based CAS provide a solid framework for direction by incorporat- in place, it will take time managing for results at the country level. The ing portfolio oversight as before the Bank is able to Bank is rightly moving its focus from instru- an element in the annual effectively manage for ments to the country. reviews for all managers Whereas the CAS has traditionally been built of country and sector results. up from the instrument level, instruments are management units. 31 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N Box 3.1: Lessons on Results-Based Management in the U.S. Federal Government The Government Performance and Results Act of 1993 requires program managers. Many federal programs, such as those that U.S. federal agencies to provide annual data on the outcomes work through other levels of government, require long periods of time of each major federal program. The primary focus has been to before they can be expected to yield the major outcomes sought. strengthen accountability for program results. Although a foundation has been established, there are still Several obstacles to using outcome data have been observed. major challenges in implementing results-oriented government. In- Some managers may not be aware of the variety of ways in which consistent commitment to achieving results among top leadership they can use outcome data, or of which particular data can be can hinder the development of results-oriented cultures in agen- used for specific purposes. Outcome data use is sometimes limited cies. Furthermore, in certain areas, federal managers continue to by data problems. Some data are old by the time they reach program have difficulty setting outcome-oriented goals; collecting useful data managers; data that are two or three years old do not have the cur- on results; and linking institutional, program, unit, and individual rency that program managers usually need to react. Some of the data performance measurement and reward systems. Finally, there is may not be broken out in sufficient detail to be useful to particular an inadequate focus on issues that cut across federal agencies. Sources: Hatry and others 2004, pp. 5 and 13; GAO 2004. 32 Chapter 4: Evaluation Highlights • IEG is requiring evaluations at the country, sector, and project lev- els to give greater attention to results-oriented M&E. • IEG’s Outreach and Communications Strategy is aimed at improving the communication and dissemination of IEG evaluation findings. • IEG client survey results highlight the need for evaluation findings to be made more operational. 4 Improving IEG’s Effectiveness I EG evaluates its own products and services as part of a continuous process of self-evaluation. This chapter discusses IEG’s response to recommen- dations for itself contained in last year’s AROE and the results of the IEG client survey, which assesses Bank staff awareness about IEG studies and their perceptions of quality. IEG’s Response to 2004 AROE outcomes of Bank CAEs will assess the focus Recommendations assistance. The CAEs will of Bank assistance on The 2004 AROE included two recommenda- also examine the degree tions for IEG: to which the Bank’s supporting the country’s assistance program in- adoption of a more Recommendation 1: Strengthen attention to cluded a focus on results-oriented results-oriented M&E in IEG by requiring supporting the country’s evaluations at the country, sector, and efforts to adopt a more framework. project levels to address systematically results-oriented framework—for example, the three criteria for results-oriented through defining outcomes indicators in PRSPs M&E, and increase the weight that IEG or other strategy documents of the country. The ratings give to the extent of results- degree of Bank support provided through loans oriented M&E and managing for results or grants to develop the capacity to improve or in Bank programs. The three criteria for make use of existing systems of data gathering assessing results-oriented M&E are the results and analysis and other monitoring will be noted. framework, the design of M&E, and the Finally, the CAEs will assess the extent to which implementation of M&E. the Bank’s efforts resulted in improvements in At the country level, beginning January 2006, the quality, timeliness, and availability to end CAEs will note the extent to which the CASs have users of monitoring indicators. become results focused, including the definition At the sector level, IEG is increasing its and analysis of a reasonable quantity and quality emphasis on monitoring and evaluation and of indicators, and the extent to which the CASs results frameworks in its sector and thematic and the CCRs made good use of the outcome evaluations. As an example, two major recent indicators to arrive at assessments of the overall studies, of Community Development (IEG 35 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N 2005b) and HIV/AIDS (IEG 2005c), contain in- Communications Strategy (IEG 2005d) is aimed depth analyses and specific recommendations at ensuring that IEG findings reach their target to help improve the quality and quantity of audiences at the right time, in the right form, M&E in the Bank. and in a manner that the recipients can access At the project level, easily. At the project level, IEG is IEG is strengthening the The new strategy encompasses three introducing a separate focus on M&E and approaches: embedding outreach and section in PPARs and ICR results frameworks. communications throughout the evaluation While most IEG PPARs cycle, targeting underserved audiences, and reviews that will discuss and many ICR reviews accelerating learning from evaluation findings. design, implementation, comment, and often IEG will capitalize on these three approaches and use of M&E. include lessons learned, by streamlining internal communications on monitoring and practices through greater consultation, earlier evaluation, IEG is introducing a separate audience identification, and systematic integra- section in these reviews that will discuss tion of communications throughout the evalua- design, implementation, and use of M&E. tion cycle. Monitoring and evaluation is receiving IEG products will be improved through better increased emphasis in IEG assessments. In content and design. Also, the targeting and some cases, inadequate M&E has been an delivery of IEG findings will be enhanced important reason for downgrading Bank through better audience identification, adoption performance. Also at the project level, IEG is of advanced electronic tools, and strategic adopting a results framework approach in its partnerships to promote dissemination. impact evaluations by testing the results Finally, learning and use of evaluation achieved by specific interventions. lessons will be accelerated through increased At the global level, IEG’s recently completed participation in learning events by Bank staff, review of global programs contains a greater sharing of IEG findings with operational recommendation on the evaluation of Bank- teams, and greater use of public forums. Pilot supported global programs. IEG is piloting initiatives are under way to promote more reviews of the Bank’s self-evaluations of effective internal communication approaches individual global programs. and to shift outreach efforts from a supply- Beginning in 2005, IEG instituted an award driven model to one that is demand-driven, to recognize good practice in M&E as part of customized, and timely. the annual IEG Awards Program. The purpose Performance indicators will be used to help of the award is to emphasize the importance of IEG assess the impact of the Outreach and M&E in ensuring the effectiveness of Bank Communications Strategy. In the annual client assistance and to create incentives for good survey, performance indicators include performance measurement and management. awareness, readership, timeliness, and quality. In addition, audience feedback will be tracked Recommendation 2: Com- regularly by monitoring changes in Web usage IEG’s Outreach and municate and dis- as measured by unique monthly visitors. Communications Strategy seminate findings Whether target audiences are being reached and lessons of IEG will be assessed by tracking Web users, the is aimed at ensuring that evaluations more content they visit, and the time they spend. IEG findings reach their broadly, both inside Targeted delivery of IEG products will be target audiences at the and outside the tracked by monitoring the percentage of right time, in the right Bank, in line with evaluation consultees who received the final the new communica- product. External dissemination events such as form, and are easily tions strategy. IEG’s press activities and external briefings will be accessible. new Outreach and monitored for the number of references to IEG 36 IMPROVING IEG’S EFFECTIVENESS products in the media and attendance at aspect as excellent). IEG found that 45 percent briefings by key stakeholders. Executive directors and of respondents were their advisors rated the IEG Client Survey Results evaluations more posi- aware of the relevant As part of IEG’s self-evaluation, it annually tively, with at least 84 evaluation — a marked surveys its clients about the readership and percent good or excellent increase from previous perceptions of its major studies. This year, IEG ratings for each aspect. conducted two surveys. The first focused on years. the Bank’s internal clients and asked 2,120 of Influence of the IEG study. them about IEG products to find out about More than three-quarters (78 percent) of Bank readership and awareness, perceptions of staff respondents indicated that the evaluation quality, the influence of the reports, and they were asked about influenced their view of recommendations for improving IEG products. the subject areas either a great deal or The second asked 926 external clients— somewhat. Among country directors and government officials, researchers, and consult- managers, 92 percent said they would use the ants—similar questions about IEG products. findings of the evaluation in their work. The following section highlights key findings. Executive directors and their advisors (95 Detailed survey results are in Appendix D. percent) indicated that IEG evaluations influenced their view of a subject area, with 40 IEG Internal Client Survey Results percent reporting a great deal of influence. This year’s survey targeted several kinds of IEG products, including eight Country Assistance Respondent recommendations to IEG. Based on the Evaluations, the Annual Review of Develop- results of the survey and recommendations ment Effectiveness, PPARs, and evaluation from respondents, IEG should make the studies. IEG surveyed a targeted sample of findings in the evalua- 2,120 internal clients, including Bank staff tions more operational. IEG needs to make its associated with the products and executive Other areas that need evaluation findings more directors and their advisors. The response rate greater attention in- operational and to better was 31 percent, an increase over the response clude strengthening the rate for the 2004 surveys (21 percent). links in evaluations be- communicate the results tween conclusions and of its work. Readership and awareness. Almost half (45 evidence. Also, IEG percent) of Bank staff who responded to the needs to continue to engage in broader consul- survey had read, participated in a discussion of, tation with Bank staff to improve its evalua- or attended an event on the evaluation for tions. Finally, IEG’s outreach can be enhanced which they were surveyed. This is a marked with better notification regarding new increase over the levels of awareness found in documents and the targeted provision of previous AROE surveys (2004: 27 percent). summaries of IEG findings. IEG’s Outreach and Eighty-five percent of executive directors and Communications Strategy is intended to be their advisors reported that they had read, responsive to these suggestions. Appendix D participated in a discussion of, or attended an has details of the survey results. event for a 2004 evaluation. IEG External Client Quality. Two-thirds or more of Bank staff respon- Survey Results IEG’s client survey found dents rated as good or excellent the depth of This year, in a survey of that two-thirds of analysis (71 percent), presentation of the evidence the Bank’s external respondents were aware (68 percent), presentation of conclusions (73 clients, 926 government percent), and relevance of lessons to the Bank’s officials, consultants, of the evaluation in work (67 percent, with 21 percent rating this and researchers were question. 37 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N asked about one of two IEG products that had stated that they thought the evaluation paper been subject to targeted external dissemina- presented information in a transparent and tion: IEG’s evaluation of Global Programs and understandable manner. Although a majority of Partnerships and IEG reports on M&E instru- all respondents rated each of the quality ments and approaches to capacity building, dimensions positively, the presentation of which had been distributed to International evidence (71 percent good or excellent) and Program for Development Evaluation Training the link between conclusions and evidence (71 (IPDET) alumni.1 percent good or excellent) were dimensions of quality for which ratings were lower than for Readership and awareness. Two-thirds of respon- other dimensions—for example, depth or dents had read, participated in a discussion of, quality of analysis (82 percent good or or attended an event on the evaluation product excellent). for which they were surveyed. The vast majority of external client respondents Influence. Seventy-four percent of respondents surveyed about the M&E reports heard about said they had used or would use the findings the product by e-mail or online, while 53 from the IEG products. About two-thirds of percent of GPP respondents heard about the GPP respondents said that the study influenced evaluation in this way. Half of the GPP respon- their view a great deal or somewhat. Variation dents indicated they had heard about the was noteworthy among respondent groups evaluation by being consulted during the with regard to the influence and use of the IEG evaluation process. products. Eighty-eight percent of IPDET respondents said they had or would use the Quality. A large majority of respondents (94 findings of the IEG evaluation product in their percent) indicated that IEG’s report was timely. work. Eighty percent of GPP respondents said the evaluation made full use of the information Recommendation for IEG available, and 83 percent noted that they As with the internal survey findings, respon- thought the evaluation used a methodology dents to the external survey most often that was transparent and clear. Ninety-two selected the recommendation for IEG to make percent of the IPDET alumni who responded findings more operational. 38 5 Conclusions and Recommendations T he Bank is making progress in strengthening the results focus of its op- erations at the country, sector, and instrument levels. The increasing em- phasis on outcomes will lead to joint accountability for outcomes. As countries assume more responsibility for and sector outputs and outcomes. It would also M&E and performance measurement, they will inform and complement the CAS Progress have to develop both monitoring systems to Report and the CCR. The 2003 AROE noted that capture outcomes and evaluation capacity to management was replacing the CPPR with a process and use the information. This will Country Portfolio Review, but this has not been require continuing Bank efforts to strengthen done. It may now be appropriate to revisit the M&E capacity at the agency and country levels, CPPR as a tool for managing country programs. harmonizing its efforts with those of the country and other development partners. As Recommendations part of CAS preparation, the Bank should systematically assess the country’s capacity to For Management generate, collect, and analyze performance information. It should identify gaps and • Incorporate a diagnosis of the country’s per- support capacity development as needed. formance measurement capacity and recom- This shift will also require governments to mendations for action to strengthen that become more aware of the need for perform- capacity in the CAS. The diagnosis and actions ance information and to develop knowledge of should be customized to country capacity and its uses. Increasing the results focus of CPPRs focused on critical information needed for the and expanding them to include AAA would government to implement its program. allow the Bank and the government to focus • Expand use of the Results-Based CAS as a tool discussion on the performance of the Bank’s to manage country programs and strengthen overall program and its contribution to achiev- the link between the Results-Based CAS and the ing country development and Bank CAS country program by providing country teams objectives. An expanded, results-focused CPPR with guidance designed to increase the results would help link the CAS outcomes to project orientation of CPPRs. 39 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N For IEG the full implementation of the communica- tion strategy. • Review the results of pilot initiatives to promote • Subject to CODE/Board endorsement of the more effective outreach and communication of new procedures, institute a formal system for IEG products and consider lessons learned in tracking and managing IEG recommendations. 40 APPENDIXES APPENDIX A: REVIEW OF CAS RESULTS MATRICES Objectives, Scope, and Methodology tion to be evaluated and were included in the analysis as a baseline or control group for com- Objectives parison with the results-oriented CASs. The objective of the review is to evaluate the ef- fectiveness of the CAS results matrix as a tool for Methodology country-level Bank management. The results The results matrices were examined to assess: (a) matrix is a key element of the Results-Based how the structure and content of the frame- CAS. The matrix separates out and defines coun- work contributed to the Bank’s results focus try development goals, CAS outcomes, Bank in- and (b) how effective the performance measures termediate outcomes, and Bank programs (as would be. columns) for each goal (as rows). By using this A maturity model1 was used to evaluate the matrix, Bank operations are expected to be more matrix framework and the performance meas- explicitly linked to overall country goals, an im- ures. Assessment criteria were established and portant change from the Program Matrix of the defined, and three levels of maturity, from Level previous (traditional) CAS. The review consid- 1 (lowest) to Level 3 (highest), were defined for ered both the usefulness of the format for pre- each criterion. Level 0 describes a CAS matrix that senting the information needed to manage the could not be evaluated in that criterion for lack Bank’s country program and the actual formu- of data. The methodology is presented in At- lation of each matrix for each CAS. A sample re- tachment 2. sults matrix is presented as Attachment 1 to this The approach used to assess the framework appendix. and the measures is described below. Scope • Analysis of the Matrix Framework: Three Twenty-four CASs presented to the Board in fis- criteria were developed to gauge the overall cal 2004 and 2005, as of end-January 2005, were maturity (strength) of the framework: (a) the reviewed. They were categorized as follows: structure of the logical sequence of the matrix (logic chain), (b) the content of the matrix • Six Results-Based CAS Pilots—Armenia, Brazil, (results chain strength), and (c) the rep- Cameroon, Mozambique, Ukraine, and Zambia resentation of partner programs (linkage to • Thirteen Results-Oriented CASs that employed other donors). For the logic chain, the some form of the results-based approach, al- columns and rows of the results matrix were though they were not formal pilots reviewed to determine how effectively links be- • Five traditional CASs that contained sufficient tween country goals, Bank outcomes, and Bank information for analysis, although they fol- operations were portrayed. For results-chain lowed the traditional CAS format. strength, the links between country goals and Bank operations for each goal were examined During this period, nine CASs were presented to determine how easy it was to follow the re- to the Board using the traditional CAS format. Of lationship between the goals, measures, and these, five contained sufficient results informa- programs. 43 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N • Analysis of Performance Measures: The comes to intermediate outcomes and to spe- performance measures in the matrices were as- cific Bank programs. All of the Results-Based sessed on the following criteria: defined, base- CAS pilots and results-oriented CASs were rated lined, targeted, and aligned2 (see Attachment at higher levels of maturity. One traditional CAS, 2 for definitions). These criteria were applied Bolivia, used its program matrix to articulate the to three sets of performance measures: (a) relationship between country goals and Bank country-level measures, (b) CAS outcomes, operations (figure A.1). and (c) intermediate indicators and mile- The Bank’s standard results matrix format stones. Each criterion pertains to qualities of has been a good starting point for many Bank individual measures. In order to determine country teams. Several have adjusted the stan- maturity for the whole set of measures in the dard format to better articulate their country matrix, maturity thresholds were established strategy and the relationship between country based on the proportion of measures that met and CAS goals, and between Bank CAS outcomes each criterion: that is, Level 1 if less than half and operations. of the measures apparently met the criterion; For example, many CAS matrices introduced Level 2 if more than half but less than three- some structure in the rows of the matrix, or- quarters of the measures met the criterion; ganizing the column information into related and Level 3 if more than three-quarters met the sets of goals and programs. This makes it easier criterion. to comprehend the linkages between different country development goals and how they jointly Results of the Matrix Structure contribute to a higher objective, and to identify Evaluation the potential synergies between Bank opera- tions that support individual country develop- Logic Chain ment goals. Not unexpectedly, the results matrix format does The review also identified some weaknesses. a better job of relating country goals to CAS out- A typical problem was the lack of a structure Figure A.1: Logic Chain 16 14 12 10 Number of CASs 8 6 4 2 0 Level 0 Level 1 Level 2 Level 3 Pilots Traditional Results oriented 44 A P P E N D I X A : R E V I E W O F C A S R E S U LT S M AT R I C E S within the columns that made it difficult to fol- traditional CAS format used by Bolivia, which low the logical links between country goals, CAS was evaluated as a Level 3 results chain. In this outcomes, intermediate outcomes, and opera- case, the country team used an enhanced version tions. For example, if one country goal was fol- of the traditional CAS program matrix to draw lowed by 10 CAS outcomes, another 20 clear results relationships between programs intermediate outcomes, and 25 programs, with and Bank and country goals. no explicit or implicit structure, it was difficult Countries with Poverty Reduction Strategy to understand the interactions between actions Papers (PRSPs) tended to have better results and results. Another weakness was a lack of un- chains. The education, health, and environment derstanding of the use and meaning of the ma- sectors were relatively better at creating a com- trix. For example, some CAS reports did not pelling, results-oriented narrative through the include any measures in the CAS outcomes col- contents of the results matrix. Even in countries umn. Others mixed long-, medium-, and short- where the matrix itself was not well developed, term outcomes between the CAS and inter- or where the other sectors were less mature, mediate columns. the programs in these sectors were frequently de- scribed effectively in results-oriented terms. In Results Chain Strength contrast, anti-corruption and governance areas Results-Based CAS pilots did a better job of ar- were often less mature, probably reflecting the ticulating the results chain, allowing the reviewer greater difficulty of establishing clear, incre- to follow the link between Bank operations, in- mental results chains from specific Bank activi- termediate outcomes, CAS outcomes, and coun- ties to country-wide results. try development goals for individual goals. The More CASs (eight) were evaluated at the high- results-oriented CAS formats, especially the pi- est level of maturity, Level 3, for results chains, lots, exhibited higher levels of maturity; more tra- compared with the logic chain criterion (five ditional CAS reports were lower on the maturity CASs). This indicates that some CAS matrices scale (figure A.2). A notable exception was the were able to overcome structural deficiencies in Figure A.2: Results-Chain Strength 16 14 12 Number of CASs 10 8 6 4 2 0 Level 0 Level 1 Level 2 Level 3 Pilots Traditional Results oriented 45 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N the presentation of the results matrix and to Matrix” (with strategic areas as columns and the show the logical relationship of country goals, different development agencies as rows) show- Bank outcomes, and Bank activities. Typically ing the estimated weighted effects of partner this is due to a well-developed underlying results activities on country goals. chain, representing a mature understanding of the relationships between activities and results Results of Performance Measure in particular programmatic areas. Evaluation Linkage to Other Donors Results-Oriented Formats More Mature The CAS matrices generally did a poorer job of than Traditional Formats showing the interaction between Bank programs The review of the performance measures found and those of other partners. There were virtu- that those in results-oriented CAS formats, es- ally no differences between the Results-Based pecially the pilot countries, generally were bet- CAS formats and the traditional formats (figure ter developed than those in traditional formats. A.3). Although not statistically significant, the Examining the measures across the three levels traditional CAS formats were more likely to have (country, CAS, and intermediate), countries with higher levels of maturity under this criterion. PRSPs tended to have more mature country- The format of the matrix itself is a constraint. A level performance measures. Some traditional two-dimensional matrix is limited in space, which CAS reports are rated Level 2 and 3 at the coun- may make it difficult to provide sufficient infor- try level for this reason. mation on partner activities alongside effective Performance measures were generally well descriptions of Bank results chains. Several CASs defined: that is, a measure is clearly identifiable used different approaches to display the rela- as a performance measure as distinct from a tionships with partners. For example, the goal or activity. Most CASs demonstrated high ma- Paraguay CAS included a separate “Partnership turity for all three levels of measures, with a Figure A.3: Linkage to Other Donors 16 14 12 Number of CASs 10 8 6 4 2 0 Level 0 Level 1 Level 2 Level 3 Pilots Traditional Results oriented 46 A P P E N D I X A : R E V I E W O F C A S R E S U LT S M AT R I C E S large number of CASs (19, or 79 percent) rated Maturity Level 0 typically indicates that there as Level 2 or 3 for country performance meas- were no measures at all, or at least no appropriate ures (figures A.4, A.5, and A.6). Part of the ex- measures at the CAS or intermediate levels. planation is the use of widely accepted measures at the country level, such as the Millennium De- Difficulties in Developing Bank Outcome velopment Goals. Measures Similar patterns were noted for the alignment A comparison of measures across the three lev- of measures. The measures generally were pre- els—country, CAS, and intermediate – indicated sented at the right levels in the results chain that the Bank finds it more difficult to develop (figures A.7, A.8, and A.9). For example, long- measures at the CAS and intermediate levels. In term, national health, or economic measures, many cases, country measures were aligned, and such as growth of GDP or reduction in national had baselines and targets, but CAS and inter- infant mortality rates, would be appropriately mediate measures were missing these qualities. aligned in the country level, but misaligned at the Since the general trends were comparable across intermediate level. The PRSP and MDGs served the three criteria (aligned, baselined, and tar- as a good starting point for measurement at the geted), the results of these assessments were ag- country level for both results-oriented and tra- gregated. When all CAS reports are summed ditional CAS formats, but are often out of place across these three measurement criteria, there at lower levels of the results chain. were 21 cases of measures at Level 0 at the CAS There is a fall-off in alignment maturity level and 19 at the intermediate level, but only through the CAS and intermediate levels. Typi- 7 at the country level (figure A.10). Conversely, cally, this indicates that a CAS report, while pos- there were 27 cases of Level 3 measures at the sessing the appropriate measures for the country country level, 10 more than at the intermediate level, inappropriately relied on country-level level and 11 more than at the CAS level. The measures in the CAS and intermediate levels. higher overall levels of maturity for country-level Figure A.4: Measurement Definition: Country Level 16 14 12 Number of CASs 10 8 6 4 2 0 Level 0 Level 1 Level 2 Level 3 Pilots Traditional Results oriented 47 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N Figure A.5: Measurement Definition: CAS Level 16 14 12 Number of CASs 10 8 6 4 2 0 Level 0 Level 1 Level 2 Level 3 Pilots Traditional Results oriented Figure A.6: Measurement Definition: Intermediate Level 16 14 12 Number of CASs 10 8 6 4 2 0 Level 0 Level 1 Level 2 Level 3 Pilots Traditional Results oriented 48 A P P E N D I X A : R E V I E W O F C A S R E S U LT S M AT R I C E S Figure A.7: Measurement Alignment: Country Level 16 14 12 Number of CASs 10 8 6 4 2 0 Level 0 Level 1 Level 2 Level 3 Pilots Traditional Results oriented Figure A.8: Measurement Alignment: CAS Level 16 14 12 Number of CASs 10 8 6 4 2 0 Level 0 Level 1 Level 2 Level 3 Pilots Traditional Results oriented 49 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N Figure A.9: Measurement Alignment: Intermediate Level 16 14 12 Number of CASs 10 8 6 4 2 0 Level 0 Level 1 Level 2 Level 3 Pilots Traditional Results oriented measures reinforces the idea that the country- ate levels, but were also prominent at the coun- level measures, typically founded on MDG or try level. PRSP measures, are more developed, while the The pattern was generally consistent across measures at the CAS and intermediate levels are country, CAS, and intermediate levels, although relatively underdeveloped. the differences were greater at the CAS and in- termediate levels, as noted above. The evaluations Problems in Baselines and Targets Point for targeting, baselining, and definition for all to Difficulties in Data three levels were added up and compared by Performance measures were typically well de- maturity level (figure A.11). Measures were bet- fined, that is – most measures did a good job of ter targeted, with close to three-quarters (73 per- specifying the evidence that would allow the cent) assessed at Level 2 and 3. The opposite is Bank to understand progress (as discussed in the true of baselining and targeting. Measures were Logic Chain section, above). However, meas- slightly more likely to have targets than base- ures across the sample were much less likely to lines, but this area also demonstrated a lack of ma- have baselines, an indication either that current turity relative to the definition of the measures. data do not exist or that the data were not avail- able to the Bank when the report was finalized. The Number of Measures Also, many measures lacked targets. For exam- Results matrices often contained a large number ple, “Increase living standard of working popu- of performance measures, raising doubts about lation measured as increase in real salaries” is a the usefulness of many measures as a manage- well-defined measure because it indicates specif- ment tool. The average number of measures (ex- ically what evidence should be used to determine cluding the traditional CASs) was 22 for country the desired effect. But it lacks both a baseline and development, 39 for CAS outcomes, and 40 for a defined target. Measures such as this were Bank intermediate measures for the CASs reviewed more typically found at the CAS and intermedi- for this report (table A.1). The number of measures 50 A P P E N D I X A : R E V I E W O F C A S R E S U LT S M AT R I C E S Figure A.10: Comparison by Type of Measure 30 25 Total number of CAS cases 20 15 10 5 0 Country CAS Intermediate Maturity level for aligned, baselined, and targeted criteria Level 0 Level 1 Level 2 Level 3 Figure A.11: Evidence of a Data Gap 18% Level 3 13% 54% 18% Level 2 15% Evaluation level 19% 40% Level 1 44% 11% 24% Level 0 28% 15% 0 10 20 30 40 50 60 Percentage of cases Targeted Baselined Defined 51 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N should reflect country program needs, so that Table A.1: Number of Performance Measures by Type large and complex programs may require a large number of measures. While there are no criteria Average Low High for a priori determining a maximum, a large num- Country 22 7 38 ber of measures would tax the government’s abil- CAS outcomes 39 15 80 ity to collect and analyze the information. Some CASs have addressed this by showing a large num- Intermediate outcomes 40 14 97 ber of measures in their results matrices but pre- senting a smaller number of core measures in the main text of the CAS. Attachment A.1: CAS Results Matrix—Definitions and Excerpt from Cameroon CAS Country development goals Issues and obstacles Definition: Longer-term or higher-order development objectives Definition: Critical issues and obstacles to achieving country identified by the country. Usually not achievable in the CAS development goals, providing the logical link to CAS outcomes. period or based solely on the CAS program. Only those to which CAS outcomes will contribute are included. Reduction in rural poverty Determinants of rural poverty • Decrease in the poverty incidence in rural areas from 49.9% (2001) • Lack of education among heads of households is a key • Maternal mortality rates decrease from 430 per 100,000 (1998) determinant of rural poverty disaggregated by rural area • Relative isolation (enclavement) due to a lack of physical • Child mortality decreases from 77 per 1,000 (1998) disaggregated infrastructure (mainly roads) and no adequate support for by rural area road maintenance • 100% primary school completion rates by 2015, from 56% (2001) • Isolation from social services (health and education disaggregated by rural area especially in rural areas) • Low level of coordination between ministries in charge of rural water, and low local capacity and skills for water supply infrastructure • 20% of the poor have no financial social coverage Source: World Bank 2005a. 52 A P P E N D I X A : R E V I E W O F C A S R E S U LT S M AT R I C E S CAS outcomes Milestones Bank program (and partners) Definition: Results achievable in the CAS period and Definition: Progress markers of Definition: Ongoing and planned lending, that the Bank expects to influence through its implementation of the Bank’s grants, and guarantees; analytical and ad- interventions. Indicators of each outcome are included, program; outputs, actions, or visory activities. Includes IBRD, IDA, IFC, with baselines and targets. outcomes expected to be realized MIGA. Partners included if cofinancing or during CAS implementation. other support of same CAS outcome. Improved delivery of basic services (sustainable • 50% of the 4,000 rural Lending - Ongoing management of service delivery at the communities targeted have fully Ag Serv. and Ext. (PNVRA) ; community level for rural communities in four implemented one micro-project Cameroon Petroleum Environment Capac- targeted provinces): (roads, schools, ity Enhancement (CAPECE) ; • Percent of rural communities in the four provinces hospitals, etc.) by 2006 Douala Infrastructure with access to basic socioeconomic services • Over 4,000 rural community-based Lending - Proposed (baseline in fiscal 2004), including: action plans designed and imple- FY04 : PNDP (community development). – Increase in access to water (baseline: 58%) mented between 2003 and FY06 : PRSC - Rural development – Decrease in time to get to school. end 2005 AAA - Ongoing – Increased use of rural health care clinics • Increased involvement of women Expenditure tracking and beneficiary as- • Percent of communities in the four provinces that and vulnerable groups in decision sessments (FY03) have functioning elected committees making, definition of priority action AAA - Proposed plans, and implementation FY04 : Health Sector Reform Note; oversight Gender strategy (IDF) FY06 : Poverty Assessment PARTNERS Cofinancing of PNDP by AFD, IFAD, KfW and Netherlands cooperation 53 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N Attachment A.2: Detailed Maturity Model and Evaluation Criteria Criterion Definition Logic chain The results matrix schematic structure effectively portrays the chain of causal assertions connecting country and Bank goals and activities. This would be expressed in a results chain that is the cascade of causal assertions between outcomes (country, medium, and short term), outputs, activities, and inputs. The results chain should be visible in the results matrix such that the links between country goals, Bank goals, and Bank activities are clear. Results chain strength The causal assertions made in the results chain are logically related and sensibly support country and Bank goals. The goals and activities of the Bank should logically connect to and support the goals of the country. Activities should be sensibly aligned to goals that they will affect. Linkage to other donors The activities of other donors and lenders, to the extent that those activities are reflected in the text of the CAS, should be referenced in the appropriate place in the results matrix. Defined The measure is clearly identifiable as a performance measure, distinct from a goal or program activity. Baselined The measure includes the current level of performance as a baseline. Targeted The measure includes a specified, time-bound performance target. Aligned The measure provides the appropriate level of measurement given its location in the results matrix. Outcome measures describe the accomplishment of changes in country practices or behaviors (for Bank outcomes), while output measures provide information on services delivered by the Bank in order to accomplish outcomes. Inputs and activity measures are used only to provide leading indicators of output or outcome measures where results are hard to measure or slow to develop. 54 A P P E N D I X A : R E V I E W O F C A S R E S U LT S M AT R I C E S Framework level Level 0 Level 1 Level 2 Level 3 The matrix portrays the appropriate The matrix contains and shows The matrix portrays the elements of the elements of a results chain, but most of the appropriate elements results chain in a way that clearly and the schematic portrayal does not of the results chain, but is directly demonstrates the linkage effective link the elements, or lacking one or two critical elements between country goals and Bank’s some elements are present or portrays some elements in a way results, as well as associated outputs, and linked but significant elements that is unclear. activities, and inputs. are lacking. Absence of Most Bank goals and activities seem Some pillars or high-level goals Most pillars or country goals are logi- evidence in the misaligned with the country goals, are well organized and supported cally supported by succinctly stated criterion area. or a profusion of goals and activities by succinctly stated Bank activities Bank activities and goals. buries the logical, causal relationships and goals, but other areas are less effectively portrayed. Other donor and lender activities are Some country goals have supporting Almost all of the key activities of other described in the CAS text but mostly activities effectively associated but donors and lenders included in the CAS ignored in the results matrix, or the others do not, or some activities text are cited in the appropriate country activities of other organizations are are placed appropriately but some or Bank goal of the results matrix. included but misaligned. key activities described in the text are missing or improperly aligned. Very few meas- Fewer than half of the measures More than half but fewer than More than three-quarters of the meas- ures in the matrix demonstrate the defined qualities. three-quarters of the measures ures in the matrix exhibit the defined demonstrate in the matrix show the defined qualities. the defined qualities. qualities. 55 APPENDIX B: COUNTRY DIRECTOR SURVEY RESULTS ON THE USE OF MONITORING AND EVALUATION SYSTEMS AND INFORMATION AT THE COUNTRY LEVEL This appendix summarizes the results of a sur- Progress toward CAS Objectives vey of selected country directors carried out as The country directors were asked what regular part of the 2005 AROE in March 2005. The ob- processes they had in place to monitor and re- jective of the survey was to obtain information view progress on the CAS: (a) within the Bank, on the use and usefulness of monitoring and (b) with the government, and (c) with other evaluation systems and information at the coun- donors. All the respondents replied that they try level for Bank country program manage- regularly review progress toward CAS objectives ment. The questionnaire covered the processes with their country teams through work pro- to review progress toward CAS outcomes, in- gramming, stock-taking (such as Country Port- formation systems, and the availability and use- folio Reviews), or regular country team meetings. fulness of management information. The survey Most (87 percent) respondents discussed was sent electronically to country directors of progress toward CAS objectives with country countries that had submitted Country Assis- counterparts. These discussions were conducted tance Strategies (CASs) to the Board in fiscal as part of regular consultations and program 2004 and 2005 (as of end-January 2005). These and portfolio reviews. In low-income countries, country directors were selected because most and Poverty Reduction Strategy Paper (PRSP) of these CASs followed or incorporated ele- and Credit (PRSC) reviews were vehicles for ments of the results approach. these discussions. In one middle-income coun- A total of 26 country directors were surveyed.1 try, the country director expected to use pro- Responses were received from 16 (62 percent) grammatic adjustment loans (a series of of the country directors (3 from staff designated Development Policy Loans) as the primary vehicle by the country director). The numbers received for dialogue with the government. Fewer re- and response rate2 by Region and country type spondents (64 percent) replied that they dis- are shown in table B.1. cuss progress toward CAS objectives regularly Table B.1: Summary of Responses By Region Received Rate (%) By country type Received Rate (%) Africa 7 78 Middle income 8 53 East Asia and Pacific 1 50 Low income 8 73 Europe and Central Asia 5 71 Total 16 62 Latin America and the Caribbean 1 20 Middle East and North Africa 1 100 South Asia 1 50 Total 16 62 57 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N with other donors. The discussions were held as project and ESW deliveries following the dis- part of joint Bank and donor reviews with the cussions with their country teams. government, such as the PRSP review, or regu- Respondents’ discussions with the govern- lar donor consultations. ment were more focused on projects. The coun- The respondents discussed results, both of the try directors in the survey responded that the CAS and Bank operations, with their country status of the project portfolio was the main item teams. When asked to rate the extent to which of discussion, followed by project readiness (the they took up various operational items with their quality of project preparation) and CAS out- country teams (with 1 equaling “not at all” and comes (Figure B.2). This result most likely reflects 5 equaling “very much”), the country directors the venue for these discussions: that is, in the survey responded that CAS outcomes and PRSP/PRSC reviews, programming discussions, project approvals were the most frequently dis- and Country Portfolio Performance Reviews. cussed items, both with an average of over 4 on a 5-point scale (figure B.1). Information Systems The averages mask differences in approach: While many respondents rely on the Bank’s some country directors placed greater empha- SAP/Business Warehouse for information to man- sis on project deliveries, while others focused age their country program, many have devel- more on CAS outcomes and less on the partic- oped their own systems to consolidate necessary ulars of the work program. For example, one information from Bank systems or to supple- country director agreed to milestones toward ment it to meet their own requirements. When CAS objectives with his country team, and these asked to rate whether they used the Bank’s SAP milestones became the focus of reviews during and Business Warehouse for information in con- the fiscal year. Most country directors adjusted ducting their reviews with the country and the Figure B.1: Discussions with Country Team 4.50 4.00 (1 = Not at all—5 = Very much) 3.50 3.00 2.50 2.00 1.50 1.00 CAS Project Portfolio Project Project/ ESW Budgets outcomes approvals status readiness ESW results deliveries 58 A P P E N D I X B : C O U N T R Y D I R E C T O R S U R V E Y R E S U LT S Figure B.2: Discussions with Government 4.50 4.00 (1 = Not at all—5 = Very much) 3.50 3.00 2.50 2.00 1.50 1.00 Portfolio Project CAS Project Project/ ESW Budgets status readiness outcomes approvals ESW results deliveries government, the responses ranged from 1 (not terest was often cited as the reason for not de- at all) to 5 (very much), with a mean of 3.5 (with veloping a joint monitoring system. Some re- a coefficient of variation of 35 percent). Sixty- spondents commented on the effort needed to seven percent of respondents indicated that they develop a common system: “It takes time and ef- were able to obtain the information needed from fort and requires sustained government attention the Bank’s information systems. However, 12 re- as well as Bank dedication to the task.” Lack of spondents (75 percent) have packaged the in- interest was the most common reason given for formation to better meet their needs. Some have not developing joint systems or processes with consolidated data from different Bank systems others donors. Several country directors in the into their own spreadsheet or have included out- survey mentioned that they intended to develop come information, such as that from PRSP re- a common system or process with other donors, views, which is not available from the Bank. most often through the PRSP. The respondents have yet to develop common systems or processes with the government or Availability and Usefulness of the donors to jointly develop or share information Management Information Available for monitoring progress on meeting CAS objec- The country directors were asked to rate, from tives. Five Country Directors in the survey re- the following list of five possible uses of infor- sponded that they have a common system or mation, which they considered important for process with the government, and only one re- managing the Bank’s country program, and sponded that they have a system with donors. which they thought were satisfactory in terms of Several respondents indicated that they were in the information currently available to them. the process of developing a joint system with the government – for example, through an IDF grant. • Communications—describing Bank country The lack of government capacity or lack of in- performance 59 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N • Accountability—assigning responsibility for Overall scores for satisfaction with the infor- success or failure for Bank operations mation the respondents have available were • Learning—developing alternative strategies or lower: the highest was 3.7 for modifying opera- approaches based on success or failure tions (figure B.4). • Business planning—allocation and adjusting human and financial resources General Comments • Modifying operations—redesigning projects, When asked to comment on the available infor- revising strategies. mation and the need for improvements, several respondents mentioned the need for a more They also were given the opportunity to raise systematic approach to making country-level in- other important uses of information not in- formation available, cluded in the above list, but no other major uses were identified. • “Standardized results measurement equiva- The respondents rated learning as the most lent to key macro indicators should become important use of information (figure B.3). They part of the normal reporting from sector de- were in relative agreement on this point (with a partments.” coefficient of variation of 15 percent). Learning • “It would be ideal to have the system generate was followed by business planning and modify- monthly country specific reports on the port- ing operations. Accountability was considered to folio performance, quality, disbursements, etc., be less important. Several commented on the dif- to avoid all the manual handling of informa- ficulty of determining the Bank’s contribution (or tion.” lack thereof) to an outcome as a reason for giv- • “Information is scattered and not readily avail- ing accountability a lower rating. able in one place. Suggested Country Director Figure B.3: Important Uses of Information 4.50 4.00 (1 = Not at all—5 = Very much) 3.50 3.00 2.50 2.00 1.50 1.00 Learning Business Modifying Communications Accountability planning operations 60 A P P E N D I X B : C O U N T R Y D I R E C T O R S U R V E Y R E S U LT S Figure B.4: Satisfaction with Information 4.50 4.00 (1 = Not at all—5 = Very much) 3.50 3.00 2.50 2.00 1.50 1.00 Modifying Learning Business Communications Accountability operations planning Portal would be a welcome introduction as it synthesize information available from Bank sys- would allow [us] to organize all of the infor- tems and from operations (such as PRSP re- mation in one place.” views). Several commented on the need for a • “Quicker feedback on ground level impact more systemic approach to data collection at would help.” the country level. • “[Information] adequate, but presentation is But only five (31 percent) of the respondents not always user friendly; information relevant indicated that they have a common system or to manage the portfolio had to be developed process with the government to jointly develop ‘in-house’ since most of what comes from DC or share information for monitoring progress on is hard to use for action.” meeting CAS objectives. Most existing systems and processes involved the PRSP. Several re- Conclusions spondents indicated that they were developing In sum, respondents reported that they see re- a joint system with the government through an viewing progress toward CAS objectives as part IDF grant or other means. Lack of government of their internal program reviews with their capacity or lack of interest were often cited as country teams and with the government. Project the reasons for not developing a joint moni- implementation and readiness feature more toring system. prominently in their discussions with the gov- The respondents saw learning, business plan- ernment, but this may be influenced by the con- ning, and modifying operations as relatively text, PRSP/PRCS reviews, and portfolio reviews. more important uses of information compared Generally, the respondents indicated they with communication and accountability. That is, were able to obtain information needed to man- they report seeing information as useful for man- age their country programs from Bank systems, aging and improving their program more than but many have developed their own systems to for reporting and enforcing accountability. 61 APPENDIX C: MONITORING AND EVALUATION FRAMEWORK IN THE BANK Table C.1: Framework for Monitoring Bank Operations Type of Tracking and intervention Defining outcomes reporting on Managing or financing and monitorable Review and implementation implementation instrument indicators quality assurance progress progress Loan/credit/ Project Appraisal Document Management and peer Implementation Status Mid-Term Review. Country grant operations (PAD) (revised) or Memo- reviews, Quality Enhance- and Results Report Portfolio Performance randum & Recommendation ment Reviews (QER), and (ISR) (new) Reviews (CPPR), annual of the President annual QAG review of QAG reviews of supervi- quality at entryb sion (QSA) ESW/AAA Concept Paper Peer and management SAP monitoring Peer review, management review, QER (few), QAG review, QAG ESW reviews ESW reviews QAG Country-level AAA Assessments (new) Trust fundsc Initiating Brief for trust funds VPU, TFO review Trust Fund Status Reports (new) Implementation Completion Memorandum (ICM), (new) Sector strategy Sector Strategy Implemen- QAG, network, and Sector Strategy Implemen- Sector Board (SB) reviews tation Update (SSIU) (new) OPCS review tation Update (SSIU) Country programd Results-Based Country SB, network, and CAS Progress Report CPPR, management Assistance Strategy OPCS review (CASPR) reviews (Results-Based CAS) (new) IDA14 Results Framework (new) Grants/global Grant proposal Task manager review Financial reporting Task manager review programs Note: ESW = economic and sector work; SAP = Systems, Applications, and Products in Data Processing; TFO = Trust Fund Operations. a. Grants and Global Programs and Knowledge Initiatives are discussed in Chapter 3. One other means of financing not discussed here is guarantees. b. QAG’s quality at entry assessment is undertaken after project approval c. Trust funds are a source of funding that is used to supplement Bank financing of various Bank and recipient development activities. d. Country program support includes CAS, PRSP, JSA, HIPC, and other development services. 63 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N Table C.21: Framework for Evaluation in Bank Operations Type of intervention or Self-evaluation Independent evaluation (IEG) financing Individual Individual instrument activity Aggregate level activity Aggregate level INSTRUMENT Loan/credit/ Implementation Lending retrospectives Evaluation summaries, Annual Review of Devel- grant operations Completion Reports Project Performance opment Effectiveness (ICRs) Assessment Reports, (ARDE) Impact Evaluations Economic and Activity Completion Regional retrospectives Periodic studies (such as Sector and thematic stud- sector work/ Summaries (ACS) for (ad hoc and rare) Public Expenditure ies, implementation re- analytical and tasks > $50,000 Reviews, Poverty views, and Country Assis- advisory services Assessments) tance Evaluations Trust funds Implementation Completion Annual Review of Trust Fund Memorandum (ICM) Portfolio Performance, some > $1 million donor reports Sector strategy n.a. n.a. Sector and thematic Annual Report on Develop- studies ment Effectiveness Country program CAS Completion Reports CAS Retrospectives Country Evaluation Annual Report on Develop- (new) Summaries (new), CAEs ment Effectiveness, Re- gional IEG Evaluations (CFA, transition economies) Grants/global Grant Completion DGF evaluations for Global Program IEG evaluation of global programs Reports (GCR) >$300,000 Reviews (pilot) public programs KNOWLEDGE INITIATIVES Knowledge For few knowledge Evaluation of Knowledge management sharing activities Sharing (IEG) 2003a) DEC research Research Project 2001 Research Support Completion Reports Budget (RSB) evaluation World Bank Individual learning events World Bank Institute Institute training evaluations of selected and capacity programs building 64 APPENDIX D: IEG CLIENT SURVEYS Since 1999, IEG has surveyed its clients annually • Project Performance Audit Reports (PPARs) to gain their perceptions of the quality and im- • Evaluation products: pact of IEG evaluations as part of the continuous – Books, Buildings, and Learning Outcomes: process of self-evaluation. The survey assesses IEG An Impact Evaluation of World Bank Sup- evaluations in four areas: readership and aware- port to Basic Education in Ghana ness, quality, the influence of the report on Bank – Evaluating The World Bank’s Approach to processes and policies, and recommendations Global Programs: Addressing the Chal- made to IEG on improving the quality and rele- lenges of Globalization vance of the reports. This year IEG conducted two – The Poverty Reduction Strategy Initiative: surveys: one of internal clients in the Bank and An Independent Evaluation of the World a second of external clients. This appendix reports Bank’s Support Through 2003 on the findings from these two surveys. – Economies in Transition: An IEG Evalua- tion of World Bank Assistance. Results of Survey of Internal Clients This year an internal survey of 2,120 Bank staff Two primary audiences were targeted by this sought to elicit their perceptions of several kinds year’s survey: of IEG products. The following IEG products were the subject of the survey: • Bank staff groups associated with the prod- ucts • Country Assistance Evaluations (CAEs): • Executive directors (EDs). – Armenia: Country Assistance Evaluation – Bosnia and Herzegovina: Post-Conflict Re- Target respondents were surveyed only once, construction and the Transition to a Mar- even if their work was relevant to more than ket Economy, An IEG Evaluation of World one assessed product. The following is a sum- Bank Support mary of the results. This year’s results are com- – Brazil: Forging a Strategic Partnership for pared with results from previous years and similar Results, An IEG Evaluation of World Bank types of respondents, where possible. Assistance – China: An Evaluation of World Bank As- Response Rates sistance The surveys were sent electronically to 2,120 staff – Croatia: Country Assistance Evaluation over a two-week period in April 2005. The re- – Jordan: Supporting Stable Development in sponse rates ranged from 23 percent (for exec- a Challenging Region utive directors) to 46 percent (for the Armenia – Rwanda: Country Assistance Evaluation CAE), and averaged 31 percent across all survey – Republic of Tunisia: Country Assistance products. This represents an increase in re- Evaluation. sponse rate relative to the 2004 surveys (21 per- • The 2003 Annual Review of Development Ef- cent). The response rate was boosted through fectiveness: The Effectiveness of Bank Support the use of multiple reminder notifications. Re- for Policy Reform minders directly targeted non-respondents. 65 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N Figure D.1: Response Rates 100% 90% 80% 70% Response rate 60% 50% 40% 30% 20% 10% 0% Evaluations PPARs Total 2001 2002 2003 2004 2005 Figure D.2: Readership and Awareness of IEG Evaluations 100 90 80 70 Percent read 60 50 40 30 20 10 0 PPARs Evaluations Type of evaluations 2000–01 2002 2003 2004 2005 Survey response rates are not directly com- could re-enter that record until they had com- parable to those earlier years, however. Prior to pleted the survey to their satisfaction. this year’s surveys, individuals did not receive a unique survey login and could enter the survey Readership and Awareness multiple times. They might have had multiple Almost half (45 percent) of non-ED respondents records within the database. For the 2005 survey, had read, participated in a discussion, or at- individuals were limited to a single record but tended an event on the evaluation for which 66 APPENDIX D: IEG CLIENT SURVEYS they were surveyed. This is a marked increase percentages for evaluations in general for these over the levels of awareness found in previous three measures (80 percent, 81 percent, and 82 AROE surveys. An additional 13 percent of re- percent respectively). A comparison with prior spondents indicated that they were aware of the years shows that evaluations have consistently findings, even though they had not read the been timely. study or participated in any dissemination events. Respondents were asked to assess several as- The level of awareness varied by product read- pects of their evaluation (or, for executive di- ership, from a low of 15 percent to a high of 79 rectors, evaluations in general). Figure D.4 shows percent awareness. Executive directors were the proportion of respondents that rated each asked if they had read, participated in a discus- aspect of the evaluations as either excellent or sion, or attended an event for any 2004 evalua- good. tion, and 85 percent said that they had. A strong majority of all respondents rated For non-ED respondents who were aware of each of the aspects positively: the evaluation, half (52 percent) heard about it online or via email. Thirty-five percent were con- • Depth of analysis (74 percent good or excel- sulted during the evaluation. For executive di- lent) rectors, three-quarters (74 percent) of them • Presentation of the evidence (70 percent) heard about these evaluations at CODE meetings, • Link between conclusions and evidence (62 and almost two-thirds (64 percent) through print percent, but 10 percent rated this aspect as copies. poor) Over three-quarters of non-ED respondents • Presentation of conclusions (75 percent) said that their evaluations were timely (89 per- • Relevance of lessons to Bank’s work (69 per- cent), made full use of available information (74 cent, with 21 percent rating this aspect as ex- percent), and used a methodology that was cellent). transparent and clear (78 percent). Similar to other respondents for a specific evaluation, ex- In general, respondents to this year’s surveys ecutive directors reported positively for large rated the evaluations the same or just slightly less Figure D.3: Timelines of IEG Sector and Thematic Evaluations 100 90 80 70 Percent yes 60 50 40 30 20 10 0 2000–01 2002 2003 2004 2005 (3 studies) (4 studies) (5 studies) (12 studies) (14 studies) Timely? 67 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N Figure D.4: Quality of Studies 90 80 Percent excellent or good 70 60 50 40 30 20 10 0 Depth of Presentation of Link between Presentation of Relevance of analysis the evidence conclusions and conclusions lessons to evidence Bank’s work 2004 2005 positively than for the 2004 surveys, with less Executive directors were asked if they had than a three percentage-point average move- used, or planned to use, the findings from any ment from excellent or good to fair or poor IEG evaluation in their work. Ninety-five per- across the five rating scales. cent said they had or would use the findings, with Overall, the executive directors rated evalua- the remaining 5 percent indicating “don’t know.” tions more positively than did the non-ED re- In a pair of related questions, non-ED re- spondents, with at least 84 percent good or spondents were asked to what extent their eval- excellent ratings for each aspect. Most positively uation influenced the Bank’s country or sector rated was depth of analysis (95 percent good or strategy, and to what extent their evaluation in- excellent, including 24 percent excellent). The fluenced the Bank’s lending and nonlending only aspect rated as poor by any ED respondents services. The respondents’ answers regarding was the link between conclusions and evidence. influence were similar overall for strategy and This aspect also received the lowest excellent services. Almost half (47 percent) of the re- rating (8 percent) among executive directors. spondents in each case said that the evaluation somewhat influenced the country or sector strat- Influence of IEG Evaluations egy or the lending and nonlending services. Over three-quarters (78 percent) of non-ED re- spondents said that their evaluation influenced Results by Demographic Profile their view of the associated subject area either a IEG checked for differences in perceptions of IEG great deal (19 percent) or somewhat (59 percent). work by staff level (manager versus non-manager) Among executive directors, almost all (95 per- for Sector and Thematic evaluations, and by lo- cent) said that IEG evaluations influenced their cation (headquarters versus field-based) for both view of a subject area, with 40 percent reporting sector and thematic and PPAR evaluations. The a great deal of influence. findings:1 Only 12 percent of non-ED respondents said they had not and would not use the findings of • Indecision or lack of knowledge about using the evaluation in their work. Seventy percent the evaluations was greatest among respon- said they had or would use the evaluation’s dents who were of level F and below (31 per- findings, while the remaining 18 percent did cent verses 16 percent for those of level G or not know. higher). Among country directors and man- 68 APPENDIX D: IEG CLIENT SURVEYS agers, 11 of 12 respondents (92 percent) said non-ED respondents). Nearly half (46 percent) that they would use the findings of the evalu- of non-ED respondents also suggested that ation in their work. broadening consultation with Bank staff could im- • With regard to evaluation influence on coun- prove their evaluation. try or sector strategy, more lower-level re- Executive directors were asked to select from spondents (12 percent of those level G and the same set of recommendations for IEG eval- under) than higher-level respondents (4 per- uations in general. A full 87 percent of ED re- cent of level H and above) said there was a great spondents said they would make the findings deal of influence. more operational. The ED respondents differed • Respondents in country offices recommend from non-ED respondents in their lack of en- more training or educational materials than thusiasm for broadening consultation with Bank those in Headquarters as a means of improv- staff, with only one in five (21 percent) selecting ing outreach. this potential improvement. ED and non-ED respondents were asked to se- Recommendations from Respondents lect recommendations that they thought would When asked what recommendations they would improve IEG’s outreach. Respondents generally make to improve their associated evaluation, called for summaries of IEG findings to be pro- the recommendation most often selected was to vided and for better notification of new docu- make findings more operational (70 percent of ments and events. Figure D.5: Recommendations for IEG 80 70 60 Percent responding 50 40 30 20 10 0 l gs ts is ls sh ff ce y n t) sis ta na all tio ne ta lys ria en da din en gli aly tio rn ks ma ra ev te na vid En ion fin te ra an Int ma an fa or nd ex pe ee an at EG B & inf 4: )o sa ion eo alu th lt or ult 0 e fI su y ed (20 nc t nm er lit at or v so en ns on oe z ide th ua uc sm mi Co um ne rie i C eo ta st d :q ev to nli /e ing ma c Ob es do 03 ag us to eo ing ind um cc ec (20 gu ew ns ibl ain ea ef lan es or sio th fn ss r M or ak et or ep ce clu in no M M M or ed ac ial on tio M er re ov kc ica at mo pr Lin if em Im ot t en rn or nt M tte Co Be 2004 2005 (excluding executive directors) 69 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N Results of Survey of External Clients Response Rates This year’s external survey queried 926 external The surveys were sent electronically to the re- clients, including government officials, consult- ported e-mail addresses of 926 external clients, ants, and researchers. They were asked about two including government officials, researchers, and IEG product types: consultants during a two-week period in April 2005. The overall response rate was 31 percent. • Two evaluation papers associated with the In- Disaggregated, the response rates were 28 per- ternational Program for Development Evalua- cent for GPP; 34 percent for IPDET1; and 33 tion Training (IPDET): percent for IPDET2. The response rates were – Monitoring and Evaluation: Some Tools, similar to those for the 2005 internal client sur- Methods, and Approaches vey. Response rates were boosted through the – Influential Evaluations: Evaluations that use of multiple reminder notifications that di- Improved Performance and Impact of De- rectly targeted non-respondents. velopment Programs. • One evaluation associated with Global Pro- Readership and awareness. Two-thirds (66 per- grams and Partnerships (GPP): cent) of all respondents read, participated in a – Addressing the Challenges of Globalization: discussion, or attended an event on the evalua- An Independent Evaluation of the World tion or evaluation paper for which they were Bank’s Approach to Global Programs. surveyed (see figure D.7). The level of awareness varied somewhat by population, from 58 percent A pool of potential survey respondents was se- awareness for GPP to 73 percent awareness for lected for each product type: IPDET and GPP . The IPDET2. IPDET pool was randomly split into two equiv- The vast majority of IPDET respondents (86 alent-sized subgroups corresponding to each percent of IPDET1 and 94 percent of IPDET2) individual evaluation paper: IPDET1 for the Mon- heard about the evaluation paper through email itoring and Evaluation (M&E) paper and or online. Also, half (53 percent) of GPP re- IPDET2 for the Influential Evaluations paper. spondents heard about the evaluation this way. The following is a summary of the results. Half of GPP respondents also said they heard Figure D.6 : Response Rates 100% 90% 80% 70% Response rate 60% 50% 40% 30% 20% 10% 0% IPDET1 IPDET2 GPP Total Types of evaluation 70 APPENDIX D: IEG CLIENT SURVEYS Figure D.7. Readership and Awareness of IEG Evaluations 100 90 80 70 Percent read 60 50 40 30 20 10 0 Total GPP IPDET1 IPDET2 Type of evaluations about it by being consulted during the evaluation. said it did. Eighty-one percent of IPDET1 and 89 IPDET respondents were asked if they were con- percent of IPDET2 respondents said their eval- sulted about the evaluation paper during its uation paper described M&E techniques appro- preparation and only 2 percent said they had priately. Eighty-three percent of GPP respondents been consulted. said they thought the evaluation used a method- ology that was transparent and clear. Similarly, 92 Quality of IEG Evaluations percent of IPDET respondents said they thought A large majority of respondents (94 percent) in- the evaluation paper presented information in dicated that their evaluation (GPP, 94 percent) a transparent and clear manner. or evaluation paper (IPDET1, 96 percent; Respondents were asked to assess several as- IPDET2, 93 percent) was timely. GPP respon- pects of their evaluation (or, for executive di- dents were asked if the evaluation made full use rectors, evaluations in general). Figure D.8 shows of available information, and 80 percent of them the proportion of respondents that rated each Figure D.8: Quality of Studies 90 Percent excellent or good 80 70 60 50 40 30 20 10 0 Depth/quality of Presentation of Link between Presentation of Relevance of analysis the evidence conclusions and conclusions lessons to evidence (GPP Bank's work only) 71 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N aspect of the evaluations either excellent or (57 percent) than were researchers and con- good. sultants (40 percent). A strong majority of all respondents rated • While 80 percent of IPDET government officials each of the aspects positively: thought their evaluation paper was good or ex- cellent in presenting evidence, this was true for • Depth (GPP) or quality (IPDET) of analysis (82 just 59 percent of IPDET researchers. percent good or excellent) • Forty-two percent of IPDET government offi- • Presentation of evidence (71 percent good or cials rated their evaluation paper to be excel- excellent) lent at presenting conclusions, verses just 19 • Link between conclusions and evidence (GPP percent of IPDET researchers. only, 71 percent good or excellent) • Indecision or lack of knowledge about using • Presentation of conclusions (74 percent good the evaluation was greatest among IPDET re- or excellent) searchers (32 percent). • Relevance of lessons to your work (68 percent • Eighty-nine percent of IPDET government of- good or excellent), including three-quarters (75 ficials and 88 percent of IPDET consultants percent) of IPDET respondents. said they had or would use the findings of the evaluation paper in their work, compared with Influence of IEG Evaluations just 59 percent of IPDET researchers. GPP respondents were asked the extent to which the evaluation influenced their view of the sub- Recommendations from Respondents ject area. One in five (20 percent) said that it in- When asked what recommendations they would fluenced their view a great deal, and another 48 make to improve their associated evaluation, percent said it influenced their view somewhat. the recommendation selected most often over- A quarter (26 percent) said it had little influ- all was to make findings more operational (61 ence, while 7 percent (just 3 of 46 respondents) percent, including 78 percent of IPDET govern- said it had no influence. ment officials) (figure D.9). Additionally, 58 per- Only 9 percent of respondents (including cent of IPDET respondents (including 75 percent none of the IPDET government officials) said of consultants but just 44 percent of govern- they had not and would not use the findings of ment officials) recommended broadening the the evaluation or evaluation paper in their work. range of M&E techniques discussed. Half (50 Seventy-four percent said they had or would use percent) of those surveyed recommended im- the findings. The remaining 17 percent did not proving depth of analysis. Broadening external know. Of the populations responding, indeci- consultation was selected by 62 percent of GPP sion or lack of knowledge about using the eval- respondents, but by only 19 percent of IPDET re- uation was highest for IPDET2 ( Influential spondents. Respondents were also asked to se- Evaluations) respondents (23 percent). lect recommendations that they thought would improve IEG’s outreach. The potential im- Results by Demographic Profile provement selected most was for summaries of For IPDET respondents, IEG checked for differ- IEG findings to be provided. Providing more ences in perception of IEG work by position. The training or education materials was the most se- findings:2 lected recommendation among IPDET respon- dents, but only 15 percent of GPP respondents • Among IPDET respondents, evaluation paper selected it. awareness was higher for researchers (76 per- cent) and consultants (77 percent) than for gov- Implications for IEG ernment officials (65 percent). The survey results highlight the need for IEG to • IPDET government officials were more likely continue to improve the communication and to share information or findings with colleagues dissemination of its evaluation products within 72 APPENDIX D: IEG CLIENT SURVEYS Figure D.9: Recommendations for IEG 70 60 50 Percent responding 40 30 20 10 0 l s sis e s is ion sh T) s ta P) ) na y PP ing ial nt lin lys all DE GP da gli aly tio at (G ve er on nd rn na (IP e( En rm n de at ra an ff te o i le da nc f sta i nm pe es o an at x an EG b of inf e ide i an u eo u th s ult nk niq tio fI th l es cs a ed ev er e or v so p Ba ns do cc ca ch oe nc de z th sm re mi ea Co du rie te ide ult ew eo st mo ve to /e ing ma or &E ns ev s fn o ag us ing e tm in pr Co ind um nM cc ec gu no to ta Im n en ea ef lan Ob i es ns or tio ra de nt M or ak sio et or ca in oa Co M M M or clu ifi ial Br M ot on er rn at kc tte em Lin Be or M the Bank and among its external clients. The formation appears to be important for dissemi- new Outreach and Communications Strategy nation, although effective targeting of audiences being launched by IEG is aimed at improving will be essential. In addition, the survey find- IEG’s performance in this area. The survey find- ings indicate that IEG needs to do a better job ings indicate that clients would find summary in- in relating its findings to operations. Finally, IEG formation about IEG products useful. Also, needs to strengthen the linkages between evi- increasing the use of online distribution of in- dence and conclusions. 73 APPENDIX E: MANAGEMENT RESPONSE Introduction (World Bank 2005a). The new approach helps dis- The 2005 Annual Report on Operations Evalu- tinguish between long-term country develop- ation (AROE) focuses on the country as the unit ment goals and intended outcomes to which of account for monitoring, managing, and evalu- the CAS program directly contributes; strength- ating performance. The report takes stock of how ens the use of lessons learned from the last CAS; well key Bank instruments and M&E procedures improves alignment of the Bank’s program with are linked to country goals and capacities, as well the country’s priorities; sharpens the design of as to core elements of results-based management the Bank program; and mobilizes country teams within the Bank and at the country level. around a common vision of delivering results on Management not only agrees with this focus, the ground. but has, in fact, been proactive in helping to Another notable recent development is self- lead the development community in that direc- evaluation of the country assistance program in tion—internationally with the Comprehensive a CAS Completion Report (CASCR). The focus of Development Framework and Poverty Reduc- the CASCR is on the achievement of CAS out- tion Support Paper (PRSP) concepts and lead- comes and on Bank performance in furthering ership on the results agenda; internally in the CAS outcomes. It includes a discussion of proj- gradual strengthening of the Country Assistance ects and analytic and advisory work and is pre- Strategy (CAS), including work on the country pared at the end of the CAS period in time to business model (Development Committee 2000), deliver useful lessons for the next CAS. All leading up to the Results-Based Country Assis- CASCRs are independently evaluated by IEG, tance Strategy (RBCAS); and in its support to and each IEG assessment is submitted to the countries. Bank’s Board in advance of the Board’s discus- This year’s AROE (which comes slightly more sion of the next CAS document for the country. than six months after RBCASs were main- streamed in the Bank) reviewed 24 CASs sub- AROE Analysis mitted to the Board in fiscal years 2004 and 2005 The AROE reviews the use and efficacy of M&E and reflects interviews with a sample of 26 coun- for country program management. It also cata- try directors. The questions posed in this AROE logues the measures taken to strengthen the re- are relevant to management’s views of strategic sults-orientation in M&E at the country, sector, priorities. The findings confirm management’s and product levels within the Bank, updating sense of the advantages and obstacles to the overview in the 2003 and 2004 AROEs. The strengthening both Bank and country-level will- analysis is a timely input for the next steps in the ingness and capacity to manage for results. evolution of the country business model and internal implementation of the Results Agenda. Recent Progress Although the RBCAS approach is new, a careful Measuring and Managing Performance review by management of seven pilot country at the Country Level cases has shown that it constitutes a significant For the most part, the AROE uses the same con- step forward in taking results more seriously cepts for measuring and managing performance 75 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N at the country level that management has used Baselines and Targeted Performance in setting out the RBCAS and the Results Agenda. Measures in CASs These concepts are: strategic planning, per- The AROE points out that lack of baselines and formance measurement, and performance man- targeted performance measures limits the ef- agement. Management welcomes the finding fectiveness of the Results Matrix as a monitoring, that of the 24 CASs reviewed for the AROE, 19 management, and evaluation tool. The report ap- attempted to establish causal linkages between propriately recognizes that the absence of base- country development goals, CAS outcomes, and lines and performance measurement most often Bank operations. It also welcomes the finding reflects lack of capacity and systems at the coun- that the Bank’s internal information systems, try level. such as the Business Warehouse, allow Bank Management agrees that baselines and tar- managers and staff to monitor budget use, prod- gets are central for managing for results, but uct deliveries, and the quality of the portfolio at notes that the RBCAS has only recently been the country level. mainstreamed, and that RBCASs are increasingly drawing attention to capacity gaps at the coun- IEG’s Country Program Rating Methodology try level. It will take time before the Bank and our The AROE cites an IEG review that found that partners develop the systems necessary to ensure one-third of country programs were rated un- that baselines and targeted performance meas- satisfactory. Management notes that one-third un- ures are in place—probably a whole RBCAS cycle satisfactory outcomes in country programs do for most countries, or longer. In the early stages not translate into an unsatisfactory Bank per- of the RBCAS more attention has been focused formance in one-third of country programs. on overcoming conceptual difficulties, such as IEG’s methodology distinguishes between the identifying those outcomes to which the Bank performance of a country program and the per- could contribute, but not exclusively, and dis- formance of the Bank in support of that program. tinguishing between longer-term country ob- In IEG’s methodological framework, the out- jectives and CAS outcomes. A key issue has been come of the Bank’s assistance program is de- striking a balance between choosing relevant termined not only by the Bank’s performance but outcomes to be monitored and availability of by the joint impact of four agents (a) the client, baseline data to monitor outcomes. Manage- (b) the Bank, (c) partners and other stakehold- ment agrees with AROE’s observation that “avail- ers, and (d) exogenous shocks (including events ability of data, however, should not be a main of nature and international economic shocks). reason for selecting measures. It may be neces- Also, despite extensive discussions, manage- sary to include indicators for which data collec- ment and IEG have until recently not found tion systems will need to be developed during agreement on the country program rating the CAS period.” Management does not want to methodology, in part because before RBCASs create incentives for outcomes to be chosen were introduced, country programs were not only because baseline data are available, or for designed to be evaluated. IEG has long rated the goal to become meeting targets—as op- the achievement of objectives, but objectives posed to achieving results. and strategies to achieve them change from one CAS to the next in response to changing coun- Capacity and Interest in Countries to Manage try circumstances, and it was not clear to man- for Results agement which objectives IEG rated. Helping country partners develop the interest Management and IEG are working to establish and institutional capacity needed to manage for a clear understanding on the basis for ratings in results is one of the three pillars of the results the context of RBCASs and CASCRs, and man- agenda, and central to the success of the entire agement expects greater convergence of views agenda. Since RBCASs are based on the client in the future. country’s own objectives as articulated in PRSPs 76 APPENDIX E: MANAGEMENT RESPONSE or similar documents, they reflect country own- Compared with conventional CASs, the five ership. The Bank has also actively supported pilot RBCASs for IDA countries paid more at- countries in strengthening their national strate- tention to deficiencies in statistics and pro- gic planning (including for poverty reduction posed more concrete plans to address those strategies) and in building results-based public deficiencies. Both pilot CASs for IBRD bor- sector management, statistical capacity, and M&E rowers proposed strengthening of monitor- systems. Specific country-level activities by the ing and evaluation capabilities. Bank include the following: Country Ownership and the RBCAS • Strengthened advice from the Poverty Reduc- While the Bank can and should play a support- tion and Economic Management Network on ive role in developing capacity, the AROE tends the design, monitoring, and evaluation of to downplay the role of the country in manag- poverty reduction strategies. ing for results. Management views performance • Demand-driven support from the Bank’s Mon- measurement and management by countries itoring and Evaluation Improvement Program themselves as central to the results agenda, be- for countries to assess their institutional readi- cause unless countries themselves manage for re- ness and strengthen results-based approaches sults, neither the Bank nor other donors can to public sector management. effectively manage for results. The most appro- • Development of the Statistical Capacity Build- priate place to “(a) determine the information ing Program—approved by the Board in March needed to monitor progress toward country 2004—which provides a sectorwide approach goals and manage the Bank country program; (b) to building capacity based on a national sta- set targets and establish baselines; (c) identify the tistical plan for providing reliable and timely sources of information; and (d) establish re- data on countries’ core development outcomes sponsibility for collection and analysis of the as articulated in their PRSPs or national devel- data” is not the RBCAS but countries’ own growth opment strategies. By the end of fiscal year and poverty reduction strategies. The RBCAS 2005, five countries had accessed this pro- can, however, “determine areas where additional gram. In addition, the multidonor Global Trust capacity building and assistance may be required Fund for Statistical Capacity Building, man- and addressed through the CAS.” One of the aged by the Bank, has supported the comple- commitments in the March 2005 Paris Declara- tion of National Statistical Development tion by partner countries was that they would Strategies in 13 out of 56 countries with a PRSP , “[e]ndeavor to establish results-oriented re- and a further 19 are in preparation. porting and assessment frameworks that moni- • Development of a system for measuring sta- tor progress against key dimensions of the tistical capacity at the country level. Starting in national and sector development strategies; and fiscal 2006, this measure is a key performance that these frameworks should track a manageable indicator in the Regional Strategic Performance number of indicators for which data are cost- Contracts (SPCs), signaling to staff the impor- effectively available” (box 2.3). tance of working with clients to build country- level capacity. Monitoring at the Sector Level • Mainstreaming of RBCASs, which also aim at The Sector Strategy Implementation Update strengthening client countries’ capacity for (SSIU) provides an integrated assessment of the managing for results and their monitoring and implementation of all sector (and thematic) evaluation systems. The Development Eco- strategies, trends in analytic and advisory serv- nomics Vice-Presidency in the Bank surveyed ices, lending commitments, and partnerships 50 recent CASs for IDA countries with the aim by sector/thematic areas. It draws on Imple- of identifying concerns about statistics and mentation Completion Reports to pull together strengthening statistical capacity (IDA 2004). selected output-level results, such as kilometers 77 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N of urban roads built/rehabilitated and the num- • Incorporate in the CAS a diagnosis of the coun- ber of rural facilities built/rehabilitated, organized try’s performance measurement capacity and by thematic area. Central elements of the re- recommendations for action to strengthen sults reporting and learning system that is under that capacity. The diagnosis and actions should development will likely include progress on sec- be customized to country capacity and focused toral outcomes in countries where the Bank is on critical information needed for the gov- active (or has focused its strategy), and lessons ernment to implement its program. learned on (a) best practice in achieving results, • Expand use of the Results-Based CAS as a tool (b) measurement issues, and (c) approaches to to manage country programs and strengthen strengthening capacity for managing for results the link between the Results-Based CAS and the at the sector/country level (including analysis country program by providing country teams of the incentive environment that determines de- with guidance designed to increase the results mand for information and statistical capacity orientation of CPPRs. building). Management Views Trust Funds and Global Programs Management agrees that a diagnosis of the coun- Management has consistently noted that trust try’s performance measurement capacity and funds are a financing vehicle to support activi- recommendations for action to strengthen that ties. What matters is that the activities be evalu- capacity is appropriate, and already recommends ated as activities, not as trust funds. There is a that an assessment of M&E and results capacity well-established agenda for M&E of activities be undertaken during CAS preparation. However, across the Bank. Many trust funds are financing management does not agree on the need to vehicles for partnership programs. Out of 115 mandate a formal document required as a CAS partnership programs, 59 are trust-funded and annex for all countries and CASs. There are cases fall under the new business process that requires where sufficient knowledge on country per- a description of objectives of the proposed work formance measurement capacity is already widely program, as well as performance indicators. This available or can easily be gained from work done is part of the new control of Global Program by partner agencies. Regarding the second rec- and Partnership (GPP) quality at point of entry. ommendation, management has already intro- Management is moving toward a situation in duced the RBCAS and CASCR to strengthen which all GPPs will have to be independently country program management. In the RBCAS evaluated every three to five years, regardless of framework, the CASCR forms a crucial bridge be- funding source. A clear and strong differentiation tween the downstream assessment of progress of the Bank’s roles as trustee and fiscal agent is toward CAS outcomes and the upstream as- important in the context of results orientation sessment of what is required in the next CAS to and accountability for results. When the Bank is support the achievement of national develop- strictly a fiscal agent, it cannot be accountable for ment goals. Results-focused CPPRs can support program results. For example, the Bank is a fis- country program management as input to the cal agent for the $3 billion Global Fund to Fight CASPR and CASCR. However, management AIDS, Tuberculosis and Malaria; however, be- would prefer to let the pilots now under way go cause the Bank is neither involved in the Fund’s forward and draw on that experience before de- secretariat nor has a vote on its Board, it cannot ciding on new staff guidance on CPPRs. be held accountable for the Fund’s outputs, out- comes, or impacts. Conclusions Management welcomes the AROE and finds its AROE Recommendations analysis timely for its work on the next step for The AROE has organized its recommendations the results agenda. Management has introduced for management consideration into two sets: RBCASs and CASCRs, so the key elements for 78 APPENDIX E: MANAGEMENT RESPONSE “strengthening country program management” this agenda. These difficulties are compounded are in place. However, management would em- in developing countries, where the requisite phasize the complexity of the changes necessary data are in short supply and the requisite insti- to significantly improve management for re- tutions are far less advanced. Success will re- sults—in client countries, in the Bank, and across quire time, money, and persistence; expecting the donor community. As the IEG notes, indus- results too quickly risks undermining support for trial country governments have struggled with the process. Matrix: Management Response to the 2005 Annual Report on Operations Evaluation IEG Recommendation Management Response 1. Incorporate in the CAS a diagnosis of the country’s Management agrees that this is an issue that should be discussed in the con- performance measurement capacity and recommen- text of RBCAS preparation. Management does not agree on the need to man- dations for action to strengthen that capacity. date any particular approach or standardize reporting in the RBCAS. A formal The diagnosis and actions should be customized to requirement applied to all countries may lead to duplication of efforts in cases country capacity and focused on critical information where sufficient knowledge can be gained from work undertaken/planned needed for the government to implement its program. through other operational work or by partner agencies. 2. Expand use of the Results-Based CAS as a tool to Management agrees with the major thrust of this recommendation. Manage- manage country programs and strengthen the link ment is mainstreaming RBCASs and CASCRs to improve the management of between the Results-Based CAS and the country country programs. CAS results are selected during CAS design, tracked during program by providing country teams with guidance implementation, and evaluated at the end of the CAS period. A CAS Progress designed to increase the results orientation of CPPRs. Report allows for midpoint stocktaking and, as necessary, course corrections. The results-orientation of the CAS and the CPPR is strengthened if the projects and analytic work focus on results. For both projects and analytic work, in the last 18 months Management has introduced a series of revised tools and for- mats to facilitate management of country programs in relation to the CAS. However, Management would wait for the results of pilot CPPR exercises before deciding on new staff guidance on CPPRs. 79 ENDNOTES Chapter 2 8. A program was prepared for Nigeria but was in- 1. Twenty-six country directors were surveyed and corporated into the Economic Reform and Gover- responses were received from 16 (62%). nance Project. 2. Of the 24 CASs reviewed, 6 were Results-Based 9. For example, the CAS mentioned that data for CAS pilots; 13 adopted a results approach, although key indicators may be nonexistent and need to be de- they were not formal pilots; and 5 followed the tradi- veloped and went on to mention how the Bank and tional format in place at the time but were included the government would collaborate to develop the in the review because they contained sufficient infor- necessary databases. mation to merit an assessment. Four other CASs also 10. These guidelines are under revision to reflect were prepared during this period following the tradi- the mainstreaming of the Results-Based CAS approach. tional format, but did not contain sufficient informa- 11. The country directors were asked to rate which tion for evaluation. The full review is in Appendix A. they considered important for managing the Bank’s 3. Performance measures are rated Level 0 if only country program from a list of five possible uses of in- a very few were well defined, baselined, or targeted. formation: (1) Learning—developing alternative strate- 4. For example, “Increase living standard of work- gies/approaches based on success or failure; (2) ing population measured as increase in real salaries” Business planning—allocation and adjusting human is a well-defined measure because it indicates specif- and financial resources; (3) Modifying operations— ically what evidence should be used to determine redesigning projects, revising strategies; (4) Com- the desired effect. But it lacks both a baseline and de- munications—describing Bank country performance; fined target that can be used to determine whether and (5) Accountability—assigning responsibility for the objective for this measure was met. success or failure for Bank operations. The results of 5. The Country Director’s Portal is expected to the survey are summarized in Appendix B. allow country directors easy access to country infor- 12. The list consists of strengthening the IDA14 re- mation such as project and AAA deliverables, portfo- sults measurement framework, the MDGs, and the Im- lio performance, country budget allocation, and plementation Follow Up-7 (IFU-7) indicators. The utilization and administrative matters, such as dead- Strategic Performance Contract is discussed in more lines and staff travel, which are available in different detail in Chapter 3. systems. 6. The stronger focus on results monitoring and Chapter 3 evaluation in the Development Policy Loans (DPLs) may 1. The sectors are investment climate/finance, create an impetus for stronger monitoring in middle- trade, education for all, hiv/aids, child and maternal income countries. In one middle-income country, the health, water supply and sanitation, environmental sus- country director expected to develop a system to mon- tainability. itor results through a series of single-tranche DPLs. 2. The indicators that will be tracked are: the qual- 7. When asked to describe difficulties they see in ity at entry of IDA-supported operations; project out- establishing a common system or process with the gov- comes for the IDA portfolio; the share of Implementation ernment, one replied: “It takes time and effort, and Completion Reports with satisfactory data on project out- requires sustained government attention as well as comes; and the share of Project Status Reports with sat- Bank dedication to the task.” isfactory baseline data for outcome monitoring. 81 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N 3. DEC has taken the lead in improving the qual- 2. Performance indicators need to be: defined, ity of the data for measuring progress on key indica- baselined, targeted, specific, aligned, timely, measur- tors, including by compiling statistical capacity scores able, attainable, and realistic. The evaluators did not for all countries that will be monitored by KPIs. have sufficient country knowledge to make an as- sessment on whether the indicators were specific, Chapter 4 timely, measurable, realistic, and attainable. The re- 1. The surveyed audience for IEG’s GPP report maining criteria (defined, baselined, targeted, and consisted of those that had been consulted during the aligned) are more technical in nature and could be evaluation, and for IEG’s M&E reports alumni of fairly assessed by evaluators with expertise in results- IPDET, and therefore considered interested in devel- based management, regardless of country knowledge. opment evaluation issues. Appendix B Appendix A 1. Thirty-two countries submitted CAS or Country 1. Capability maturity models were first developed Partnership Strategies to the Board during this period. at Carnegie-Mellon University in the mid-1980s as a way Given multiple country assignments, four country di- of comparing management practices of private sector rectors submitted 2 CASs and one country director companies that provide services to the U.S. Depart- submitted 3 CASs. In order to facilitate the response, ment of Defense. Maturity models have since been de- these country directors were asked to submit a re- veloped and employed across a wide variety of sponse for only one country of their choice. management practices. A maturity model identifies key 2. Responses as a percentage of the total number characteristics that an evaluator expects to find as of questionnaires sent out. the process in question becomes more fully developed and integrated into related management activities. Appendix D The maturity model was chosen for this analysis be- 1. Statistical significance was tested at 95 percent cause it does not require a particular process to be em- confidence. ployed, only that the process chosen produces the 2. Statistical significance was tested at 95 percent desired characteristics. confidence. 82 REFERENCES African Development Bank, Asian Development IEG (Independent Evaluation Group, World Bank, Inter-American Development Bank, Eu- Bank; formerly OED). 2005a. Country Assis- ropean Bank for Reconstruction and Devel- tance Evaluation (CAE) Retrospective: An opment, the World Bank, and the OED Self-Evaluation. IEG Study Series. Wash- Development Assistance Committee of the ington, DC: World Bank. OECD. 2004. “Managing for Development Re- ———. 2005b. The Effectiveness of World Bank sults.” Second International Roundtable Joint Support for Community-Based and -Driven Marrakech Memorandum, March. Development. IEG Study Series. Washington, Binnendijk, A. 2001. “Results-Based Management DC: World Bank. in the Development Co-operation Agencies.” ———. 2005c. Committing to Results: Improving Executive Summary. Paris, OECD/DAC. the Effectiveness of HIV/AIDS Assistance. IEG CDF Evaluation Management Group. 2003. To- Study Series. Washington, DC: World Bank. ward Country-Led Development: A Multi- ———. 2005d. “Outreach and Communications Partner Evaluation of the Comprehensive Strategy,” March 8, 2005. IEGKE. Washington, Development Framework (Synthesis Report). DC. Washington, DC: World Bank ———. 2004a. The Poverty Reduction Strategy Development Committee. 2002. “Better Mea- Initiative: An Independent Evaluation of the suring, Monitoring and Managing for Devel- World Bank’s Support Through 2003. IEG opment Results.” DC2002-0019. Washington, Study Series. Washington, DC: World Bank. DC. ———. 2004b. Evaluation Capacity Development: ———. 2000. “Supporting Country Develop- OED Self-Evaluation. June. Washington, DC. ment: World Bank Role and Instruments in ———. 2004c. 2004 Annual Report on Opera- Low- and Middle-Income Countries.” DC2000- tions Evaluation. Washington, DC. 19. Washington, DC. ———. 2004d. Addressing the Challenges of GAO (U.S. General Accounting Office). 2004. Globalization: An Independent Evaluation Results Oriented Government: GPRA Has Es- of the World Bank’s Approach to Global Pro- tablished a Solid Foundation for Achieving grams. IEG Study Series. Washington, DC: Greater Results. GAO- 4-38. Washington, DC: World Bank. U.S. Government Printing Office. ———. 2004e. The CGIAR at 31: An Independent Hatry, H., E. Morley, S. Rossman, and J. Wholey. Evaluation of the Consultative Group on In- 2003. How Federal Programs Use Outcome ternational Agricultural Research. IEG Study Information: Opportunities for Federal Man- Series. Washington, DC: World Bank. agers. NAPA/IBM Center for the Business of ———. 2003. 2003 Annual Report on Opera- Government. Managing for Results Series. tions Evaluation. Washington, DC. Washington, DC: Center for the Business of ———. 2002. The World Bank’s Approach to Government. Global Programs. IEG Study Series. Wash- IDA (International Development Association). ington, DC: World Bank. 2004. “Measuring Results: Improving National OED (Operations Evaluation Department, World Statistics in IDA Countries.” Washington, DC. Bank). See IEG. 83 2 0 0 5 A N N U A L R E P O R T O N O P E R AT I O N S E VA L U AT I O N World Bank, 2005a. “Results Focus in Country As- Annexes.” Operational Policy and Country sistance Strategies: A Stocktaking of Results- Services. Washington, DC. Based CASs.” Operational Policy and Country ———. 2004b. “Country Assistance Strategy Services. Washington, DC. (CAS)/CAS Progress Report Guidelines.” Op- ———. 2005b. “Improving the Bank’s Analytic erational Policy and Country Services. Wash- and Advisory Services.” Operational Policy ington, DC. and Country Services. Washington, DC. ———. 2004c. “Reform of World Bank–Admin- ———.2005c. “Annual Report on Portfolio Per- istered Trust Funds: The Way Forward.” Report formance: Fiscal Year 2004.” Quality Assur- No. 29626. Washington, DC. ance Group. Washington, DC. ———. 2004d. “Methods for Monitoring Achieve- ———. 2005d. “Quality of Supervision in FY03- ments Made toward IDA Results Indicators: An 04 (QSA6)—A QAG Assessment.” Quality As- External Panel Review.” Washington, DC. surance Group. Washington, DC. ———.2003a. “Better Measuring, Monitoring, ———. 2005e. Global Monitoring Report 2005. and Managing for Development Results: Washington, DC: World Bank. Implementation Action Plan.” Operational ———. 2004a. “Implementation of the Agenda Policy and Country Services. Washington, on Managing for Results: Progress Report and DC. 84 IEG PUBLICATIONS Study Series 2004 Annual Review of Development Effectiveness: The Bank’s Contributions to Poverty Reduction Addressing the Challenges of Globalization: An Independent Evaluation of the World Bank’s Approach to Global Programs Agricultural Extension: The Kenya Experience Assisting Russia’s Transition: An Unprecedented Challenge Bangladesh: Progress Through Partnership Brazil: Forging a Strategic Partnership for Results—An OED Evaluation of World Bank Assistance Bridging Troubled Waters: Assessing the World Bank Water Resources Strategy Capacity Building in Africa: An OED Evaluation of World Bank Support The CIGAR at 31: An Independent Meta-Evaluation of the Consultative Group on International Agricultural Research Country Assistance Evaluation Retrospective: OED Self-Evaluation Debt Relief for the Poorest: An OED Review of the HIPC Initiative Developing Towns and Cities: Lessons from Brazil and the Philippines The Drive to Partnership: Aid Coordination and the World Bank Economies in Transition: An OED Evaluation of World Bank Assistance The Effectiveness of World Bank Support for Community-Based and –Driven Development: An OED Evaluation Evaluating a Decade of World Bank Gender Policy: 1990–99 Evaluation of World Bank Assistance to Pacific Member Countries, 1992–2002 Financial Sector Reform: A Review of World Bank Assistance Financing the Global Benefits of Forests: The Bank’s GEF Portfolio and the 1991 Forest Strategy and Its Implementation Fiscal Management in Adjustment Lending IDA’s Partnership for Poverty Reduction Improving the Lives of the Poor Through Investment in Cities India: The Dairy Revolution Information Infrastructure: The World Bank Group’s Experience Investing in Health: Development Effectiveness in the Health, Nutrition, and Population Sector Jordan: Supporting Stable Development in a Challenging Region Lesotho: Development in a Challenging Environment Mainstreaming Gender in World Bank Lending: An Update Maintaining Momentum to 2015? An Impact Evaluation of Interventions to Improve Maternal and Child Health and Nutrition Outcomes in Bangladesh The Next Ascent: An Evaluation of the Aga Khan Rural Support Program, Pakistan Nongovernmental Organizations in World Bank–Supported Projects: A Review Poland Country Assistance Review: Partnership in a Transition Economy Poverty Reduction in the 1990s: An Evaluation of Strategy and Performance The Poverty Reduction Strategy Initiative: An Independent Evaluation of the World Bank’s Support Through 2003 Power for Development: A Review of the World Bank Group’s Experience with Private Participation in the Electricity Sector Promoting Environmental Sustainability in Development Putting Social Development to Work for the Poor: An OED Review of World Bank Activities Reforming Agriculture: The World Bank Goes to Market Sharing Knowledge: Innovations and Remaining Challenges Social Funds: Assessing Effectiveness Tunisia: Understanding Successful Socioeconomic Development Uganda: Policy, Participation, People The World Bank’s Experience with Post-Conflict Reconstruction The World Bank’s Forest Strategy: Striking the Right Balance Zambia Country Assistance Review: Turning an Economy Around Evaluation Country Case Series Bosnia and Herzegovina: Post-Conflict Reconstruction Brazil: Forests in the Balance: Challenges of Conservation with Development Cameroon: Forest Sector Development in a Difficult Political Economy China: From Afforestation to Poverty Alleviation and Natural Forest Management Costa Rica: Forest Strategy and the Evolution of Land Use El Salvador: Post-Conflict Reconstruction India: Alleviating Poverty through Forest Development Indonesia: The Challenges of World Bank Involvement in Forests The Poverty Reduction Strategy Initiative: Findings from 10 Country Case Studies of World Bank and IMF Support Uganda: Post-Conflict Reconstruction Proceedings Global Public Policies and Programs: Implications for Financing and Evaluation Lessons of Fiscal Adjustment Lesson from Urban Transport Evaluating the Gender Impact of World Bank Assistance Evaluation and Development: The Institutional Dimension (Transaction Publishers) Evaluation and Poverty Reduction Monitoring & Evaluation Capacity Development in Africa Public Sector Performance—The Critical Role of Evaluation All IEG evaluations are available, in whole or in part, in languages other than English. For our multilingual selection, please visit http://www.worldbank.org/ieg THE WORLD BANK ISBN 0-8213-6521-5