WORLD BANK OPERATIONS EVALUATION DEPARTMENT c m 23763 2000-2001 Annual Report on Operations Evaluation Madhur Gautam Director-General, Operations Evaluation: Robert Picciotto Director: Gregory Ingran Manager: Victoria M. Elliot Task Manager: Madhur Gautam 2002 The World Bank http://ww.worldbank.org/oed Washington, D.C. Y 1 © 2002 The International Bank for Reconstruction and Development / The World Bank 1818 H Street, NW Washington, DC 20433 All rights reserved. OPERATIONS EVALUATION DEPARTMENT ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVAL UA TION The Operations Evaluation Department (OED) is an independent unit within the World Bank; it reports directly to the Bank's Board ofExecutive Directors. OED assesses what works, and what does not; how a borrower plans to run and maintain a project; and the lasting contribution of the Bank to a country's overall development. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank's work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. The findings, interpretations, and conclusions expressed here are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. The World Bank cannot guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply on the part of the World Bank any judgment of the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is copyrighted. No part of this work may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or inclusion in any information storage and retrieval system, without the prior written permission of the World Bank. The World Bank encourages dissemination of its work and will normally grant permission promptly. For permission to photocopy or reprint, please send a request with complete information to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, USA, telephone 978-750-8400, fax 978-750-4470, www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, World Bank, 1818 H Street NW, Washington, DC 20433, USA, fax 202- 522-2422, e-mail pubrights@worldbank.org. CONTENTS v Acknowledgments vii Foreword xi Abbreviations and Acronyms 1 1. Introduction 2 The Bank's Evaluation and Control System 2 The COSO Framework 5 2. Control Environment 6 Selectivity and Strategic Focus 6 Matrix Management 7 Overload 7 Internal Culture 8 Aligning Incentives with Results 9 3. Risk Assessment 9 Project Design 10 Project Implementation 11 Beyond the Project Level 12 Partner Risks 13 4. Control Activities 13 Operational Policies and Strategies 14 Quality Assurance 15 Quality Assurance for ESW 15 Project Self-Evaluation 16 Corporate Self-Evaluation 16 Evaluation for New Initiatives 17 5. Monitoring 17 Portfolio Monitoring 17 Monitoring and Evaluation 18 Corporate and Sector Scorecards 18 Inspection Panel 21 6. Information and Communication 21 Information Systems 22 Knowledge Management 22 Creating Knowledge 22 Imparting Knowledge: Training 25 7. The OED Agenda 25 OED Work Program and Outputs 26 Harmonizing Evaluation Methodology 27 OED Outcomes and Impact 27 Reach, Quality, and Usefulness: Results from Client Surveys 30 Implications for OED iii 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION 33 8. Conclusions and Recommendations 33 The Evaluation and Control Framework 35 OED Agenda 35 Recommendations 37 Attachment 1: Management Response 41 Annexes 41 A- The Internal Control-Integrated Framework (COSO) 43 B: Fifth ICR Process Review 55 C: Evaluation Capacity Development 57 D: OED Knowledge Management Activities 59 E: Outreach, Dissemination, and Publications 61 F: FY01 OED Summary Work Plan and Results Framework (as.of Q3, FY01) 63 G: OED Client Surveys 67 H: The Influence ofThree OED Evaluations: A Tracer Study 69 I: Chariman's Statement, Committee on Development Effectiveness (CODE) 71 Endnotes Boxes 3 1.1 Components of the COSO Framework 11 3.1 Trading One Risk for Another? 26 7.1 OED's Renewal Strategy (1997) Figures 28 7.1 Relevance and Influence of OED Evaluations 44 B.1 ICR Quality (FY95-00 Exits) Tables 43 B.1 Annual Completion Report and Audit Output, FY96-00 44 B.2 ICR Quality Ratings (FY97-00 Exits) 45 B.3 Exemplary ICRs 47 B.4 Summary of Overall and Net Outcome Ratings Changes (PSR to ICR, ICR to ES, and ES to PAR), FY96-00 Exits 48 B.5 Trend in Outcome Rating Discrepancies, FY96-00 Exits (percent) 48 B.6 Trends in Sustainability and Institutional Development Ratings Changes, FY96-00 Exits (percent) 49 B.7 Summary of Occurrence of at Least One Rating Change (FY96-00 Exits) 50 B.8 Project Audit Priority and Performance Ratings from ES to PAR (FY96-00 Exits) 51 B.9 ICR-ES Ratings Summary, by Evaluation Delay (FY96-00 Exits) 51 B.10 Ratings Changes by Audit Delay, ES to PAR (FY96-00 Exits) 51 B.11 Evaluation Delay, ICR to ES (FY96-00 Exits) 52 B.12 Summary of Outcome Ratings Changes, CAI versus ILI, FY00 53 B.13 Comparison by ICR Styles 63 G.1 Summary of Surveys and Response Rates iv ACKNOWLEDGMENTS This report was prepared by a core team are greatly appreciated. William Hurlbut and consisting of Madhur Gautam and Deepa Caroline McEuen provided editorial support. Chakrapani, under the overall guidance Annisa Cline-Thomas, Julia Ooro, and Parveen of Victoria Elliott. Other members of the AROE Moses provided administrative support. team were Diana Qualls, Zamir Islamshah, Mita The study was published in the Partnership and Marra, and Laura Raney. Contributions were also Knowledge Group (OEDPK) by the Outreach made by Ruchira Banerjee-Corcoran and Oliver and Dissemination Unit. The task team includes Rajakaruna. The AROE team would like to Elizabeth Campbell-Pag6 (task team leader), acknowledge the timely inputs and guidance of Nils Caroline McEuen (editor), and Juicy Qureishi- Fostvedt, and the assistance of Rebecca Alfaro- Haq (administrative assistant). Flores, William Battaile, Nischint Bhatanagar, Patrick Grasso, Pierre-Joseph Kingbo, Patricia Laverley, Kevin Lumbila, Alex McKenzie, and Director-General, Operations Evaluation: Maria Mar at various stages of the report. The Robert Picciotto report benefited from substantial input from Director, Operations Evaluation Department: many colleagues in OED and many parts of the Gregory Ingram Bank, including the Regions, Networks, and other Manager, Corporate Evaluation and Methods: Bank units including OPCS, DEC, WBI, QAG, Victoria Elliott QACU, and CTR. The responses to OED sur- Task Manager: Madhur Gautam veys from many individuals across the Bank Task Team: Deepa Chakrapani V FOREWORD The Annual Report on Operations Evaluation (AROE) fulfills OED's statutory obligation to assess the adequacy of the Bank's operations evaluation system. Following a suggestion by the Vice President & Controller, this year's AROE uses, for the first time, the Internal Control- Integrated Framework (COSO), adopted by the Bank with a view to contributing to a coherent, Bank-wide, integrated risk review process. The COSO framework is designed to identify Group now provides real-time quality assessment, areas of risk in an organization's internal controls the Quality Assurance and Compliance Unit seeks system. Applying this framework, the AROE to improve compliance with safeguard policies, and notes key areas of strength and recent progress but Regional quality assurance teams provide ex-ante gives particular emphasis to risks and vulnerabil- quality control. In parallel, the Internal Auditing ities. Thus, the AROE does not claim to present Department has improved its oversight of man- a balanced scorecard of the overall quality of agement controls, and a new Department of the development effectiveness risk assessment Institutional Integrity has been set up to investi- system. Instead, it gives pride of place to the gate allegations of fraud and corruption. remaining challenges that face the Bank as it goes These additions to the Banks evaluation and about strengthening its development risk con- control activities are complemented by an inde- trol and evaluation follow-up system. pendent Inspection Panel, which provides a voice Considerable progress has in fact been achieved. to local communities potentially affected by Bank The reforms of the last few years have increased interventions and by the independent Operations the Bank's development rewards through stronger Evaluation Department (OED), which has re- country focus, improved responsiveness, and newed its processes and practices to take account higher operational quality. Evaluation and con- of changes occurring in the Bank. These activities, trol systems have contributed to the improve- while adding to the responsibilities of already ment as they evolved to fill in many earlier gaps highly stressed operational staff, have delivered in development effectiveness oversight and qual- significant corporate benefits by strengthening ity assurance. Specifically, the Quality Assurance accountability and organizational learning. vii 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION Changes in the Bank's product mix-partic- The AROE recommends that action be taken ularly toward programmatic lending, knowl- on four broad fronts to ensure the relevance edge services, collaborative programs, and global and effectiveness of the Bank's evaluation and public goods-are continuously generating control processes. Of these, three concern actions new demands for evaluation and control. Thus, by management and one requires action even with the major strides already made, by OED: challenges remain. Many of the Bank's internal * Enhance the evaluation and controlfamework incentives and processes need to be realigned to encompass emerging priorities and initiatives: to promote desired organizational behaviors To keep pace with the Bank's emerging oper- in line with the new Bank's vision and opera- ational modalities, management should tional approaches. In particular, the still un- ensure that coherent evaluation processes balanced matrix structure has not been able are in place for the new Knowledge Bank; to close the gaps between evolving corporate develop guidelines on evaluation standards for priorities and country strategies. The assessment partnerships; and mainstream evaluation and management of development risks need capacity development. further strengthening-in lending operations, * Complete the process ofconverting and updating Country Assistance Strategies (CASs), and the Operational Policies: Maintaining up-to- Sector Strategy Papers (SSPs). With the objec- date and clear policies and guidelines is im- tive of becoming a "Knowledge Bank," non- portant to minimize development and lending activities have grown rapidly in recent reputational risks. Even though the conver- years, but without an agreed evaluation frame- sion process is complex and time-consuming, work, with the notable exception of economic it is important to establish a timeframe and dear and sector work, for which there is an ongoing norms for public consultations to guide the program of self-evaluation based on an estab- process. Updating policies, especially safe- lished methodology. guards, and completing ICR guidelines for The process of recasting and updating poli- new instruments, such as APLs and LILs, need cies, especially for adjustment lending and social to be done expeditiously. safeguard policies, has been complex and chal- * Ensure that CASs are consistent with priorities lenging. The new demands posed by the as set out in SSPs:An important but avoidable CDF/PRSP process and the HIPC initiative source of development risk is the continuing present considerable challenges to the evaluation gap between the corporate sectoral priorities and control functions. More generally, the in- and country programs. Actions can be taken creasing involvement of partners in the Bank's at three levels. First, in the development work offers significant benefits, but also poten- of SSPs, sector boards should be held tial risks. Progress has been substantial with re- responsible for including explicit implemen- gard to partnerships receiving support from the tation plans, providing strategic directions Bank's Development Grant Facility, but criteria based on the Bank's comparative advantage, for evaluating other partnerships remain to be and giving guidance on managing tensions developed. between client and Bank priorities. Second, Monitoring and evaluation, a long-standing Regional sector managers should be held concern reflecting weakness in borrowers' evalu- accountable for ensuring the consistent ation capacities, has become even more important. application of sector priorities in operational In CDF and PRSP countries, initiatives are work. Third, CASs should include indicators underway to strengthen borrower monitoring to monitor progress in priority sectors, and and statistical systems. It is important now to country directors should be held accountable complement these efforts by mainstreaming eval- for the consistency of CASs with corporate uation capacity development. priorities. viii FOREWORD * OED should strengthen its methods and pro- transparency of the data sources, the evalua- cedures: To keep abreast of changes in the tion approach, and assumptions used in its Bank and enhance its influence, primarily evaluations. through learning, on the Bank's development effectiveness, OED should review and improve its procedures for evaluating adjust- ment lending and ensure new instruments are fully covered. OED also needs to review and simplify project evaluation instruments Robert Picciotto and test and refine the country evaluation Director-General methodology. It should further increase the Operations Evaluation ix ABBREVIATIONS AND ACRONYMS AAA Analytical and Advisory Services APL Adaptable Program Loan ARDE Annual Review of Development Effectiveness AROE Annual Report on Operations Evaluation ARPP Annual Review of Portfolio Performance BP Bank Policy CAE Country Assistance Evaluation CAI Core Accountability ICR CAS Country Assistance Strategy C & P Corporate and Process CDF Comprehensive Development Framework CIF Country Information Form CODE Committee on Development Effectiveness COSO Internal Control-Integrated Framework CPIA Country Policy and Institutional Assessment CPPR Country Portfolio Performance Review CSA Control Self-Assessment CTR Controllers' Vice-Presidency DEC Development Economics Vice-Presidency DGF Development Grant Facility DGO Director-General, Operations Evaluation ECD Evaluation Capacity Development ECG Evaluation Cooperation Group ECWG Evaluation and Control Working Group ED Executive Director EDP Executive Development Program ERM Enterprise Risk Management ES Executive Summary ESSD Environmentally and Socially Sustainable Development ESW Economic and Sector Work HIPC Heavily Indebted Poor Country IAD Internal Audit Department ICR Implementation Completion Report IDG International Development Goal (now the Millennium Development Goals) IFC International Finance Corporation ILI Intensive Learning ICR ISR Information Systems Renewal LIL Learning and Innovation Loan M&E Monitoring and Evaluation MAR Management Action Record MD Managing Director MDG Millennium Development Goal MIS Management Information System OD Operational Directive OED Operations Evaluation Department OP Operational Policy xi 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION OVP Operations Vice Presidents OPCS Operations Policy and Country Services PAD Project Appraisal Document PAR Performance Audit Report PRSP Poverty Reduction Strategy Paper PSR Project Supervision Report QACU Quality Assurance and Compliance Unit QAG Quality Assurance Group QBR Quarterly Business Review QEA Quality at Entry Assessment QER Quality Enhancement Review QET Quality Enhancement Team RSB Research Support Budget RVP Regional Vice-President S&T Sector and Thematic SSP Sector Strategy Paper WBI World Bank Institute xii INTRODUCTION The Annual Report on Operations Evaluation (AROE) assesses the adequacy of the Bank's operations evaluation system and performance management practices.' The Bank adopted the Internal Control-Integrated Framework (also known as the COSO framework, described in Annex A) in 1995 to guide the establishment of effective internal controls. One of the objectives of a control system is to provide a reasonable level of assurance to management and the Board of Directors regarding the efficiency and effectiveness of operations. Evaluation is a part of the overall control framework, and the Operations Evaluation Department (OED) has the mandate to assess the adequacy of the Controller's Vice-Presidency (CTR) and the the Bank's evaluation processes. This report assesses annual report on the World Bank's internal the status of those processes, broadly referred to controls from the Internal Audit Department as the Bank's evaluation and control system for de- (LAD) on the efficiency and effectiveness of the velopment effectiveness, and makes recommen- Bank's internal control processes in managing dations for its improvement. This year, for the first operations and business risks. The financial and time, the AROE uses the five interrelated COSO business operational control issues discussed components as its organizing principle. The Bank's in these reports also critically affect the overall actions in response to evaluation findings are development effectiveness of Bank operations. tracked by OED. This AROE is based on information from The focus of this report is the development a wide variety of sources, including reports effectiveness of the Bank's operational work from OED, the Bank's Quality Assurance (lending and nonlending services, research, and Group (QAG), other oversight units, and recent training). As such, it is concerned with evaluation task force reports. It also draws on interviews and control processes that directly affect the with selected staff and managers involved in outcomes of Bank development activities. (or affected by) evaluation and risk management It complements the annual COSO report from activities. 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION The Bank's Evaluation and anti-corruption policies and strategies, and the new Control System Department of Institutional Integrity investigates The World Bank is improving its performance allegations of fraud and corruption. despite an increasingly challenging environment By and large, these arrangements provide and a mandate that has both broadened and deep- an adequate control framework for project ened over the past decade. The 2000 Annual lending. But the Bank is repositioning itself Review ofDevelopment Effectiveness (ARDE 2000)2 to become a stronger "Knowledge Bank." And described the progress achieved in Bank lending it faces new and more complex demands at performance since the initiation of the Strategic the country and global levels. With increasing Compact. The ARDE also found that the Bank's transparency and a rapidly changing authorizing agenda is increasingly ambitious and that, in environment, the pressure to demonstrate results allocating Bank resources, it must manage tensions has never been greater, especially in reducing between (a) client and corporate priorities, poverty. To manage development and reputa- (b) global prescriptions and local conditions, tional risks, it is imperative to ensure the ade- (c) country performance and poverty incidence, quacy and sufficiency of the Bank's controls over and (d) short-term efficiency and a long-term, the entire results chain. The Comprehensive holistic vision of development. The ARDE con- Development Framework (CDF), the Poverty cluded that the Bank can further improve its Reduction Strategy Paper (PRSP) initiative, and development effectiveness and reduce current the Heavily-Indebted Poor Countries (HIPC) levels of stress by being more selective; making debt-reduction program have improved the better use of partnerships; and adapting its orga- Bank's support for country ownership and en- nizational structure, business practices, and hanced its capacity-building and aid coordina- operational instruments. tion roles in low-income countries. A revised The Bank has a long history of evaluation and approach toward middle-income countries is control for its operational work, but new chal- being developed. Interest is growing in the Bank's lenges and shifting priorities pose novel and contribution to collaborative programs focused unprecedented risks. The current system is on the provision of global public goods. These appropriately grounded in self-evaluation activi- changes have intensified the need for new skills, ties. These, together with independent evalua- instruments, processes, and relationships, as well tion, fiduciary functions, and the formal guidelines as for a revamping of the risk-management and procedures for staff, constitute the overall framework. framework. Evaluation and control functions have changed in response to the Bank's evolving The COSO Framework operating environment and mandate. IAD has im- The COSO model has been adopted by the proved its oversight of management controls. The Bank as the appropriate standard for the assess- Inspection Panel now provides a voice to local ment of the Bank's internal control system. It communities that may be adversely affected by is broadly serviceable for all aspects of risk Bank-supported activities. Other enhancements management, including development effectiveness. include the establishment of QAG, entrusted (Box 1.1 gives a brief description of the with real-time assessment of operational work; COSO components. A detailed description is the Quality Assurance and Compliance Unit in Annex A.) Adapting the framework from (QACU), which seeks to improve compliance financial objectives (profitability and preser- with safeguard policies; and the Regional quality vation of financial assets) to development assurance teams, which are charged with ex-ante effectiveness (and, hence, the Bank's reputational assurance on all aspects of operational quality. assets) is conceptually straightforward. In practice, The Oversight Committee on Fraud and Cor- however, the increased reliance on partnerships ruption oversees the implementation of the Bank's in Bank operations involves special challenges. To 2 INTRODUCTION Box 1.1 * The control environment provides an effective organizational nical, economic, and financial standards; and suit country and incentive structure for the promotion of desirable be- social and institutional circumstances and reflect their havior. It is focused on results and on acting in consonance priorities and needs. with well-defined, clearly articulated, and well-communi- * Monitoring is essential to ensure that control systems are cated objectives. adequate and effective and are properly aligned to ensure * Risk assessment mechanisms identify, analyze, and man- achievement of operational and corporate objectives. age emerging problems at each stage of operations to * Information and communication encompass the other four ensure timely achievement of relevant development components, and access to timely and accurate infonmation objectives. and effective communication up, down, and across the * Control activities ensure that management directives are car- organization constitute a necessary condition for the effi- red out and operations comply with established standans and cient conduct of the Bank's business and the discharge of staff policies, consistent with Bank priorities; meet agreed tech- responsibilities. ensure accountability, the distinctive respon- among partners and in developing member sibility of partners must be delineated and countries. attribution of development outcomes assigned Chapters 2-6 of this report are organized among them so that sources of control failure around the five components of the COSO frame- can be identified and remedied. This is a tough work. Chapter 7 discusses the OED agenda, and challenge given the weaknesses of monitoring Chapter 8 provides conclusions and recommen- and evaluation arrangements and the often in- dations. Attachment 1 provides the management adequate capacity for monitoring and evaluation response to the OED recommendations. 3 2 CONTROL ENVIRONMENT The institutional environment is the foundation of a sound control framework, because the effectiveness of control activities depends on the accountability and governance of the organization as a whole. The Bank's control environment can be seen as having four dimensions: its mandate, structures, functions, and internal culture. The Bank's fundamental mandate, once somewhat taken for granted, has been challenged-and thereby refined and sharpened---over the past few years to reach a clear focus on poverty reduction. The organizational arrangements, with oversight from the Board of Directors and senior management, constitute the structural increasingly complex setting. Management's con- dimension of the Bank's control framework. The sistent emphasis on improving the quality of basic chain of accountability is well defined and Bank operations and efforts to improve institu- has been tested over time. This structure is com- tional learning through improved self-evalua- plemented by a variety of specialized functions tion, as noted in different parts of this report, designed to improve controls. QAG, LAD, the have raised the control consciousness of staff. As Inspection Panel, OED, and QACU form an a result, both lending and nonlending activities evaluation "family" that has taken on new or continue to register improving performance, broadened functions for ex-ante, real-time, and meeting or exceeding the targets set by man- ex-post quality assurance and evaluation. Finally, agement. The increasing complexity of Bank the organization's culture-its values, incentives, operations and the significant changes of the and behaviors-determines the nature of inter- past few years have increased stress on staff and actions among staff and between management and stretched institutional capacities, including that staff, and is critical for controls to function of the evaluation framework. To ensure that effectively. progress is sustained, some concerns warrant These elements of the Bank's control envi- close attention. This chapter focuses on five ronment have served the institution well in an emerging issues in the control environment. 5 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION Selectivity and Strategic Focus environment. The resulting structure has been a The Bank's evolving operating environment has subject of debate since its introduction. It was widened its mandates, resulting in an expansion adjusted in October 1999 to better balance net- of activities. Recent evaluations (ARDE 2000; Aid work and Regional resource allocation and to Coordination') have observed that a lack of clarify roles and responsibilities. selectivity can pose development risks. An Matrix management has helped increase the assessment of the Strategic Compact found that country focus, improve the technical capacity the recent initiatives have resulted in unantici- and response capability (e.g., financial crises, pated changes in the allocation of budget, natural disasters, and post-conflict situations) of including lower than expected expenditures on the Bank, and reduced the fragmentation of tech- economic and sector work (ESW) and lending, nical staff across Regional administrative units. which may affect sustainability of the gains that A continuing concern, though, is the role of the have been made.2 A 2000 CAS retrospective networks in quality assurance, which is largely identified a lack of selectivity at the country level limited to advisory and support functions. An- as well. Few Country Assistance Strategies (CASs) chor units and sector boards have insufficient use selectivity as an analytical tool to focus on par- budgets and inadequate authority to influence the ticular sectors or themes, and few indicate why quality of individual operations and to ensure given instruments were chosen. This can inhibit alignment with the Bank's sectoral strategies and the implementation of a CDF approach focused priorities. This presents a challenge in aligning on core competencies and comparative advantage. the Bank's corporate priorities and commitments This said, the selectivity and results orientation at the sectoral and thematic levels with individ- of CASs have improved considerably, and the ual country assistance programs (as also noted in CDF and PRSP initiatives have begun to address an assessment of the Strategic Compact). issues of strategic selectivity among donors. Man- The new development approach adopted by the agement has recently taken additional steps toward Bank is thematic and crosscutting, demanding addressing selectivity with the establishment of coordination across sectors. In general, however, a new Management Committee "to provide professional groups continue to be compart- corporate guidance and to take up issues of mentalized along sectoral lines, partly because of priorities and strategy." The Committee's main the limited role played by the network councils. role is to align corporate strategies, ensure insti- For example, in the area of water resources, most tutional selectivity and development effective- operations continue to focus on individual proj- ness, and manage tensions between corporate ects and issues within traditional water subsectors, priorities and country programs. Bank manage- even though the Bank's strategy emphasizes com- ment has identified operational emphases grouped prehensive water resource management. This into Corporate Advocacy Priorities, Global Goods weak integration at the sector-wide level means Priorities, and Core Competencies, and the that operational integration has to take place at proposed FY02 budget has been prepared within the country level. To achieve this, the function- this framework. A new Partnerships Council has ing of country teams needs to be improved. An also been formed to provide oversight to the assessment of the Compact found that country diverse and proliferating list of collaborative teams are not seen to be attending sufficiently to arrangements that have been created over the the sector strategies or providing strong enough past few years. feedback for input into the network/sector board strategies. The increasing number of country de- Matrix Management partments has also intensified the challenge of fos- Matrix management has added new dimensions tering cross-country learning.5 to the Bank's accountability structures. It Management is aware of the issues and has has altered the responsibilities of managers and recently reconfigured the Matrix Steering Group staff, with implications for the internal control to better understand the functioning of the 6 CONTROL ENVIRONMENT matrix and address remaining challenges. The overall system to utilize individual findings for group will focus on ensuring that all sector boards, scaled-up, higher-level analyses. country teams, and sector units-including their The evaluation and control units also need to staff in field offices-function effectively, and it improve their ability to communicate the will also provide the needed support through rationale for current evaluation activities to the reforms in budgeting and human resource staff. As noted by the Evaluation and Control systems. The group's FY02 program is focused on Working Group, many staff may not fully convergence of best practices for sector and coun- understand the mandates and activities of the try management, country teams, and sector different evaluation and control units (OED, boards, and expects the Operations Vice Presi- QAG, QACU, Operations Policy and Country dents (OVPs) to provide the correct incentives to Services [OPCS], Inspection Panel, IAD, managers and staff in support of those objectives. Investigation Unit, and Regional quality assur- Management is also in the process of rationaliz- ance teams). There is also the need to commu- ing the span of control for Regional sector man- nicate to managers and staff the proven value agement units. of evaluation and control activities in their contribution to improved quality, better out- Overload comes, and reduced corporate risks.6 Senior Credible and efficient evaluation is an essential management leadership is critical to ensure that component of the control environment. A per- evaluation and control activities contribute to ception of excessive or poorly planned evaluation organizational learning. and control activities can weaken the desired impact on accountability and stunt institutional Internal Culture learning. Growing concern about "overload" led The culture of an organization is important for management to establish in 1999 an Evaluation the effective functioning of controls. Trans- and Control Working Group to address the parency and trust influence the interaction issue. Acting on its recommendations, the Bank between staff and managers, staff perceptions of has made substantial progress in better coordi- decisionmaking, and the identification of risks nating evaluation and control units (QAG, OED, through honest and effective feedback. Based LAD) to minimize overlap. Rules are in place to on recent Control Self-Assessments, the Con- avoid evaluating the same activity twice in the troller notes continued staff perceptions that same year. OED has reduced the number of its those in leadership positions do not consistently major products, while enriching their content. set a good example. Feedback from Executive QAG has implemented changes in its sample Development Program (EDP) graduates at the selection procedures to minimize impact on Strategic Forum also raised the issue of internal individuals and proposes to broaden its quality culture and the lack of candid interaction assurance role with the networks, while reduc- between managers and staff. Management has ing the frequency of its lending and supervi- responded by making internal culture a priority sion quality reviews. At the same time, explicit concern, and the President has made personal resourcing of staff time for evaluation and con- efforts to elicit candid feedback directly from trol activities has not yet been provided, al- staff A new office of Internal Communication though the ongoing budget reform process is has been established to promote direct links expected to address this issue. between managing directors and working-level The perception of overload continues despite staff. Better communication, greater clarity on these changes. In response, the evaluation and operational priorities, clear lines of responsibil- control units plan to further review and coordi- ity, and appropriate incentives to support desired nate their activities and develop an integrated organizational behaviors should help promote a evaluation and control program. This would help culture that allows for honest feedback and bet- improve systemic efficiency and the ability of the ter morale. 7 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION Aligning Incentives with Results the formal personnel evaluation/incentives and The results orientation principle of the CDF and budget allocation criteria could be more closely the centrality of poverty reduction in all Bank tied to performance. The recasting of the policy activities demonstrate the Bank's commitment framework needs to keep pace with the Bank's to a results-based development paradigm. The needs, especially with regard to safeguard policies. focus on performance measurement under the Sector Strategy Papers (SSPs) have started to lay Strategic Compact has also encouraged progress out specific program and success benchmarks, toward results-based management. For example, and it is important to make sure this is consistently human resource policies were significantly re- done. The evaluation of the CDF and PRSP will vised and a new Overall Performance Evaluation be made more challenging by their focus on system introduced to help increase the focus on process rather than measurable outcomes. results. Significant progress has been made as a re- A key area for improving the Bank's focus on sult of the close attention by management to results, and currently a source of vulnerability, is monitoring of work programs. For example, the in transparently setting out verifiable performance Dashboard has been added to the management indicators for progress against poverty. The Bank's information system to track expenditures and adoption of the Millennium Development Goals deliverables. These steps have improved the con- (MDGs) heightens the need for improvements in trol environment, although the transition to a this area. The 2000AnnualReview ofDevelopment fully results-oriented institution is not yet com- Effectiveness (ARDE), supported by QAG and plete. The Banks internal incentives and processes other Bank data, has identified the need to link need to be further aligned with the Bank's vision the Bank's objectives and policies more explicitly and operational approaches. As identified by an to poverty reduction and to set up monitorable assessment of the Strategic Compact, the link to outcomes within CASs, SSPs, and project development results could be made stronger and appraisal Documents (PADs).7 8 3 RISK ASSESSMENT The identification and analysis of risks that could jeopardize the achievement of organizational objectives is central to the objective of the evaluation and control framework. Systematic risk assessment provides the basis for managing risks. The increasing complexities of the Bank's operating environment and the demandingness of the new operational initiatives make rigorous and continuous risk-assessment processes critical. The Bank's assessment of development effective- mitigation steps at the business-unit level. The ness risks improved considerably with the estab- Bank has adopted the Control Self-Assessment lishment of QAG, and more recently QACU and (CSA) approach in implementing COSO through the Regional quality assurance teams. The frag- two main processes. The CSA workshops have mentation of risk-management activities led man- a strong focus on operational teams and behav- agement to commission the Risk Management ioral risks, and their potential impact on business Task Force in 2000. The work of the Task Force objectives. In finance, the Enterprise Risk Man- highlighted current weaknesses and offered sen- agement (ERM) tool is being used, and the sible recommendations. A follow-up working methodology is planned to be piloted in the group is currently working on operational risk Regions. ERM ensures a comprehensive and management, and a report is due shortly with rec- rigorous approach to financial risk management ommendations on strengthening the operational and ensures that risks are mitigated, with ac- risk-management framework. countabilities clearly assigned. A corporate risk The new Management Committee will also management structure is thus taking form, but serve as the Risk Management Committee, as some aspects need additional attention. recommended by the Risk Management Task Force, and will address all types of risks, includ- Project Design ing those to the Bank's reputation, development For informed decisionmaking, management and effectiveness, and financial and business opera- the Board need a good assessment of project risks, tions. In addition, through the quarterly business their likelihood and consequences, and bench- review (QBR), the Vice-Presidential units are marks to compare and aggregate risks across proj- expected to provide an assessment of risks and ects. QAG quality at entry assessments (QEAs) 9 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION find that the overall project quality at entry has this function for approved projects and QACU steadily improved from 82 percent satisfactory or helps with regard to safeguard issues. In gen- better in CY97 to 89 percent in CY99. But it notes eral, ex-ante risk management is heavily de- that risk assessment remains one of its weakest as- pendent on self-assessment. The networks offer pects. At the level of individual operations, the a potentially objective source of advice to ensure quality of risk analysis has not improved over the that Regional strategies and operations are con- past three years. Both QAG and the Risk Man- sistent with corporate priorities and policies, but agement Task Force point out that risks are not their role is currently limited. New organiza- candidly discussed in the PAD. Thus, although tional solutions are needed to ensure both ade- the use of log-frames has increased, with a whole- quate risk management and the independence of some impact on project design, it has not as yet networks' oversight functions without adding had a significant impact on the quality of risk as- to internal transactions cost. One example is the sessments. Past QAG reviews have found risk as- centralized unit for safeguard compliance as- sessments to be weak in adjustment operations. sessment in the International Finance Corpora- While the latest QEA shows that the risk assess- tion (IFC). Good practice examples from other ment ratings for adjustment operations (79 per- elite public sector organizations and private com- cent) are now higher than for investment lending panies should be explored. (69 percent), QAG cautions that with the short An innovative approach to risk profiling is time available between tranches, the ability to QAG's portfolio-at-risk ratings. These ratings adapt to changing circumstances can often be have helped to focus the attention of Regional and substantially compromised.' Also, despite the im- corporate management on selected projects and provements found in the successive QEAs, the ad- sectors. But the reliability of these ratings de- justment lending record points to the need to pends on the quality of the project supervision re- explore a more consistent risk management frame- ports, and-as noted below-these self-evaluation work, as well as the increasing complexity of ad- reports are often overoptimistic. QAG's periodic justment operations. Given the rising importance assessments of project quality should, in due of adjustment lending in the Bank's portfolio, course, help to bridge the gap, provided Regional these findings warrant closer attention to risks in managers take action on QAG recommendations. adjustment lending operations. The Regional Quality Enhancement Teams Project Implementation (QETs) aim to address this problem by focusing The supervision process, as captured in the proj- on the identification of risks at the project con- ect status report (PSR), is the Bank's main means cept and appraisal stages. Regional Safeguards of identifying and managing development risks Coordinators have been appointed to deal with during project implementation. QAG's assess- safeguard issues and, with assistance from QACU, ment of supervision quality in FY00 (QSA4) they identify special-risk projects (projects with found a steady improvement in the quality of su- potential reputational risks to the Bank). But pervision since FY97, with an increased focus on there are no standard criteria for these assess- development effectiveness and sustainability, ments. Each Region has its own system.2 While and increased attention to safeguard and fidu- attention to safeguard and fiduciary compliance ciary requirements. Almost 92 percent of the dimensions of risk has improved considerably, sample was rated satisfactory or better on over- the assessment of the broader development risk all supervision quality. This represents a signif- of the project failing to achieve its objectives or icant strengthening of the risk-assessment of being misaligned with corporate priorities re- framework. At the same time, QSA4 found the mains weak. realism of supervision reporting and the track- A separation of the assessment of compliance ing of development outcomes to be the weakest from that of program implementation is crucial aspects of supervision, with no improvement for risk management. QAG effectively performs since FY97.3 An analysis of the differences 10 RISK ASSESSMENT between PSR and Implementation Completion teria for risk identification and/or assessment. As Report (ICR) assessments of project performance noted in ARDE 2000, OED country evaluations (in Chapter 4) is consistent with this finding. have found a pattern of overoptimism about QSA4 identified weaknesses in Bank inputs and the receptivity of government to Bank advice, its processes that can contribute to development willingness to undertake reforms, and its capac- risks. Supervision budgets were found to be in- ity to implement recommended measures. Inad- adequate for 25 percent of the sample by QAG equate tools for risk analysis have contributed to panelists, and almost 40 percent of the TTLs sur- a tendency to underestimate risk and overestimate veyed expressed the same concern. QSA4 finds the influence of proposed risk mitigation meas- that this is partially the result of the failure of ures. Based on evidence from Country Assistance management to provide sufficient budgetary re- Evaluations (CAEs) and the CAS retrospective, sources for the complexity, degree of difficulty, ARDE 2000 also notes that CASs rarely state the and array of problems faced by supervision strategic rationale for choosing one instrument teams. This suggests that some aspects of su- over another. pervision may not be fully covered (see box 3.1). The introduction of the SSP has improved The assessment also found some dissatisfaction corporate strategies at the sector and thematic on the part of TTLs with management support levels. But weaknesses in self-evaluation have per- for supervision, with about a third of the TTLs sisted, inhibiting an accurate assessment of strate- indicating they were not getting timely deci- gic risks. An ongoing stocktaking of SSPs by sions or guidance by their managers at critical OPCS has identified some best practice examples, stages. There is no independent field verification but found their analytical quality to vary consid- for PSR findings, including those on safeguard erably. And while most SSPs build on general and fiduciary issues, either by Regional or net- lessons of experience and their integration of work quality teams. Filling the gaps in the real- OED sector evaluations is improving, few SSPs time risk assessment framework calls for a careful try to derive criteria for instrument choice from review of the cost-effectiveness of additional risk lessons of experience in the sector. To strengthen mitigation arrangements.4 the Bank's overall risk management framework, OED has recommended enhancing risk profiling Beyond the Project Level and classification, and stressed the need for the Countrywide risks can affect individual projects' Bank to explicitly address strategic risks and mon- development outcomes. According to a 2000 itor performance against strategic objectives. The CAS retrospective, slightly over a third of the failure of some strategies to identify the Bank's CASs analyze the impact of key risks satisfacto- comparative advantages in relation to partners rily and fewer than a third classify risks accord- and the lack of output and outcome performance ing to their perceived importance-high, medium, indicators were seen-as deficiencies. ARDE 2000 and low. There is no standard methodology or cri- notes that few SSPs thoroughly analyze country Box 3.1 Greater awareness of reputational risks associated with failure assessment of supervision qualityfound that 71 percent of the TrLs to comply with the "do no hann" policies and increased focus would use an incremental $10,000 for either more frequent mis- on fiduciary issues has made operational staff keenly aware of sions or longer field visits (37 percent) or to hire additional tech- the need to addressthese risks during supervision. But attention nical inputs (34 percent). Safeguard and fiduciary issues ranked to risks in other operational quality dimensions may need to be lower. This suggests that budget allocation decisions may be af- strengthenedLAsurvey of Task Team Leaders (TTLs) in GAG's FY00 fecting quality control on development risk matters. 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION strategies to identify risks in reconciling Bank has established criteria to guide decisions on priorities and client objectives. grant programs, and the process involves sector boards and networks. The Partnership Council Partner Risks provides support for more proactive manage- The transformation to a "full service" Bank; the ment of partnerships while ensuring that ac- need to make operations more effective, that is, countability and approval lie with individual putting the country at the center of development Regional Vice-Presidents (RVPs) and Manag- programs; and an increasing role of the Bank in ing Directors (MDs). The Bank has also recently global public goods have made partnerships and established criteria for engaging in partnerships. global programs an important part of the Bank's The DGF has made significant advances in work. These partnerships present significant addressing OED's previous recommendations. benefits, but also potential risks. These risks in- The DGF has moved to fill an important gap: clude reputational risk (being associated with a the need for a high-quality, independent evalu- partner engaged in activities or practices ation to draw strategic lessons. Six independent inconsistent with the Bank's policies or strategies); evaluations have been conducted in FY00 and inefficiency (costs of partnership are more than FY01, and evaluations for all major programs outcome benefits or resources are diverted to have been planned over FY02-04.5 Additional nonpriority areas); and conflict of interest (of up- steps are needed to address remaining risks in stream advisers with subsequent roles as con- governance and the implementation of exit cri- sultants or for procurement opportunities, or teria. The Partnership Council is in the process partnering with private sector or NGO groups of prioritizing and rationalizing the list of Bank in activities that are linked to the expansion of partnerships, and it is also developing criteria for their services or markets for their products). exit strategies. The monitoring and evaluation To better prioritize and manage rapidly ex- of outputs, outcomes, and impacts remains a panding partnership links, the Bank established challenge. Some of these issues will be reviewed the Development Grant Facility (DGF) in FY98 in the forthcoming OED evaluation of global and the Partnership Council in FY00. The DGF programs. 12 CONTROL ACTIVITIES Control activities are the policies, rules, and procedures established to make sure that management directives are carried out. They support actions to ensure the achievement of development objectives and to effectively address the risks identified through the risk-assessment processes. The Bank has long had manuals for staff conduct and for Operational Policies and procedures. The internal processes have been substantially revised and continue to evolve to keep pace with the new operating environment of decentralization and matrix management. Operational Policies and Strategies public consultations. Another challenging area The Bank's Operational Policies form the basic set has been the application of fiduciary and safeguard of controls to direct Bank operations. These poli- standards to adjustment lending, although sub- cies embody corporate priorities and provide stantial progress has been made on developing a guidance on the implementation of the Bank's de- consensus on the fiduciary side. The planned velopment strategy. The process of converting conversion and update of OD8.60 (in FY02) on Operational Directives (ODs) to definitive Op- adjustment lending is expected to address these erational Policies (OPs) and Bank Procedures issues comprehensively. Beyond the conversion of (BPs) began in 1993. By 1998, a majority of the ODs, 29 of the 71 OPs are considered "current."3 ODs had been recast, with the notable exception The 1998 review by OPCS recommended main- of six policies where the conversion was consid- taining and monitoring progress against time- ered to be complex.' Of these, the OD on envi- lines for the stages of the policy revision process. ronmental assessment has since been converted. This recommendation remains valid today. The three social safeguard policies are in the A common issue across several recent OED process of conversion, as are the policies and evaluations is the need to strengthen and clarify guidelines on adjustment lending and poverty operational policies.4 The 2000 ARDE notes that reduction.2 The conversion process for the social ambiguities or gaps in the policy framework have safeguard policies has been complex and slower made Bank staff, management, clients, and part- than expected, involving extensive internal and ex- ners uncertain about expectations and require- ternal interactions, but without clear norms for ments. The fact that important policies are not 13 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION up-to-date and well understood by staff constitutes tary basis, or for high-risk projects, better incen- a development effectiveness risk. While there has tives are needed for staff to identify a project as been notable progress in providing guidance to high risk.5 Stronger participation by the networks staff on safeguard policies (as noted in the next along with the Regions in monitoring the super- section), and work is also progressing on other vision of these projects would help to evaluate and policies, it is important for this work to quickly record lessons learned. culminate in the issuance of unambiguous policy In safeguards compliance, management has statements to guide operational work. recognized the need to strengthen oversight. Sub- The introduction of the SSPs in 1996 was stantial resources have been allocated to QACU. aimed at addressing the previous weaknesses in sec- All Regions have appointed Regional Safeguard tor strategy development. As ARDE 2000 notes, Coordinators, with dual reporting requirements early sector strategy documents evaluated by OED to the Regional Safeguards Directors (also the Re- lacked explicit plans for implementation, self- gional economically and socially sustainable de- evaluation had been weak, and they generally did velopment [ESSD] managers) and QACU. not provide strategic direction based on the Bank's Several steps have been taken to improve the comparative advantage. Few SSPs analyzed the identification of projects with safeguard issues and country strategies and tradeoffs between client and ensure compliance with policies, including the Bank priorities, or the relative strengths of lend- introduction of the integrated data sheets, ing and nonlending services. There has been im- issuance of safeguards policy matrices, short guid- provement in the FY98-01 cohort of SSPs. ance notes for staff, training activities, and Although an ongoing stock-taking by management resources to guide operational staff. ESSD and is identifying areas for further work-given the Regional Vice-Presidents have joint sign-off on complexity of the remaining issues and new ex- corporate risk projects. Finally, QACU has started pectations about the role of SSPs-it is beginning to undertake a systematic Bankwide aggregation to have an impact on SSP quality, as demon- of information relating to safeguard issues, which strated in the recent preparation of a sector strat- should be a valuable input in the recasting and egy for information and communications updating of policies. technology (ICT). These changes are a significant improvement over a relatively short period of time. However, Quality Assurance some issues remain. While corporate risk proj- QAG's periodic and systematic assessments of ects are closely monitored, there is in general quality at entry and quality of supervision have little safeguard monitoring of other projects helped identify areas ofweakness and improve op- during supervision. There is no mechanism for erational quality over time, as noted earlier. To field verification-the initial notion of a sample complement these assessments of approved proj- "audit" of the portfolio was dropped. The ects, management has taken steps to enhance ex- Regional safeguards staff, the first line of defense ante quality assurance for lending operations by against noncompliance, faces a potential conflict establishing Regional QETs and the use of ex-ante of interest. Without full-time commitments quality enhancement reviews (QERs). The QET to safeguard issues, the Regional safeguard and QER functions and procedures vary across coordinators act both as monitors for safeguard Regions and networks, but generally they re- compliance and as technical specialists with spond to demand and self-identified issues from Regional operational responsibilities. The within a Region. QER and QET activities tend introduction of dual OPEs for Regional safe- to focus on the individual project design, and guards coordinators will be introduced in FY02 less on the alignment between the goals of the and are expected to address this issue. Finally, project, the CAS, the underlying analytical and there is a continuing need to harmonize safeguard advisory activities (AAA), and corporate objectives. requirements with those of client countries and Since QERs are undertaken mainly on a volun- international partners. 14 CONTROL ACTIVITIES Quality Assurance for ESW account for capacity-building objectives and (b) The importance of ESW, which has long under- address ESW impact at the program rather than pinned country strategies, policy dialogue, and at the project level.' project design, has further increased to comple- ment the evolution of CDF/PRSP processes Project Self-Evaluation and of the Knowledge Bank. Accordingly, The ICR is the Bank's main self-evaluation and management has paid considerable attention to learning tool. It is central to the Bank's evaluation improving the quality of ESW through periodic system and integral to its knowledge manage- assessments of ESW quality by QAG and ment system. OED periodically evaluates the in-depth self-assessments. Management formally quality of the ICR process (the findings from launched an ESW reform effort in July 1999, OED's fifth ICR process review are given in which led to the adoption of quality enhancement Annex B). The quality of the ICRs remains high, action plans in all Regions. The impact of these with 94 percent rated satisfactory or better in efforts is shown in QAG's assessment of ESW- FY99, with improvements in several dimensions. the quality of ESW was found to be much im- A problem area identified by the previous re- proved in FY00, with 86 percent rated satisfactory views, and a source of risk for the sustainability or better, up from 73 percent in the previous two of projects' development outcomes, is the failure assessments. The gains have been seen on several to include a plan for future operations (renamed dimensions of ESW quality, but there are some the transition arrangements). The share of ICRs key areas where more attention is needed. with a satisfactory discussion of transition QAG's assessment of ESW found both the arrangements improved from 24 percent to 45 quantity and quality to be lower in countries percent over the past year, leaving ample room for with weak policy and institutional frameworks, as further improvement.7 The quality of ICR ratings measured by their Country Policy and Institutional for project outcomes has also improved: The Assessment (CPIA) ratings. Quality was rated as number of ICR ratings changed after OED review uneven for two "due diligence" products, Poverty has declined from 8 percent in FY96-97 to 4 per- Assessments and Public Expenditure Reviews. cent in FY98-00. The realism of supervision The analysis of implications for the poor and for (PSR) reporting shows less improvement. Ratings gender sensitivity was also an area of weakness. changes between the final PSR and ICR remained The assessment notes that the improvement in at 11 percent between the periods FY96-97 and quality is not mirrored in a commensurate gain FY98-00. The total ratings changed between the in the effectiveness of Bank processes, including PSR and OED review have declined from 14 sustained attention by operational managers. Peer percent in FY96-97 to 12 percent in FY98-00, review has long been a cornerstone of ESW which is large enough to cause concern about quality assurance, but despite the improved the realism of self-evaluation. quality of peer reviews, the quality at entry There has been no improvement in the real- assessment concludes that their advice does not ism of ratings in the key quality dimensions of in- often have an impact on the final product. The stitutional development and sustainability. task teams also do not adequately draw upon the Between the ICR and OED review, these ratings analytical expertise or the knowledge base in the were changed for 20 percent of the projects re- networks, Development Economics (DEC), or the viewed in FY98-00 in both dimensions (virtually World Bank Institute (WBI). The networks' the same as the 19 percent of FY96-97).' current "support" role does not fully exploit their In FY00 the Bank introduced a significant comparative advantage. program to reform the ICR process, introducing Overall, these assessments point to the need to the Intensive Learning ICRS (ILIs). About 30 strengthen the quality control and supervision percent of the projects were to have ILIs, and the functions for ESW. There is also a need to adjust rest a Core Accountability ICR (CAI). In the the evaluation methodology for ESW to (a) partial sample of the FY00 ICRs received by 15 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION OED (to July 2000), 15 percent were ILls, 10 per- HIPC and PRSP initiatives require substantial im- cent followed the previous guidelines, and the provements in client countries' statistical and rest were CAls (see Annex B). The initial results monitoring systems to track the poverty impact are inconclusive as to the difference in the qual- of interventions. The Bank and other partners are ity of ILIs and CAls. supporting these improvements. Initial efforts have focused on the countries' monitoring activ- Corporate Self-Evaluation ities, and these must now be complemented by The Bank has significantly increased self-evalua- attention to their evaluation capacities. OED is tions for major institutional products and services planning to undertake process evaluations of these in recent years. A particularly noteworthy achieve- initiatives. ment is the just-completed assessment of the Strate- A particular challenge for the CDF is that it gic Compact, which was recognized by the Board focuses on process improvements and neither as analytical and fair, offering important lessons that baseline data nor an evaluation framework was in- can help the Bank move forward. It provides a com- cluded at the outset. Management has put in prehensive assessment of not only how the Com- place a program to monitor progress in imple- pact initiatives fared but also how the Bank works, menting CDF principles in CDF/PRSP countries. identifying several weaknesses that the management In addition, OED and DEC, together with sev- has swiftly moved to address. The Bank has also eral other donor and client partners, have jointly made effective use of task forces to address systemic initiated a process evaluation of the CDF, which issues. The regular budget retrospective should will be completed in FY03. adopt a similarly rigorous and objective stance in order to institutionalize learning and improve ef- Adaptable Lending Instruments ficiency in resource allocation and use. From their introduction in 1997 until mid-June, The Bank also undertakes regular stock-taking 2001, the Board has approved 87 Adaptable Pro- or retrospective exercises on major issues, prod- gram Loans (APLs) and 85 Learning and Inno- ucts, and processes. For example, the CAS retro- vation Loans (LILs). A particular challenge to spectives have provided important insights on the control framework ofAPLs is the development CAS quality and process, identifying areas for of appropriate performance triggers and the mon- improvement. Other retrospectives have played itoring of the incorporation of lessons between important roles in improving ESW quality, fo- phases. The APLs do not require special ICRs; that cusing attention on adjustment lending and other is, each phase is evaluated individually and it is portfolio issues. The leadership of the OPCS expected that the ICR will be undertaken six Vice-Presidency has been critical to the inculca- months ahead of the last disbursement. The ICR tion of a self-evaluation culture in Operations. guidelines need to be amplified to include per- formance triggers for approval of the next phase. Evaluation for New Initiatives LILs can make an important contribution both During the Strategic Compact period, the Bank for learning and as risk-management tools up- has expanded and strengthened in-country part- stream of operations addressing complex or dif- nerships and introduced new lending instruments ficult development issues. A development risk to provide flexibility and promote innovation for LILs is failure to reach their learning objective and learning. The evolving emphasis on country (affecting the quality of a follow-up operation focus in the Bank's work, particularly in the con- or, conversely, the suppression of a good idea). text of the CDF/PRSP approach and the HIPC Sound monitoring and evaluation (M&E) initiative, will place new demands on the control arrangements, quality, and realistic requirements framework, because accountability for results is and procedures are critical dimensions for effec- shared between the Bank and the borrower. The tive LILs. 16 5 MONITORING or effective management, the evaluation and control system requires regular and systematic monitoring to assess its quality and for the prompt identification and resolution of problems. Monitoring is also essential to ensure that appropriate actions are taken in time to address emerging risks to the efficiency and effectiveness of operations. The Bank's system was significantly enhanced time as OED's ARDE to allow Board members with the establishment of QAG for real-time to consider performance trends in both the monitoring of operational quality. This provided ongoing and closed portfolios. The portfolio a much-needed complement to the ex-post performance review is a valuable analytical tool, independent evaluation function of OED. addressing key emerging issues and identifying Similarly, QACU has filled an important gap in actual and potential risk areas in need of man- the monitoring of ex-ante safeguard policy com- agement attention. It is constructed primarily pliance. The networks are also monitoring the from materials from regular portfolio monitoring portfolio to ensure quality in their respective functions (including QAG quality assessments), thematic areas or sectors more systematically and supplemented by data in the Bank's management regularly. The PREM Network, for example, information system. The analysis at the country produces an annual progress report on poverty level uses the Country Portfolio Performance reduction.' This enables monitoring of progress Reviews (CPPR). The CPPR is used by the in the overall poverty reduction effort, although Regional Management Teams to engage the bor- it still falls short of scrutinizing progress on poverty rowers in high-level discussions about the Bank's reduction dimensions of individual sector assis- portfolio performance. These discussions are im- tance efforts. portant, both to address the accountability issues that affect the performance of individual opera- Portfolio Monitoring tions and those that are systemic in nature. At the corporate level, QAG's annual review of portfolio performance is the primary operational Monitoring and Evaluation monitoring tool. The findings of the review are M&E is essential to track development outcomes, presented to CODE and the Board at the same to identify emerging risks, and to achieve results. 17 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION As a tool for performance measurement and man- sponsors of the program. On fiduciary issues, as agement, it is a critical feature of the evaluation part of a broader governance approach, the Banks and control framework. The M&E in Bank op- support for more effective public expenditure erations has been chronically deficient. Thus, de- management and accountability has increased spite indications of increasing operational quality considerably in recent years through greater and and project performance, the Bank does not have more effective fiduciary ESW and institution a solid foundation to convincingly demonstrate building efforts. For development outcomes, there results on the ground. are several efforts under way to build up moni- To address this issue, in 1999 the Bank estab- toring and statistical systems, but, as noted lished a working group to develop a compre- earlier, there is a need to complement them by hensive action plan to achieve sustainable im- mainstreaming evaluation capacity development. provements in M&E in Bank-financed projects. The working group dealt only with investment Corporate and Sector Scorecards projects. It recommended a two-pronged Challenges remain in completing the corporate approach phased over a four-year period: to pilot and sectoral scorecards. An assessment of the results-based M&E in 8 countries and 4 sectors Strategic Compact found that there is a corporate and to build the Bank's internal capacity through scorecard, which is balanced and impact- a cadre of an equivalent of 16 full-time senior focused-but it lacks data, particularly on Bank M&E specialists to support task teams and country and sector assistance, and is not being used borrowers.2 The action plan is being implemented: meaningfully. The assessment also notes that the Diagnostic work is underway in six pilot coun- scorecard is reasonably complete for Tier III (Bank tries, and two M&E specialists are in place in internal measures), which is expected to translate OPCS. The Regions have adopted different strate- into an overall improvement in the Bank's effec- gies to identify and build in-house capacity to fill tiveness in achieving poverty reduction. Efforts the need for M&E expertise. to fill in Tier II (the performance or impact The pilot approach recommended by the M&E measures for Bank performance at the country and working group has found support in the Bank, sector levels) of the scorecard are ongoing. A joint provided M&E was applied across the board in OED-OPCS task force is developing a common accordance with current Bank policies. The need view on how to measure Bank performance at the to address evaluation concerns for systemic issues country level. OED is developing a template such as adjustment lending to make M&E more for the Country Information Form (CIF) to effective has also been identified as a priority area. evaluate the outcome of Bank assistance, which The findings of the latest QAG assessments for could serve as a starting point for this effort. both quality at entry and quality of supervision Management has commented on earlier versions indicate that M&E continues to be weak relative of the CIF, and OED has revised the CIF to to other dimensions of quality. incorporate those comments. Early in FY02, M&E is important for improving borrowers' based on the outcome of this work, the task force management of investment programs and for will convene to agree on how to implement the better controlling the risks to the development CIF, on a pilot basis, as an input for the corre- effectiveness of Bank operations (by enhancing sponding module in the scorecard. The Tier II supervision quality as well as learning). The indicators will eventually be linked to the Tier I significance and duration of the problem warrants indicators, the results dimension of the scorecard a more comprehensive and accelerated approach. as embodied in the MDGs. This is especially true for PRSP countries. A recent study on tracking poverty-reducing pub- Inspection Panel lic spending noted that most of the HIPCs could The Inspection Panel was created in September not track poverty expenditures, a particularly 1993 by the Bank's Board and is an independent risky situation for the Bank as one of the primary mechanism to enhance the transparency and 18 MONITORING accountability of Bank operations. It serves as a operational processes. The Inspection Panel find- valuable instrument for the Board to independ- ings point to important control failures, includ- ently verify whether the Bank is following its ing inappropriate incentives for staff and managers, own policies and procedures, particularly those lack of clarity and the misinterpretation of poli- intended to protect the environment and the cies, and a preoccupation with inputs and outputs interests of the people affected by Bank projects. instead of adequate attention to impacts. These Inspection is triggered by a written complaint, findings confirm some of the conclusions of this which gives an effective voice to local communi- report. Thus, the Inspection Panel is helping the ties that may be adversely affected by Bank Bank improve its development effectiveness and operations.4 reduce its reputational risks. At the same time, So far the Panel has received 19 requests for however, steps need to be taken to ensure that investigations. All but the one received in FYO1 concerns about the potential impact of Inspection have been completed. As an instrument of last Panel investigations on the careers of Bank resort, the Panel reports provide useful insights managers and staff do not lead to excessive risk for improving policy implementation and avoidance. 19 6 INFORMATION AND COMMUNICATION Pertinent, timely, and accurate information and effective communication are necessary for management and staff to carry out their responsibilities. Decentralization of work, the matrix structure, and increased partnerships and consultative processes make efficient and reliable communications critical. For the Bank's development work, the role of information goes beyond financial and administrative reporting. Information and knowledge management are inextricably linked. To improve the efficiency and effectiveness of development activities, it is important that the Bank not only have the "hardware" to communicate information but also the "soft- information system, is tracked and reported by ware," or the knowledge, to contribute to the IAD and CTR. content of what is being conveyed. Harnessing de- The enhancement to information technology velopment knowledge has been a comparative systems and MIS should address the initial gaps, advantage of the Bank. To build on this asset, the such as underemphasis on useful management Bank committed itself to becoming a Knowledge reports and difficulties in recording and accessing Bank in 1996, and the Strategic Compact made relevant information. Overall, the ISR succeeded "retooling the Bank's knowledge base" one of its in streamlining processes, instituting electronic four priorities for change. workflow and approval systems, improving in- ternal processing efficiency, effectively linking Information Systems the Bank's worldwide telecommunications and The Bank has invested substantially to upgrade computer networks, and contributing to the its information technology, communications, and smooth decentralization of Bank activities. Bet- Management Information System (MIS). The ter access to knowledge and information by staff progress on the Information Systems Renewal has no doubt contributed to the improvement in (ISR) Program, including the integrity of the in- operational quality and the Bank's development formation and security systems and the financial effectiveness. Further changes are expected 21 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION following the recent work by management on the full "knowledge cycle," from creating, to shar- better measurement of products and services, ing, to applying knowledge, and the process is ex- which has identified problems in tracking even the pected to result in a framework to guide inputs and expenditures on several products and investment, instruments, and evaluation of the specific initiatives, such as AIDS,. financial and program's impact. The management of such a leveraged products such as HIPC, country-based system will be challenging, and it will be partic- and network-based nonlending services such as the ularly difficult to monitor and evaluate the ac- PRSP, and additional cost drivers such as fiduci- tivities of the new Knowledge Bank. There is a ary and safeguard policies and crosscutting the- pressing need to set up a coherent evaluation and matic issues. This information will be critical in control framework. evaluating these efforts. Creating Knowledge Knowledge Management The Bank has traditionally invested heavily The Bank's knowledge management activities in knowledge creation through ESW and have grown significantly in the last few years. research. Issues related to ESW are discussed in They include sharing knowledge internally and Chapter 4. For research, the Bank-wide Research with the external development community, a Committee is responsible for increasing the value rapid expansion ofknowledge communities or the- of Bank research through allocation of central matic groups, and increased client training. The Research Support Budget (RSB) funds. All RSB- Bank has made considerable investment in tech- supported activities are subjected to evaluation. nology, including in the Global Development Since RSB funds are often used in combination Learning Network, the Development Forum, the with other funds for research, the RSB evaluation Development Gateway, websites, and databases. covers a larger share of the Bank's research The basic infrastructure for documenting and expenditures than are supported by the RSB disseminating knowledge is in place. alone. The non-RSB-supported research by The focus under the Strategic Compact was on DEC is subject to formal professional review knowledge sharing, a previously neglected di- procedures, since an important individual per- mension, to maximize benefits from the Bank's ac- formance evaluation criteria in DEC is. publica- cumulated development knowledge. The Bank has tion in refereed journals. DEC has recently not yet had an explicit strategy for pursuing its vi- introduced a quarterly assessment by the Re- sion of the Knowledge Bank, nor is there a frame- search Committee to rate all of its research reports work to evaluate current activities. Indicators to for "pertinence" to the Bank's operational work. monitor output or outcomes have not been well A questionnaire asks the members of the specified, and there has been no assessment of im- Research Committee to rate the listed projects pact or cost-effectiveness. An assessment of the submitted for their relevance to Bank policy and Compact found that much of the knowledge is operations. Another indicator of relevance is not easily or intuitively available, and there is lit- the feedback from biennial surveys conducted tIe quality control on its content. It also found that by DEC. the current knowledge management system is fragmented and not well integrated with opera- Imparting Knowledge: Training tional processes, and that there is little evidence As part of a broader capacity building effort, the of systematic learning or rationalization. Strategic Compact increased support for staff and Management is in the process of articulating client learning programs. Client training is con- a strategy for the Knowledge Bank and a stronger ducted through WBI. About 15 percent of staff governance system to best serve client and oper- training is through WBI; networks and Human ational needs. It has just finalized its strategy on Resources deliver the rest. WBI also evaluates Information and Communications Technology. the training activities. Since 1997, the evalua- The definition of the Knowledge Bank includes tion of learning activities has been significantly 22 INFORMATION AND COMMUNICATION strengthened. Most client training events are eval- expected. Impact evaluations of staff learning uated to assess their usefulness and effectiveness, started in FY00 (with both leadership and with less focus on behavioral change and impact. decentralization) courses. Such evaluations are Evaluations of strategic initiatives, such as the expected to increase in the future. Distance Learning and Partnership Program, have The Bank's new staff learning framework aims also been conducted. to shift training away from individual technical About 30 percent of classroom training for skills to managerial, behavioral, socio-political, staff is currently evaluated. The evaluation of Sec- and professional skills suited to the Bank's emerg- tor Weeks and nonformal learning events (lectures, ing needs and products. To assist program de- seminars, clinics, action-learning workshops) has velopment, ensure consistency of learning increased, but the coverage of the latter is ad hoc. activities with corporate goals and needs, and The results indicate that Sector Weeks need bet- evaluate impact, a comprehensive evaluation ter organization to increase their value to partic- framework is needed with a clear strategy and an ipants. Nonformal events are less interactive than enhanced feedback loop to the Knowledge Bank. 23 7 THE OED AGENDA In addition to independently evaluating the development effectiveness of Bank operations, the Director-General, Operations Evaluation (DGO), is responsible, through OED, for promoting the incorporation of evaluation assessments and findings into recommendations that will help improve the efficiency and effectiveness of the Banks programs and activities, and their responsiveness to member countries' needs and concerns. To ensure continued relevance of independent evaluation and to increase its impact on Bank performance, OED adopted a comprehensive renewal program in 1997 (box 7.1). Consistent with this strategy, OED shifted re- products delivered and their timeliness in relation sources to accommodate new priorities. It has to operational milestones. given high priority to evaluation capacity build- Regular CODE and Board oversight of OED's ing in client countries (Annex C), investing in work program is a key element of the control knowledge (Annex D), and outreach and dis- framework for OED itself. While OED's renewal semination (Annex E). strategy and strategic objectives continue to remain Because OED provides independent evaluation relevant in responding to Bank priorities and of Bank operations to the Board, it has a special the evolving operational agenda, there are three obligation to evaluate its own effectiveness and im- areas in need of attention. On harmonization of pact, and to reflect the results in its work. The fol- evaluation, the emphasis on evaluation capacity lowing sections describe how OED's inputs, development (ECD) and collaborative evalua- outputs, outcomes, and impacts are currently tion efforts is well placed, but OED should evaluated, and conclude with the implications ensure against the possible impact of evaluation for OED's agenda going forward. partnerships on the quality and reliability of OED's work: OED needs to maintain its OED Work Program and Outputs independence as it engages in more participatory Annex F presents the indicators used by OED and global joint evaluations. OED also needs to measure its inputs and deliverables for the to clarify its plans to evaluate the impact of past year. These measure both the number of programmatic lending. More generally, in the area 25 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION Box 7.1 Mission: 3. Build evaluation capacity within and outside the Bank to To contribute to development effectiveness through excellence promote self-evaluation. and independence in evaluation. 4. Invest in knowledge and partnerships to maintain OED's intellectual leadership and develop alliances as instruments Objectives: of strategy. 1. Move to a higher evaluative plane; that is, from,projects to 5. Manage for results, since evaluations that influence country, sector and thematic, and global evaluations. behavior and contribute to the overall success of the Bank's 2. Shorten the feedback loop to ensure relevance, strategy add value. of methodology, OED needs to focus on developing become more standardized in scope and method- evaluation methodologies and establishing bench- ology. But there is still room for improvement, marks to underpin eventual evaluations for new which is why OED has begun consulting an ever- products and initiatives (e.g., PRSPs, sector-wide widening circle of stakeholders, inside and out- approaches). A more systematic assessment side the Bank, to develop a common methodology of the strategic impact of ESW and upstream that could be utilized to assess the effectiveness of nonlending services remains a priority, as does all external assistance. revisiting the inclusion of impact evaluations in To this end, OED and OPCS established a the work program. joint Task Force on Country Assistance Method- ology in August 2000. Considerable progress has Harmonizing Evaluation Methodology been made toward development of a common Internally, OED and OPCS have made sub- methodology between Operations and OED in stantial progress toward a common evaluation evaluating country assistance. A shortened version framework for the project cycle, and focusing on of the CIF (a standardized method of rating the results. Progress on achieving consistency be- performance of the Bank and Bank assistance) has tween the methodologies used for ex-post proj- been drafted, and is expected to go to the Joint ect evaluation between the ICR and OED is Task Force in June. At that time, OED will seek reflected in the declining disconnect for project agreement to pilot the CIF with a small group of performance between the two. Some additional country directors, asking them to assess their own work is needed to develop consistent guidelines programs prior to their next scheduled CAS. Ul- for the relevance and institutional development timately, the CIF is expected to serve as the pre- aspects of performance measurement. Progress in cursor to a standard set of country assistance these areas has been slower than expected. The evaluation criteria that will be integrated into the persistent difference in the ratings for sustain- CAS design, analogous to what has already oc- ability is also puzzling and warrants further ac- curred at the project level with shared ICR/PIF tion. At the supervision stage, there is less guidelines. OED used an early version of the consistency, as reflected in the persistent differ- methodology as the framework for its FY01 CAEs. ence in project performance ratings between the OED actively participates in efforts to har- PSRs and the ICR, which currently accounts for monize evaluation standards across partners. This the major share of the disconnect between su- fosters greater use of partnerships and upgrades pervision and OED ratings. the global standards of evaluation. A Multilateral Over the past five years, OED has evaluated the Development Bank Evaluation Cooperation development impact of the Bank's country assis- Group (ECG) report in December 1999 on har- tance programs in nearly 40 countries. With this monizing evaluation criteria led to an agreed set broad experience at hand, CAEs have gradually of eight main criteria, and work is ongoing to 26 THE OED AGENDA harmonize and improve evaluation methods for group"). Another set was sent to staff associated private and public sector evaluations.1 Efforts in with specific products: recent project Performance harmonizing completion reporting and per- Audit Reports (PARs), recent CAEs, members formance review processes are also under way. of relevant thematic groups on topics related to The public sector group is working on develop- recent sector and thematic studies, and staff work- ing a paper on Good Practice standards, and one ing on CASs for countries with no CAE. The main for private sector operations is already available. findings are as follows. OED Outcomes and Impact Reach OED has traditionally measured outcomes indi- OED's reports are widely read by its general client rectly in terms of the improved quality of the groups: 91 percent of respondents have read at least self-evaluation process; the adoption rates for one product in the last six months. ED staff fre- its recommendations; and in the improved quently read all products, particularly CAEs, sec- development effectiveness of Bank operations, as tor and thematic (S&T) studies, and corporate and measured by the improvement in the quality process (C&P) evaluations. The random and of the Bank's portfolio. Overall, management's quality groups are less likely to have read an OED adoption of OED recommendations is relatively product in the last six months. Executive sum- good, and improving. The review of CAS dis- maries (ES), S&T studies, and, to a lesser extent, cussions summarized below reveals that they are PARs and CAEs are popular among these read- amply reflected in CASs and in the Board's ers. This may partly reflect the specificity of these discussion of them. reports to countries, sectors, and projects, while A key recommendation that emerged from the samples were from across the spectrum. Read- OED's work program review last year was for the ership is higher among those associated with a spe- department to undertake a systematic review of cific OED study. More than three-quarters of its impact on the Bank to guide its future work the targeted respondents read"the CAE and PAR. program. This section reports on the findings of The CAE also finds readers beyond immediate au- this review, which assessed three specific dimen- diences, with half of the CAS (without CAE) sions of OED's work: task managers having read a CAE in the last six * The reach, quality, and usefulness of OED months. The surveys for S&T studies were much products, as measured by client surveys more broadly targeted than those for the PARs and * OED's influence on policies and thinking in CAEs. They are read by about one-third of the re- the Bank, as measured by a qualitative study spondents. tracing the influence of three major products The use of OED databases more than tripled * OED's influence on country,and corporate when compared with findings from Bank staff sur- strategies, as evidenced in CAS documents veyed last year (53 percent, up from 14 percent). and records of Board discussions. The OED website is by far the preferred source of OED information and an efficient vehicle for Reach, Quality, and Usefulness: information delivery, with the majority of users Results from Client Surveys rating the site as both accessible and user-friendly. OED surveyed Bank staff and borrowers about their views on OED's major products. Between Quality and Learning February and March 2001, 10 surveys were ad- The overall quality of OED reports is rated sat- ministered to more than 1,800 Bank staff.2 (Annex isfactory, particularly by the ED staff, but also by G describes the survey design and more detailed the quality and random groups. The satisfactory results.) One set of surveys was sent to staff work- ratings for PARs are also quite high (75 percent), ing in Executive Directors' offices ("EDs' staff"), but for CAEs, fewer than half of those respond- knowledge managers ("quality group"), and a ing rated overall quality as satisfactory (45 percent). random sample of operational staff ("random The relatively lower ratings by those directly 27 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION involved in the evaluations, as for CAEs and the evaluation influenced their current thinking PARs, is a matter of concern for OED and need and helped change their minds on an issue. A further investigation.3 Among the individual di- larger (83 percent) number of non-CAE CAS mensions of quality, the weakest aspects of OED task managers acknowledged the CAE's role in evaluations are seen as their timeliness, rigor, and changing their mind on an issue. methodology. The two products for which time- Finally, as shown in figure 7.1, a vast majority liness is rated high are CAEs and S&T evaluations, of the respondents think the evaluations reflect reflecting their direct link to the timetables for Bank priorities and are grounded in the current CASs and SSPs. OED's objectivity and relevance state of knowledge. Nevertheless, their influence are generally well-rated (about 80 percent on on Bank strategies and operational services is average), attenuating possible concerns about bias much lower, with almost a third finding no in- in OED work.' The quality of CAEs is rated fluence on the different dimensions. more favorably by CAS task managers in coun- tries without a CAE: They are more likely than Influence on Learning: CAS task managers in countries with CAEs to find Findings From a Tracer Study CAEs to be objective, methodologically sound, To complement the survey in assessing OED's in- and offering conclusions based on the evidence fluence on learning and decisionmaking, OED provided. conducted a tracer study of recent OED evalua- OED evaluations and findings appear to play tions on forestry, large dams, and the impact of a positive role in staff learning. Three-quarters of Public Expenditure Reviews (see Annex H for the quality group, who have a leadership role in details). A consultant conducted one-on-one con- knowledge management, indicate that OED eval- fidential interviews with 43 selected individuals uations have helped increase their knowledge of across the Bank (including Board members, sen- their sector and of Bank operations generally. ior managers, and operational staff) to learn how Among CAE respondents, almost half indicate that they used the studies, what change the evaluations Figure 7.1 Relevance Inffluence strategy & policy Influence activities 100 90 . ... 80 -- - - - - - - - -- - 70 S60 l 50 IDI S40 - , - -- - - a. 30 20 10 - 28 28 THE OED AGENDA brought about, and what factors favor or impede its fundamental strength, giving it legitimacy to their usefulness. (The study design is described fur- pass judgment on Bank operations. Some re- ther in Annex H.) spondents, though, noted that OED tends to ac- The study shows that the evaluations have in- cept the Bank's prevailing values and priorities too fluenced the actions and thinking of Bank deci- readily, suggesting that it should perhaps subject sionmakers and staff, both directly-through them to a more rigorous independent assessment. specific subsequent decisions and action-and indirectly-by facilitating, albeit over a period of Influence on Country and Corporate Strategies: time, a deeper understanding of the issues. Bank Evidence from Documents staff more often become familiar with a report's Finally, OED assessed its influence on country findings through informal networks than by strategies and corporate and Board deliberations reading the reports. Both Bank staff and OED by reviewing CAS documents and discussions evaluators recognize OED's institutional mandate from selected Board meetings. As a planning tool to report to the Board; however, they consider for Bank management and the client, and the managers and operational staff as the primary Board's instrument for reviewing the Bank's strate- audience for OED evaluations. This suggests a gic directions at the country level, the CAS is an need for sharper differentiation of clients formore important vehicle for the fulfillment of OED's effective dissemination of evaluative findings and learning and accountability mandates.5 OED's recommendations. influence on CASs depends significantly on the The interviewees appreciate the value of the existence of a CAE: There is substantial evidence systematic assessment, analytical framework, in- of CAE influence on design, strategy, and specific dependence of judgment, and action orientation actions. Among the nine non-CAE CASs, how- of OED reports. But their initial reaction is to ever, only four discuss OED findings, of which screen the evaluation through the filter of their only two included substantive details. own knowledge and beliefs, and they often re- CODE is OED's focal point for interaction with sist challenging findings. As time passes, how- the Board. Beyond its direct interventions at -ever, findings that challenge the status quo are CODE, discussions of operational issues at se- increasingly accepted, and recommendations lected meetings of the full Board were reviewed to that had initially appeared heretical are eventu- identify whether and how reference was made to ally received with favor and adopted. OED eval- OED findings and recommendations.6 The review uators need, therefore, to better anticipate the found that OED figures prominently in many of reactions of the intended users of their evalua- the Board discussions. Its findings and recom- tions. Greater participation, more consultative mendations were substantively referred to in all four evaluations, and early dissemination and feed- meetings on SSPs, and virtually all (seven of eight) back may facilitate knowledge transfers. policy and operational discussions. Board mention Some respondents expressed skepticism re- of OED findings in discussions of CASs reflects garding the rigor of OED's methodology and the existence of a CAE, as found by the CAS re- data collection. Yet many are comfortable leaving view. There was substantive discussion of OED all methodological decisions to OED. Not sur- findings in all four CAE-CASs, but in only one prisingly, methodology becomes a significant issue of the non-CAE/CAS discussions. Thus, although when controversy about ratings arises. Bank staff the staff survey suggests that the CAEs may not prefer recommendations that are explicit and ac- be greatly appreciated by the affected staff, they tionable, but find OED's recommendations at seem to be an effective tool for integrating OED times too generic, "statements of good inten- findings in a way that adds value to the CAS tions," or difficult to implement. Most assert that process and is used by the Board in its oversight they or their subordinates are already acting on role. In countries without a CAE, findings from the evaluation recommendations. The majority OED evaluations appear to have limited influence also acknowledge that OED's independence is on the Board's CAS discussions. This suggests a 29 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION need to provide Executive Directors with past disseminate its evaluation methodology to pro- OED findings of relevance to Board delibera- mote a better understanding among opera- tions, even in the absence of a recent CAE. tional staff and wider adoption of the underlying principles in self-evaluations. Implications for OED * Participation and Communication: Greater par- These findings, along with guidance from CODE ticipation, more consultative approaches, and on OED's work program, imply that OED should early dissemination and feedback help to fos- focus on the following areas for improvement. ter understanding and acceptance of evaluation * Timeliness and Relevance: The positive feedback findings and lessons. OED needs to further from the synergistic programming of CAEs consolidate its current practice of "no sur- and S&T studies with CAS and SSP schedules prises" through transparent and participatory provides important lessons for the timeliness evaluation processes by involving relevant Bank and enhanced relevance of other OED prod- staff and stakeholders early in the design and ucts. OED should use strategic selectivity to planning of its evaluations and through regu- more closely link the evaluation agenda with lar progress updates. To answer calls for more operational trends and milestones. To further actionable recommendations, OED has pro- enhance the relevance of its evaluations, OED vided its staff with training and guidelines for should pay increased attention to corporate preparing more effective recommendations. thematic priorities, particularly the poverty OED managers and peer reviewers should pay orientation of Bank operations. particular attention to this key learning aspect * Methodology: In view of the concerns expressed of OED's work. about the rigor and clarity of OED's method- * Dissemination: OED's audiences and their uses ology, there is an urgent need for OED to re- of OED evaluations are varied. For example, examine, revise, and enhance, where the Board and staff/management are OED's appropriate, its methodological tool kit. In two main audiences, but their uses of OED methodology development, OED has a broad products differ. OED's dissemination strategies agenda to pursue: review its procedures for should be further customized and differenti- evaluating adjustment operations; ensure that ated to meet the diverse objectives of these programmatic lending instruments are ade- audiences. quately covered; simplify project evaluation - Among Board members, the readership and instruments; reach agreement and test the CIF use of OED findings is relatively high. A re- and the corresponding country evaluation maining challenge is to make the lessons methodology, and adapt OED methods and from PARs, ESs, and S&T studies more practices to accommodate new instruments readily accessible and useable in discussions (APLs, LILs, guarantee-financed operations, of CASs when no CAE is available. To this and the like) in line with the adaptation of ICR end, OED should consider ways to collate guidelines for these instruments. OED should strategic and operational lessons of experi- continue to help harmonize its evaluation ence and be more proactive in briefing methodologies and practices with both Oper- Board members on how to access and use ations and with external evaluation partners. OED products. It should establish protocols to ensure that - OED's website and publications seem to collaboration does not mean dilution of ob- be well-suited to the staff, but need more jectivity or diminution of quality. In addition, publicity, especially for new reports, and OED evaluations need to be even more trans- extended distribution of summary prod- parent in indicating data sources, articulating ucts such as Fast Track Briefs and Pr6cis. the evaluation approach, and detailing the an- There is also considerable scope to increase alytical assumptions maintained to arrive at outreach through a more refined matching conclusions. OED should also more actively of products and target audiences. This would 30 THE OED AGENDA entail more purposeful targeting of such Sef-evaluation: Building on this year's survey, audiences as thematic groups and knowledge and using it as a baseline, OED should mon- managers, and better access to documents itor future progress on outcomes through an- (e.g., full and abridged versions on the web). nual or biennial surveys. It should develop a Discounted or free distribution of priced cost-effective core staff survey and expand on publications may be advisable in selected in- this year's pilot for borrowers for more sys- stances. tematic feedback. 31 CONCLUSIONS AND RECOMMENDATIONS The changes in the Bank over the past five years have increased development rewards through a stronger country focus, improved responsiveness, and enhanced operational quality. But increasing demands at both the country and global levels, expanding roles for capacity building and aid coordination, and increasing complexity of operations have intensified the need for new skills, instruments, processes, and relationships, and for a revamping of the risk management framework. The pressure to demonstrate results, especially in reducing poverty, has increased, putting high priority on ensuring the adequacy of the evaluation and control system. The Evaluation and Control Framework evaluation and controls system using the COSO Viewed through the lens of the COSO frame- standards helps identify areas that need attention. work, the current evaluation and control arrange- ments provide an adequate and improving Control Environment framework for project lending. Evaluation and The Bank's control environment has served it well control units have contributed to the Bank's im- in an increasingly complex setting. Consistent proving performance as they evolved to fill ear- focus by management on quality and institutional lier gaps in development effectiveness oversight learning has improved the control conscious- and quality assurance. But even as this system is ness of staff. But significant changes and ex- being consolidated, the operational domain is panding mandates have increased stress and shifting. Changes in the Bank's product mix and stretched institutional capacity. Management is an increased emphasis on knowledge services, col- taking steps to address selectivity issues. A laborative programs, and programmatic lending remaining challenge is to align corporate prior- are stressing the capacity of the current evalua- ities with country strategies. The new organiza- tion and control system. The transition will be tional structure is being fine-tuned, and further challenging for the Bank as well as for its bor- adjustments may be needed to promote behav- rowers and partners. An analysis of the compo- iors consistent with the new operational nents of the Bank's development effectiveness approaches. The Bank has adopted a new results- 33 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION based development paradigm, and the internal standards, and remaining gaps in the compliance incentives and processes need to be realigned with testing processes are sources of risk to the Bank's the new Bank's vision and operational ap- development effectiveness. The process of re- proaches. To enhance development effective- casting and updating of operational policies, es- ness, there is a need to link Bank objectives and pecially for adjustment lending and social policies more explicitly to results, especially on safeguard policies, has been complex and chal- poverty reduction, and to set up monitorable out- lenging. While notable progress has been made in comes in CASs, SSPs, and PADs. providing guidance to staff on safeguard policies and work is progressing on other policies, it is im- Risk Assessment portant for this work to culminate in issuance of A corporate risk management structure is also unambiguous policy statements to guide opera- taking form. The framework for risk assessment tional work. for project lending-the Bank's traditional line of The quality assurance for lending operations business-is in place. The management of safe- and ESW has improved in recent years, with guard and fiduciary risks has improved consider- some areas in need of further strengthening. The ably, but processes for assessing and managing quality of self-evaluation at project completion development risks at project entry and during continues to improve; weaknesses remain in the implementation need further review and strength- realism of self-assessment on sustainability and in- ening. The assessment of risks in adjustment stitutional development. Corporate self-evaluations operations needs to be watched because of their have increased substantially and are more sys- rising importance in the Bank's portfolio. At the tematic and analytical. A noteworthy example is CAS and SSP level as well, risk assessments need the recently completed assessment of the Strate- strengthening. The Bank's current, especially gic Compact, which has offered a number of ex-ante, risk-management system relies on self- lessons as the Bank moves forward. Evaluation of assessment. New organizational solutions need to the CDF and PRSPs pose considerable challenges be explored to provide adequate, independent to the evaluation and control framework. oversight, without adding to transaction costs. Networks offer a potential solution, but their Monitoring role is currently limited. Good Practice examples Overall, the portfolio monitoring systems are from elite public sector organizations and pri- working well. A difficult area is the measure- vate companies should be explored. ment of performance at the country and sec- A rapidly evolving dimension in the Banks toral levels. As a result, the scorecard remains work is partnerships, which offer significant ben- incomplete, in part because of a lack of agreed efits, but also potential risks. The establishment methodology to measure the relevant indicators. of the Partnership Council is expected to ration- At the project level, a critical gap, previously alize the process Bank-wide and the Develop- identified by OED, is the weakness in M&E, in ment Grant Facility (DGF) has adopted part due to weaknesses in borrowers' evaluation procedures to evaluate its programs. It is not clear, capacities. The Bank is currently implementing however, what evaluation standards will be used the recommendations of a recent working group in Bank partnerships. Monitoring and the eval- on the subject. For the CDF/PRSP approach uation of outcomes from joint programs also and the HIPC initiative, it is clear that borrow- present a major challenge. ers often do not even have systems to track in- puts and outputs. Several initiatives are under way Control Activities to improve public expenditure management and Policies and procedures are fundamental controls accountability and to address the weakness in bor- to guide Bank operations. A policy framework rower statistical systems. It is important now to that is not fully up-to-date, lack of clarity in also address the weakness in evaluation capacity. 34 CONCLUSIONS AND RECOMMENDATIONS Given the importance of adequate M&E for the areas of concern, including procedures for eval- controls framework for development effectiveness, uating adjustment operations and for new prod- a more aggressive and comprehensive approach ucts and processes; agreeing and testing the CIF; is warranted to mainstream ECD. continuing to harmonize evaluation practices within and outside the Bank; and being even Information and Communications more transparent about data sources and Information systems have been substantially up- approaches in its evaluation reports. OED also graded. Knowledge management activities have needs to be more participatory and to revisit its also grown significantly, but without any explicit dissemination strategy. strategy. Nor is there a framework to evaluate these activities. The Bank is currently formulat- Recommendations ing a strategy and a stronger governance structure 1. Enhance the evaluation and control frame- for the Knowledge Bank. The vision is to de- work to encompass emerging priorities and velop a framework that encompasses all stages of initiatives. the knowledge cycle. The management of such a Suggested actions: system will be challenging, but the monitoring and a. Ensure that a coherent framework is in evaluation of the activities of the new Knowl- place to evaluate knowledge management edge Bank promises to be even more difficult. activities (including creating, sharing, and While separate efforts are under way to assess applying knowledge). and improve the effectiveness of ESW, client and b. Develop guidelines on evaluation standards staff training, and research activities, the frame- for partnerships through harmonization and work is fragmented and incomplete. As the Knowl- joint, up-front agreements with partners. edge Bank moves forward, it will be important to c. Mainstream ECD to ensure all CDF and ensure that a coherent evaluation and control PRSP countries have basic M&E arrange- framework is in place to guide its progress and to ments to permit evaluation of the programs. eventually permit its evaluation. 2. Complete the process of converting and up- dating the Operational Policies. OED Agenda Sugested actions: Following its renewal strategy, OED has shifted a. Convert the remaining policies, particu- resources to accommodate new priorities, par- larly safeguard policies, within a fixed time ticularly ECD, knowledge management, and out- frame. reach and dissemination. OED's work program b. Fill the gaps in the policy framework with has been endorsed by CODE as being relevant to respect to policies and guidelines for new current Bank priorities and the evolving opera- instruments. tional agenda. A priority for this AROE was to 3. Ensure that CASs are consistent with priorities assess OED's own effectiveness and impact. The as set out in SSPs. findings indicate that OED has an impact on Suggested actions:. staff learning and accountability, but there are areas a. Hold sector boards accountable for ensur- that need improvement. Timeliness, rigor, and ing that SSPs include explicit implementa- methodology are identified as the weakest aspects tion plans, provide strategic directions based of OED evaluations. The positive feedback on the Bank's comparative advantage, and is that OED evaluations are widely perceived give guidance on managing the tensions to be objective, relevant, and well grounded in between client and Bank priorities. current knowledge. b. Hold Regional sector managers account- OED needs to focus on several areas. These able, through the sector boards, for en- include improving the timeliness of its evaluations; suring consistent application of sector strengthening its evaluation methods to address priorities. 35 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION c. Develop sectoral indicators to be included ment lending and ensure new instruments in all CASs and hold country directors are adequately covered. accountable for consistency of Bank assis- b. Review and simplify project evaluation in- tance with identified sectoral priorities. struments and test and refine the CIE 4. OED should strengthen its methods and pro- c. Increase transparency of OED evaluations cedures. with respect to data, methodology, and Suggested actions: assumptions maintained to arrive at con- a. Review and improve procedures for adjust- clusions. 36 ATTACHMENT 1: MANAGEMENT RESPONSE Introduction countries. All are important areas that have re- Management welcomes the opportunity to dis- ceived recent Management attention. Specifi- cuss the 2000-2001 Annual Report on Opera- cally, Management will report to Executive tions Evaluation (AROE) by the Operations Directors in the fall on the management of knowl- Evaluation Department (OED). We appreciate edge activities, including the evaluation and the recognition in the AROE of the considerable control framework. The second set of recom- progress that has been achieved in recent years in mendations concerns operational policies. As the use of independent evaluation and self-eval- OED notes, the process for updating sensitive uation findings to improve client focus, respon- policies is complex and time-consuming. How- siveness, and operational quality. Management ever, in the spirit of transparency, it is important also appreciates the transmittal letter's clarifica- that key stakeholders have the opportunity to be tion that the AROE is not a balanced scorecard heard on significant policy issues, and this takes but by design highlights the remaining chal- time. In FY01, there was good progress on key lenges and risks, per the COSO framework. We safeguard updates, but in a world of change, pol- believe that it is important to address these chal- icy updating is continuous. Management will lenges, as set out below. report to Executive Directors before the end of the calendar year on the overall policy updating AROE Recommendations process and on its work program going forward. The AROE offers four sets of recommendations: The third set of recommendations concerns the three concern proposed actions by Management relationship between Country Assistance Strate- and the fourth concerns actions proposed for gies (CASs) and Sector Strategy Papers (SSPs): OED. In the attached matrix, Management how to balance country focus against sectoral responds to the first three sets, noting areas of priorities. The upcoming discussion of the SSP agreement and the actions it intends to take. stocktaking in the fall will be an opportunity to Management also offers its views in the matrix on address this issue. The last set of recommenda- the actions that OED recommends for itself, tions come from OED's self-evaluation and cov- since these actions will have an impact on ers the strengthening of OED's evaluation OED evaluations, an integral component of the methods and procedures. Management will process of managing for quality and development continue to work with OED as it proceeds with effectiveness. this work program, notably on lending evalua- tion and Country Assistance Evaluation (CAE) Overview of OED Recommendations. procedures, to align its self-evaluation method- The first set of recommendations addresses the ologies with those of OED. Better joint under- evaluation and control framework for emerging standing of OED's methodologies and data priorities and initiatives. The recommendations sources will improve the ability of operational staff focus on knowledge management, partnerships, to understand and make use of OED recom- and evaluation capacity development in client mendations. 37 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION Conclusion plan. Additionally, Management will maintain as The period since the last AROE has been one of a priority its focus on policy implementation, continued progress in using independent evalu- building on recent positive results. The two-way ation and self-evaluation as key tools for learn- linkages between CASs and SSPs will be strength- ing and for improving quality management. ened. We will continue to work on the difficult While not intended as a balanced scorecard, the task of moving up the self-assessment ladder to- AROE cites many of the improvements in the ba- ward stronger measures of Bank performance, to- sics for development effectiveness-quality, com- ward what OED recommends-verifiable pliance with policies, and delivery. However, as performance indicators for progress against the AROE notes, much more remains to be done; poverty-but we must emphasize that this process and it highlights several of the key remaining chal- is complex and time-consuming. Development lenges. Management agrees with most of the outcomes in client countries are a composite mix AROE recommendations and is taking action in of many internal and external factors. With this these areas. Management is working on evalua- degree of complexity, it is difficult to sort out at- tion and control frameworks for new activities, tribution, and the process will never be mecha- especially knowledge management. Operational nistic. In that regard, OED and operational staff policy updating is proceeding as fast as possible, need to improve their mutual understanding of in view of the many constraints and sensitivities, the framework used by OED for its CAEs and Management will continue to monitor progress for country program self-evaluation, a priority and will review the process and set out its work highlighted by OED for the coming year. 38 ATTACHMENT 1 OED Recommendation Management Response 1. Enhance the evaluation and control * Management agrees with the need to enhance the evaluation and control framework for framework to encompass emerging priori- new initiatives, notably the knowledge initiative. Management actions are detailed below. ties and initiatives. a) Ensure that a coherent framework is in * Management is in the process of articulating a unifying framework for the Bank's knowl- place to evaluate knowledge manage- edge work to consolidate and focus current knowledge-sharing activities on the front line ment activities (including creating, to ensure strong client/operational relevance and to strengthen monitoring and evaluation sharing and applying knowledge). of knowledge management activities. * Management will report to the Board in the fall on progress on knowledge activities, in- cluding the evaluation and control framework. b) Develop guidelines on evaluation stan- * The Bank has developed a strategic approach to partnership with the implementation of dards for partnerships through harmo- the partnership oversight process (November 2000) and the creation of, and refinements nization and joint, up-front agreements to, the Development Grant Facility(DGF). The DGF's FY01 Annual Review and FY02 Budget, with partners. recently approved by the Board, integrates evaluation, based on an evaluation and leam- ing strategy developed in consultation with OED, as a key requirement for ongoing DGF fi- nancing. Management is assessing the extent to which this systematic approach to partnership and program evaluation can offer lessons for all partnership arrangements. * Management will also look to OED's ongoing evaluation of global programs to inform this work. c) Mainstream Evaluation Capacity De- * In principle, Management agrees to the importance of ECD in all client countries, notably velopment (ECD) to ensure all CDF and in CDF/PRSP countries, but would raise the issue of timing and sequencing. The Bank con- PRSP countries have basic M&E tinues to provide support to ECD through a variety of mechanisms, including M&E compo- arrangements to permit evaluation of nents of loans, technical assistance, and grants. the programs. * Since traditional approaches have not necessarily produced sustained results, the Bank has implemented a pilot program to study and understand the issues of incentives, roles, and accountabilities for M&E (the institutional setting) and capacity within borrowers' govem- mental frameworks. * M&E methodology is also an issue. As knowledge of the CDF/PRSP process deepens, the M&E methodology needs to be carefully analyzed and refined. Management is looking to the OED/DEC CDF evaluation to help inform the development and implementation of M&E capacity in borrower countries. * Management will report to Executive Directors at the end of FY02 on progress under the pilot and future directions. 2. Complete the process of converting and * Management agrees on the importance of converting and updating operational policies. How- updating the Operational Policies. ever, we have learned that in a world of change, the policy process is an ongoing one, in- volving continuous evaluation, learning, and review. a) Convert the remaining policies, particu- * Substantial progress in converting and updating was made in FY01, and the groundwork larly safeguard policies, within a fixed was laid for further progress in FY02. Three safeguard policies-Natura/ Habitats, Projects timeframe. on Interational Waterways, andProjects in DisputedAreas-were updated in FY01. Three others, Involuntary Resettlement, Indigenous Peoples, and Cultural Heritage (now Physical Cultural Resources), passed critical milestones and submission to Executive Directors is ex- pected in FY02. The only remaining safeguard policy to be updated is Forestry, which is in draft and is also likely to be submitted to Executive Directors in FY02. * Management will come back to the Board by end December with a progress note on the overall policy conversion/updating process, outlining the Bank's work plan. b) Fill the gaps in the policy framework * The AROE notes specificallythe need to tailor Implementation Completion Report(ICR) guide- with respect to policies and guidelines lines to the special features of learning and innovation loans(LIts), and adaptable program for new instruents. ioans (APLs). * ULs: We need to ensure a consistent evaluation framework from concept stage to completion. The documentation proposing and appraising LILs has very recently been simplified and re- vised and QAG quality at entry criteria have been adapted. We have agreed with OED to work together on revising the ICR guidelines, with issuance in FY02. Continued 39 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION OED Recommendation Management Response 9 APLs: The gaps identifiled in the AROE have also been noted in Management's progress re- ports to the Board on adaptable lending. We have conducted training for staff on APLs, with a special focus on the development of appropriate performance triggers. Quality assurance at the Regional level has also focused on this. This year, we will disseminate good prac- tice examples of performance triggers, drawing on GAG reviews and, by end CY01, will am- plify the ICR guidelines to explicitly cover special features of APLs (performance triggers as wellI the learning from one phase to the next). 3. Ensure that CASs are consistent with pri- 9 Management agrees that CAS/SSP consistency is essential and is addressing this issue from orities set out in SSPs. both perspectives: a GAS Retrospective is currently under preparation for discussion with Executive Directors in the third quarter of FY0J2 and Management is completing an SSP stock- taking for discussion with Executive Directors in the fall. a) Hold sector boards accountable for en- * Management agrees with the first two points and will discuss how it proposes to ensure suring that SSPs include explicit imple- their implementation in the context of the upcoming SSP review. Management will address mentation plans, provide strategic the third point---on guidance for managing the tensions between client and Bank priori- directions based on the Bank's com- ties- as part of the SSP review parative advantage, and give guidance on managing the tensions between client and Bank priorities. b) Hold regional sector managers ac- * Management agrees with the need to ensure consistent application of sector priorities at countable, through the sector boards, the Regional level. While the Regional sector managers have an important role to play in for insuring consistent application of ensuring SSP/CAS linkages, the Regional vice presidents are ultimately accountable for Re- sector priorities. gional programs. c) Develop sectoral indicators to be in- * Management agrees that a comprehensive GAS diagnosis, looking across sectors, is es- cluded in all CASs and hold country di- sential for focusing on areas where country performance most constrains the pursuit and rectors (CDs) accountable for achievement of sustained poverty reduction. Sectoral indicators produced as part of the im- consistency of Bank assistance with plernentation of SSPs are important in this regard. identified sectoral priorities. a Management agrees in principle that CDs are accountable for the consistency of Bank as- sistance with identified sectoral priorities. However this wfll happen over time as sectoral priorities are sharpened and new CASs produced. The SSP stocktaking will propose mech- anisnms to sharpen the sectoral advice within each SSP; management is also exploring mech- anisms to improve dissemination, of SSP priorities/recommendations to country teams, including easy reference guidance toolkits. In the meantime, the Country Policy and Insti- tutional Assessment JCPIA) is proving to beaa useful tool for upstream identification of areas where sector performance is weak and should potentially be addressed in CASs. *However, Bank country assistance programs reflect other factors, in addition to identified sectoral priorities, that must be taken into account These include country vision and own- ership, country perfoima-fice, and the potential contribution of other partners, 4. DIED should strengthen its methods and e Management welcomes this undertaking. procedures. $a) Review and improve procedures for ad- * Management welcomes QED's plan to review and improve its procedures for evaluating ad- justment lending and ensure new in- justment lending and notes that most methodological issues arising for adjustment lend- struineins are adequately covered. Ing similarly arise for investment lending. For these reasons, a review of OEDYs evaluation methodology for both investment and adjustment lending would be useful. *Management stands ready to work with QED to establish agreed methodologies for self- vvaluation and (lED evaluation for lending products b) Review and simplify project evaluation *Management agrees and is ready to work during FY02 with (lED on the draft COF it recently instruments and test and refine the forwarded to us and on CAE methodology and country program self-evaluation in general. Country Information Form (C IF). c) Increase transparency of DIED ovalua- *Management welcome QlED's efforts in this regard since it should improve the rigor and quai- tons with respect to data, methodol- ity of (lED reports. A broader, deeper understanding of OlED's analytic base will improve the Dgy and assumptions maintained to Bank's understanding and acceptance of, and ability to make use of DED recommendations, arrive at their conclusions. 40 ANNEXES ANNEX A: THE INTERNAL CONTROL-INTEGRATED FRAMEWORK (COSO) The Committee of Sponsoring Organizations philosophy and operating style; the way man- (COSO) of the Treadway Commission devel- agement assigns authority and responsibility, oped an internal control framework to help or- and organizes and develops its people; and the ganizations to reduce the risk of asset loss, ensure attention and direction provided by the Board the reliability of financial statements, comply of Directors. with laws and regulations, and promote efficiency Risk Assessment: Every entity faces a variety of and effectiveness.' The COSO framework is risks from external and internal sources that widely recognized by professional bodies as a must be assessed. A precondition to risk as- standard in internal control evaluation criteria. sessment is the establishment of objectives, Internal control is broadly defined as a process, linked at different levels and internally con- effected by an entitys board of directors, man- sistent. Risk assessment is the identification and agement, and other personnel, designed to provide analysis of relevant risks to achievement of the reasonable assurance regarding the achievement of objectives, forming a basis for determining objectives in the following categories: how the risks should be managed. Because * Efficiency and effectiveness of operations economic, industry, regulatory, and operating * Reliability of financial reporting conditions will continue to change, mecha- * Compliance with laws and regulations. nisms are needed to identify and deal with Internal control systems operate at different special risks associated with change. levels of effectiveness. Internal control can be * Control Activities: Control activities are the judged effective in each of the three categories, policies and procedures that help ensure man- respectively, if the Board of Directors and man- agement directives are carried out. They help agement have reasonable assurance that: ensure that necessary actions are taken to * They understand the extent to which the address risks to achievement of the entity's operational objectives are being achieved. objectives. Control activities occur throughout * Published financial statements are being pre- the organization, at all levels and in all func- pared reliably. tions. They include a range of activities as * Applicable laws and regulations are being com- diverse as approvals, authorizations, verifica- plied with. tions, reconciliations, reviews of operating Internal control consists of five interrelated performance, security of assets, and segregation components: of duties. * Control Environment: The control environ- Info rmtion and Communication: Pertinent in- ment sets the tone of an organization, influ- formation must be identified, captured, and encing the control consciousness of its people. communicated in a form and timeframe that It is the foundation for all other components enables staff to carry out responsibilities. In- of internal control, providing discipline and formation systems produce reports, containing structure. Control environment factors in- operational, financial, and compliance-related clude the integrity, ethical values, and compe- information that make it possible to run and tence of the entity's people; management's control the business. They deal not only with 41 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION internally generated data, but also informa- combination of the two. Ongoing monitoring tion about external events, activities, and con- occurs in the course of operations. It includes ditions necessary to informed business regular management and supervisory activities decisionmaking and external reporting. Effec- and other actions personnel take in perform- tive communication also must occur in a ing their duties. The scope and frequency of broader sense, flowing down, across, and up the separate evaluations will depend primarily on organization. All personnel must receive a clear an assessment of risks and the effectiveness of message from top management that control re- ongoing monitoring procedures. Internal con- sponsibilities must be taken seriously. They trol deficiencies should be reported upstream, must understand their own role in the inter- with serious matters reported to top manage- nal control system, as well as how individual ment and the Board. activities relate to the work of others. They must have a means of communicating significant COSO at the Bank information upstream. There also needs to be The Bank adopted the COSO framework in effective communication with external par- 1995 to establish a common definition of man- ties, such as customers, suppliers, regulators, agement controls for all Bank units. It provides and shareholders. a standard against which to assess the adequacy Monitoring. Internal control systems need to of the Bank's internal controls. The overall re- be monitored-a process that assesses the sponsibility for COSO implementation rests with quality of the system's performance over time. management and is coordinated through Con- This is accomplished through ongoing mon- troller's. In implementing COSO, the Bank uses itoring activities, separate evaluations, or a the Control Self-Assessment (CSA) approach. 42 ANNEX B: FIFTH ICR PROCESS REVIEW An Implementation Completion Report (ICR) is includes a partial sample of FY00 exits (roughly required for each lending operation, and is the 45 percent). Bank's primary self-evaluation instrument. Com- Table B. 1 summarizes the data on completion pletion represents a milestone in the project cycle, reports received and reviewed by OED and the marking the transition from implementation to number of performance audits conducted since the project's future operation. OED conducts an FY93. These figures represent reports received independent review of each ICR. Through Feb- and evaluated each year, and do not correspond ruary 1998, this review was in the form of a writ- to project exit years. Differences between total ten Evaluative Memorandum (EVM), and since numbers received and reviewed are due to the then through an electronic document, the Eval- bunching of ICRs toward the end of the fiscal year. uation Summary (ES). This 2001 ICR process review is the fifth ICR Quality in a series reporting on the quality of project self- evaluations at completion. The third review Overall Quality of ICRs Continues To Be High reported on the quality of the last group of But May Be on the Decline ICRs reviewed in the EVM format. The fourth As summarized in table B.2, ICR quality ratings review presented information on the quality (assessed by project exit year) continue to be high of ICRs reviewed by OED through June 1999 and appear to be holding steady, from 95 percent under the new ES format. This fifth review for the FY97 evaluations to 94 percent for FY99 covers ICRs received through July 2000, which evaluations. The results for the partial sample of Table B.1 - FY96 FY97 FY98 FY99 FY00 Completion reports received 268 397 218 291 307 Adjustment 43 65 24 45 36 Investment 224 332 186 236 251 Completion reports evaluated 249 313 284 268 270 Adjustment 44 45 38 36 50 Investment 204 268 246 232 220 Completion reports audited 100 79 71 69 72 Adjustment 17 13 13 14 14 Investment 83 66 58 55 58 Audit ratio (0 audit/ICRI 40 25 25 26 27 Note: Figures in the table represent number of reports received and evaluated by OED in each fiscal year. 43 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION Table B.2 FY97 exits FY98 exits FY99 exits FY00 exits # Rated % Sat # Rated % Sat #Rated % Sat # Rated % Sat REGION Africa 66 92 86 99 77 , 90 34 79 East Asia and Pacific 40 98 38 100 39 95 .. 23 100 Europe and Central Asia 22 100 30 90 40 98 27 93 Latin America and Caribbean 54 93 53 94 53 92 17 10 Middle East and North Africa 21 95 17 88 14 100 5 100 South Asia 26 96 41 95 29 97 9 100 NETWORK Environmentally & Socially Sustainable Deeiopment 63 95 58 98 45 96 27 81 Finance, Private Sector. & Infrastructure 92 96 109 95 112 93 44 95 Human Development 38 92 54 94 55 93 30 97 Poverty Reduction & Economic Management 36 94 44 95 40 95 14 93 Total/average 229 95 265 96 252 94 115 92 Note:The data for FYOO exits represent a partial sample and reflect the processing of all ICRs received through July 2000. The number reviewed excludes projects rated not available, not applicable, and not rated. Sat. = Satisfactory. Figure B.1 100 - 95 96 5 r 92 92 92 91 88 III 80 .1 __ . . [' I 1995 1996 1997 1998 1999 2000* Exit FY Note: The data for FY00 exits represent a partial sample and reflect processing of all ICRs received through July 2000. 44 ANNEX B FY00 exits, at 92 percent, suggest a decline from networks and Regions are represented, with the the FY98 peak of 96 percent. To maintain an exception of MNA. Eight of the 23 (35 percent) overall 95 percent, of the remaining (about 55 per- exemplary ICRs in FY00 had unsatisfactory out- cent) FY00 exits, 97 percent would have to be come ratings, an increase of 10 percentage points rated satisfactory, which is feasible but needs to from FY99. There were no outcome ratings be watched. changes from ICR to ES. Sustainability was down- The findings from all five ICR process re- graded twice and upgraded once from ICR to ES. views, displayed in figure B. 1, reveal a rising trend Institutional development impact was down- in the quality of ICRs from FY95 to FY98, and graded twice and upgraded twice from ICR to ES. a steady record since then, assuming that the final There was at least one change in 30 percent (7 of FY00 results will be higher than the preliminary 23) of cases from ICR to ES, compared with 38 results. It should be noted that the ICR format percent (99 of 262) for all ICRs reviewed in FY00 and process were revised starting with the FY00 (among the ICRs judged to be of satisfactory exits, and it is too early to infer whether this has quality, 35 percent involved at least one change had any influence on ICR quality ratings. Over- in ratings, while among the unsatisfactory ICRs, all, however, it is evident that operational staff are 83 percent recorded at least one change). internalizing the general completion reporting The overall satisfactory level of ICRs reflects methodology. an improved focus on the quality of the self-eval- uation, and particularly the internal consistency The Share of Exemplary ICRs Has Increased of the report. In general, ICRs adhered to the by 50 Percent Since the Third Review basic reporting requirements, and the overall In addition to the overall satisfactory ICR qual- improvement in focus is an important achieve- ity, a steady number of ICRs are being rated ment. However, looking to the future, sustain- by OED as exemplary (table B.3). In FY00, 9 ing project outcomes beyond project closing, percent of the ICRs reviewed (23 of 270 ICRs the need to involve borrowers in evaluation, to received) were in this category (as in FY99, 24 work toward bridging the gap in borrower eval- of 266), compared with less than 5 percent (16 uation capacity, are areas where ICRs have been out of 334) in the third review. In table B.3, all chronically deficient. The following sections Table B.3 - . Pmject naie Country Project name Country Irngation Rehabilitation Albania Structural Adjustment Kenya Rural Development Albania Universities Investment Kenya Provincial Reform Argentina Structural Adjustment Latvia Structural Adjustment Chad Technical Assistance Mauritius Structural Adjustment Credit 3 Chad National Population Nigeria Structural Adjustment Credit 2 Chad Leyle-Cebu Geothermal Philippines Daguangba Multipurpose China Community Water Supply and Sanitation Sri Lanka Changchun Water Supply China Finance Companies Restructuring Thailand SE Coast Sewerage and Drainage Cyprus Economic and Financial Management Uganda Energy II Project Guinea Water Supply Rehab. Uruguay Maharashtra Power 2 India Low-Income Bamos improvement Venezuela Cirata Hydroelectric Phase 2 Indonesia 45 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION document the most recent findings, based on a transition arrangements. There is much room for review of ICRs for FY00 exits. Although based improvement. on a partial sample, the analysis uses FY00 exits Of the other two requirements for transition to serve the dual purpose of assessing the forward- arrangements, the percentage of ICRs with a sat- looking orientation and borrower input into the isfactory evaluation of the transition arrange- ICR process and reviewing the early experience ments is slightly lower, at 43 percent, and 41 with the new-style ICRs, which were designed percent do not even attempt an evaluation. A to address their traditional weaknesses. significantly lower number of ICRs, 19 percent, identify adequate performance indicators to guide Forward-Looking Orientation: future operations. Another 7 percent provide par- A Continuing Weakness tial indicators, but a vast majority (74 percent) do ICRs are expected to have a forward-looking not list any performance indicators. This is sur- orientation, rather than simply verifying imple- prising, since 68 percent of the ICRs contain mentation goals. In the past two ICR process performance indicators for the implementation reviews, particular attention was given to "future phase of the project and another 19 percent pro- operation" and borrower's input or evalua- vide at least partial indicators. One reason for tion of the project. These aspects are reviewed this could be that staff are not fully aware of the again here, along with the quality of perform- ICR guidelines. ance indicators used to substantiate project performance. Partner Comments: Borrower Input Improving The revised ICR guidelines stipulate three Under the revised guidelines, borrower input is key requirements for transition arrangements to be included in the section on partner com- for the project's future operation: (a) a descrip- ments, with the substantive borrower evalua- tion of the arrangements, (b) an evaluation of tion, if one exists, attached as an annex. In arrangements, and (c) a list of performance in- general, the review finds a need for further clar- dicators to be used to monitor and evaluate the ification on what is expected under the new project in the future. guidelines (it is not clear if the included com- A content analysis reveals an improvement rel- ments are "partner's comments" or "comments ative to previous years, perhaps because of the on the borrower's input"). changes introduced. In FY98 and FY99, 54 per- The main findings are that 7 percent of the cent and 39 percent of ICRs, respectively, had no ICRs did not have either partner comments or plans for future operations. Another 25 percent a borrower's ICR, evaluation report, or sum- and 37 percent in the respective years missed the mary of borrower input. About 12 percent in- intent of the OP (that is, they discussed future op- cluded only partial comments in lieu of borrower erations in the country/sector, but not the future input. About 22 percent included an accept- operation of the project in question). Overall, able summary of comments, and a majority (58 only 22 percent of projects in FY98 and 24 per- percent) incorporated a detailed contribution cent in FY99 had satisfactory plans for future by the borrower. These findings maintain the operations. By comparison, of the 117 FY00 trend observed in the past two years-the per- ICRs reviewed for this report, 35 percent have no cent of ICRs with no borrower comments has description of transition arrangements, and an- fallen from 22 percent in FY98 to 9 percent in other 19 percent have an unsatisfactory descrip- FY99 and to 7 percent in the partial FY00 sam- tion. Overall, about 46 percent of the ICRs have ple. The previous year's analysis did not pro- a satisfactory description of transition arrange- vide as detailed a breakdown as was done this year ments (including 13 percent with a marginally sat- in the quality of borrower input, so a compari- isfactory description). This is a significant son is not possible. improvement from FY98 and FY99, although a Overall, the findings indicate substantial im- majority still do not provide a description of the provement in borrowers' contributions to the 46 ANNEX B ICR process, although work on improving their respectively, for FY96-97. Total ratings dis- quality remains to be done. crepancies between the PSR and ES amount to 12 percent, with net changes at 8 percent of Ratings Changes FY98-00 exits. The changes in ratings between ICR and ES Principal Ratings at Closing for sustainability and institutional development Table B.4 shows the changes in ratings at differ- continue to be substantial at 20 percent. Fur- ent stages from the final PSR to PAR in the case ther, 23 percent for sustainability and 25 per- of outcomes, and ICR to PAR in the case of sus- cent for institutional development are changed tainability and institutional development. The between the ES and PAR. For the aggregate port- breakdown of all FY96-00 exits into two periods folio, however, the implied corrections are smaller, (FY96-97 and FY98-00) does not reveal any sig- but nevertheless important, for the sustainability nificant differences, with the exception of changes rating, with 15 percent and 10 percent successive in outcome rating from ICR to ES. The data in- downgrades from ICR to ES and from ES to dicate a decline in the downgrades from the ICR PAR. The implications for the aggregate portfo- to ES and from ES to PAR for outcomes, and from lio performance for institutional development ES to PAR for sustainability and institutional de- are less severe: with 3 percent and 5 percent velopment ratings. downgrades from ICR to ES and then from ES The largest number of outcome rating changes to PAR. occurs between the PSR to ICR, at 11 percent The Regional and network breakdowns of for the period FY98-00. These figures have re- the ratings discrepancies are given in tables B.5 mained virtually the same since FY96-97. In con- and B.6, which show that the greatest number trast, the changes between ICR and ES occur in of changes in outcome ratings are for Africa and 4 percent of the cases. In net downgrades, the ESSD, largely at the PSR-ICR stage (that is, change from PSR to ICR is again larger, at 5 per- before the ICR gets to OED). Sustainability cent, than between ICR and ES, at 2 percent in ratings changes occurred least often in ECA and FY98-00, down from 6 percent and 4 percent, LCR (14 and 15 percent, respectively), and most Table B.4 I FY96-97 exits FY9S--00 exits FY96--0 exits % % % % % % % % % % % % Down Up Overall Net Down Up Overall Net Down Up Overall Net Outcome PSR to ICR 9 3 11 6 8 3 11 5 8 3 11 6 ICR to ES 6 2 8 4 3 1 4 2 5 1 6 3 PSR to ES 12 2 14 10 10 2 12 8 11 2 13 9 ES to PAR 7 2 9 5 4 4 8 0 6 3 9 3 Sustainability ICR to ES 17 1 19 16 17 3 20 14 17 2 19 15 ES to PAR 19 6 25 13 13 10 23 4 17 7 24 10 Institutional Development Impact ICR to ES 12 7 19 5 11 9 20 3 11 8 19 3 ES to PAR 18 11 29 7 13 11 25 2 16 11 27 5 47 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION Table B.5 PSR to ICR: outcome ICR to ES: outcome ES to PAR: outcome FY98 FY99 FY00 FY96-0 FY98 FY99 FY00 FY96-IH FY98-00 FY91"0 Region Africa 19 12 15 14 5 8 6 8 11 12 East Asia and Pacific 13 5 4 8 3 3 0 5 0 14 Europe and Central Asia 3 5 4 6 3 10 4 5 13 6 Latin America and Caribbean 11 4 0 9 9 0 0 6 0 3 Middle East and North Africa 0 0 20 9 0 0 0 5 0 0 South Asia 18 24 11 17 7 0 0.5 8 10 Network Environmentally and Socially Sustainable Development 21 13 12 13 5 7 8 8 0 14 Finance, Private Sector and Infrastructure 15 10 9 12 6 5 0 6 12 11 Human Development 7 2 7 8 4 2 3 6 0 0 Poverty Reduction and Economic Management 7 10 0 10 7 5 0 5 25 4 Total/average 13 9 8 11 5 4 3 6 8 9 Note: The data for FY00 exits represent a partial sample and reflect the processing of all ICRs received through July 2000. Table B.6 . Sustainability Institutional Development FY98 FY99 FYO FY96-MO FY98 FY99 FYOD FY96-- Region Africa 23 19 24 22 20 26 16 19 East Asia and Pacific 24 18 22 19 26 26 17 23 Europe and Central Asia 20 20 4 14 17 23 15 19 Latin America and Caribbean 15 14 24 15 12 20 29 19 Middle East and North Africa 24 21 20 27 6 21 20 18 South Asia 32 14 11 21 20 14 22 17 Network Environmentally and Socially Sustainable Development 19 18 33 19 21 16 24 18 Finance, Private Sector and Infrastructure 20 19 12 19 22 26 21 23 Human Development 37 18 13 25 11 25 20 18 Poverty Reduction & Economic Management 16 13 14 13 12 16 0 15 Total/average 23 18 18 19 18 22 19 19 Note:The data for FY00 exits represent a partial sample and reflect the processing of all CRs received through July 2000. 48 ANNEX B often in MNA (27 percent). For HD, the figure bias in the ratings. For example, some projects are was 25 percent, and for PREM, 13 percent. audited because of a difference of opinion between ESSD and FPSI were tied at 1.9 percent. ID im- OED and the Region, stemming from a dis- pact changes occurred most often in EAP (23 per- agreement on ratings between the ICR and ES. cent) and least often in SAR (17 percent). FPSI In other cases, lack of information in the ICR or had the most changes (23 percent), and PREM other reasons raise doubts about project out- had the least (15 percent). comes and can lead to a project being audited. In total, 37 percent of ICRs reviewed by OED Hence, the changes in ratings at the audit stage received a ratings change in at least one category may not allow generalizations about the overall (table B.7). Based on the findings of performance quality of ICR self-evaluations or the implied audits, 43 percent of the audited projects had overall performance of the Bank's portfolio. further ratings changes in at least one category. At the time of the ICR review, each project is categorized by its priority for a performance audit ES Ratings Are Only Part of the Picture (high, medium, or low). Actual audits conducted, The data presented above clearly indicate that a however, do not always target projects rated as high majority of the ratings changes for outcomes priority for several valid reasons. These include the occur before the ICR comes to OED (table B.4). need to evaluate project performance to feed into This suggests that many task managers have an a CAE to serve as a building block for a sector or unrealistic view of the quality of their projects. thematic evaluation, to form a cluster for a coun- Interpretation of changes that occur from the try sector review, or to be part of a cluster audit ES to the PAR needs to be done with care. Since for cost-efficiency reasons. While the majority only 25 percent of the projects for which ICRs of projects audited belong to the high-priority are reviewed by OED are audited, there is a po- category, medium- and low-priority projects tential for a selection bias, and hence a potential permit a comparison across the three categories Table B.7 L..-IJL- ICR to ES ES to PAR At least one change At least one change # Rated # Changed % Changed # Rated # Changed % Changed Region Africa 331 131 40 42 18 43 East Asia and Pacific 167 62 37 21 11 52 Europe and Central Asia 140 43 31 18 6 33 Latin America and Caribbean 216 74 34 32 15 47 Middle East and North Africa 76 31 41 4 3 75 South Asia 132 47 36 31 10 32 Network Environmentally and Socially Sustainable Development 248 91 37 35 12 34 Finance, Private Sector and Infrastructure 434 171 39 64 30 47 Human Development 217 8? 38 23 11 48 Poverty Reduction and Economic Management 163 44 27 26 10 38 Total/average 1.062 388 37 148 63 43 Note: The data for FY00 exits represent a partial sample and reflect the processing of all ICRs received through July 2000. 49 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION to identify whether or not the ratings changes for far. An important dimension for these ratings is high-priority projects are representative or gen- the elapsed time between the project implemen- eralizable. Based on these priorities, the projects tation, which ends with the writing of the com- with high priority for audit received the most pletion reports, and OED evaluation. This delay changes (FY96-00 exiting projects audited thus permits a more accurate view of the longer-term far; see table B.8). The results show a surprising project impact, especially for audits. Table B.9 uniformity across all audit-priority categoriza- shows that the longer the evaluation delay, the tions. The ratings discrepancy for all three meas- more ratings are changed, for ICR to ES. The same ures-outcome, sustainability, and institutional is also true for PARs, which are undertaken 1-3 development-is the highest in both overall and years after the ES, as shown in table B. 10. net terms for the projects rated as low priority Table B. 11 shows the delay between ICR and for audit. ES by OED. For FY96 exits, 50 percent of ICRs As noted earlier (table B.7), OED made a had been reviewed 3 months after the ICR date; change from the ES in at least one rating category for FY99 exits, 79 percent were reviewed within in 43 percent of the FY96-00 exits audited thus 3 months. The average delay declined from 3.83 Table B.8 -T -1" Audit priority High Meditan Low All Number of ES 249 240 588 1,077 Number of audits 83 31 33 147 Percent audited 33 13 14 Number of changes (ES TO PAR) One change (%) 20 39 33 27 Two changes (%) 12 16 15 14 Three changes (%) 1 0 26 2 Total number with any change (%) 34 55 55 43 Outcome Ratings Number rated (%) 83 31 33 147 Upgraded(%) 2 6 0 4 Downgraded (%) 6 0 12 10 Total changed (%) 8 6 12 14 Sustainability Ratings Number rated (%) 82 31 33 146 Upgraded (%) 9 10 3 8 Downgraded (%) 16 13 24 17 Total changed 24 23 27 25 ID Impact Ratings Number rated (%) 83 31 33 147 Upgraded (%) 6 19 15 11 Downgraded (%) 10 23 27 16 Total changed (%) 16 42 42 27 Note: The data for FY00 exits represent a partial sample and the processing of all ICRs received through July 2000. 50 ANNEX B Table B.9 Outcome changes Sustainability changes ID impact changes # % % % # % % % # % % % ICR to ES Delay Rated Down Up No chg Rated Down Up No chg Rated Down Up No chg 0-1 months 434 2 2 96 436 12 2 86 426 10 8 8 2-3 months 395 5 1 94 393 20 2 78 391 13 8 79 4-5 months 173 8 1 91 173 20 2 79 167 10 6 84 6-7 months 61 7 3 90 60 22 5 73 60 15 15 70 8-9 months 11 9 9 82 11 27 9 64 11 18 9 73 1(0+months 3 67 33 0 3 33 0 67 3 33 0 67 Total 1,077 5 1 94 1,076 17 2 81 1,058 11 8 81 Table B.10 'f'''I~Z L~ 7 Outcome Sustainability ID impact Audit delay interval # # Changed % Changed #Changed % Changed # Changed % Changed 0 (before ES) 12 0 0 3 25 3 25 Less than 1 year 48 2 4 10 21 10 21 lyear 60 8 13 19 32 21 35 2 years 25 3 12 4 16 6 24 3 years 3 0 0 0 0 0 0 Total 148 13 9 36 24 40 27 Table B.11 . FY96 FY97 FY98 FY99 FYOD FYSID Interval # % # % # % # % # % # % 0-1 months 12 5 64 28 134 51 137 55 87 78 434 40 2-3 months 99 45 131 57 81 31 61 24 23 21 395 37 4-5 months 73 33 28 12 35 13 36 14 1 1 173 16 6-7 months 26 12 4 2 13 5 17 7 1 1 61 6 8-9 months 8 4 2 1 1 0 0 0 11 1 1(h+months 3 1 0 0 0 0 3 0 Total 221 229 264 251 112 1,077 Average (months) 3 83 2.46 2.09 2.06 1 21 2 42 51 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION months in FY96 to 2.06 months FY99. Of the Comparing the overall quality of the two types partial sample available for the FY00 exits, 99 of ICRs, 96 percent of CAls and 94 percent or percent were reviewed within 3 months of the ILIs are rated as satisfactory. There is little dif- ICR date. ference between the ILls and the CAls in any of the major ratings (table B.12). One important New-Style ICRs: Differences in Quality change visible is that with the ILls, 59 percent Among ILls and CAls of the ICRs were written by the task manager of In the first half of FY00, 117 projects were in the the project at closing, compared with 33 percent sample that exited the Bank's portfolio, and their of CAls. This is consistent with the intent of main features are analyzed by Region and are the intensive learning feature of the ILIs, and also shown in tables B. 12 and B. 13. The ICR process with the anticipated higher quality of ILIs (table review introduced changes in the requirements and B. 13). On other counts, however, CAls perform an electronic format that has enforced ICR adop- better with respect to borrowers' input and part- tion of the new guidelines, although some ex- ner comments (89 percent versus 71 percent), and ceptions remain. Among the changes introduced, stand even on performance indicators. ILls per- the new guidelines, in the form of OP/BP 13.55, form better on the description of transition require that 30 percent of the ICRs be ILIs (In- arrangements (53 percent versus 46 percent), tensive Learning ICRs) and rest to follow a leaner but poorly on the evaluation of these arrange- format, called the CAI (Core Accountability ments (35 percent versus 46 percent) as well as ICRs). Of the ICRs received so far, only 15 per- on the inclusion of performance indicators for cent are ILIs, while 10 percent (12 ICRs) have fol- future operations (12 percent versus 20 percent lowed the previous ICR guidelines. having satisfactory indicators). Table B. 12 FYO0 Exits Down Up No. change Down Up No change Overall Net Outcome PSR to ICR Core 6 2 76 7 2 90 10 5 ILl 1 0 16 6 0 94 6 6 ICR to ES Core 1 2 81 1 2 96 4 -1 ILl 0 0 17 0 0 100 0 0 Sustainability ICR to ES Core 9 3 72 11 4 86 14 7 ILl 2 0 15 12 0 88 12 12 ID impact ICR to ES Core 13 4 67 15 5 80 20 11 ILl 2 0 15 12 0 88 12 12 52 ANNEX B Table B.13 Core ICRs using Intensive Accountability previous Learning ICRs ICRs guidelines All ICRs % No % No % No % No Total completion reports 100 17 100 84 100 12 100 113 Lending instrument ERL 6 5 6 FIL 1 1 1 1 SAD 5 4 4 4 SAL 6 1 2 2 17 2 4 5 SIL 88 15 64 54 75 9 69 78 SIM 10 8 7 8 TAL 6 1 11 9 8 1 10 11 ICR author Not TM at completion 41 7 67 56 67 8 63 71 TM at completion 59 10 33 28 33 4 37 42 Performance indicators Adequate 71 12 71 60 42 5 6h 47 None 0 10 8 58 7 13 15 Partial 29 5 19 16 19 21 Partner comments Detailed contribution included 65 11 63 53 17 2 58 66 Summary comments included 6 1 25 21 25 3 22 25 Partial comments only 24 4 8 17 2 12 14 No comments 6 1 2 2 42 5 7 8 Transition arrangements 4 1. Description Satisfactory 12 2 42 35 33 37 Marginally satisfactory 41 7 4 3 42 5 13 15 Unsatisfactory 29 5 18 15 8 1 19 21 None 18 3 37 31 50 6 35 40 2. Evaluation Satisfactory 35 6 42 35 25 3 39 44 Marginally satisfactory 0 0 4 3 8 1 4 4 Unsatisfactory 18 3 18 15 8 1 17 19 None 47 8 37 31 58 7 41 46 3 Performance indicators ifor future operavon Satisfactory 12 2 20 17 17 2 19 21 Partial 12 2 7 6 0 7 B None 76 13 73 61 83 10 74 84 Note: Of the FY00 completion reports received, three were PCNs, which have been excluded from this analysis. 53 ANNEX C: EVALUATION CAPACITY DEVELOPMENT Improving M&E remains a corporate challenge. that will be used to track performance against na- An important control gap identified by OED in tional budget targets. The Bank is also working the past has been a lack of information on what to deepen its partnership relationships with other is achieved in terms of outcomes and impacts donors in these countries. Further, 25 operations from Bank-financed activities. This reflects the in the pilot countries (at the approval and im- weaknesses of M&E arrangements, due in part to plementation stages) are also using participatory weak capacity for M&E in borrowing countries. workshops to focus on RM&E. Borrower commitment to M&E is also a key OED is also supporting efforts to strengthen issue. In an effort to fill this gap, the Bank, with capacity for M&E in borrowing countries. OED's support from OED and other units, has pro- main objective is to help mainstream M&E in the moted Evaluation Capacity Development (ECD). Bank in the sense that country teams recognize Several Bank units are currently involved in the importance of ensuring in-country M&E ca- ECD activities. They include OED in its long- pacity and include ECD in their public sector re- standing catalytic role, WBI in its provision of form efforts. To this end, OED is supporting M&E and other PRSP-related training, DEC country teams in designing sound M&E systems and the Regional VPUs working on statistical ca- at the national and sectoral levels. It has prepared pacity-building, OPCS in its support role to Op- resource material for Bank staff and borrowers, erations, individual country teams, and the public such as diagnostic guides, country case studies, sector group. Information exchange among these publications, workshops, and a website. It has units on ECD-related activities already exists and also facilitated provision of training and trainer plans are under way to strengthen them. training in M&E techniques. Currently, OED is As part of the effort to assess the Strategic providing support to the country teams for Ghana, Compact, the Bank established an M&E work- the Kyrgyz Republic, and Uganda. Catalytic sup- ing group in 1999 to develop a comprehensive ac- port has also been given to Benin, Brazil, Hon- tion plan to achieve sustainable improvements in duras, Madagascar, the Philippines, Poland, M&E in Bank-financed projects. On the basis of Romania, and Tanzania. By OED estimates, cur- recommendations of the working group, the rently about 10 country teams are actively sup- Bank has chosen five pilot countries where ECD porting early-stage ECD efforts. will be a major thrust. These are the Kyrgyz Re- Once ECD is successfully mainstreamed into public, Madagascar, the Philippines, Romania, Bank operations, OED expects to limit its role to and Tanzania. The pilots are varied in scope and periodic evaluations of the Bank's ECD activities. in various stages of implementation. For exam- Given the recent and varied nature of most of these ple, in the Kyrgyz Republic, Romania, and the efforts, there is currently little evaluative infor- Philippines, the pilot is broad-based in scope. mation on the impact of the Bank's efforts in The governments are in the process of defining ECD. Thus, OED has also been asked to play an results-based monitoring and evaluation (RM&E) oversight role. The Board has asked OED to pre- frameworks in the central and line ministries pare annual reports on ECD, starting in FY02. 55 s ANNEX D: OED KNOWLEDGE MANAGEMENT ACTIVITIES In FY99, OED developed a strategy to promote in developing their knowledge management the use of OED's evaluation results and to de- (KM) systems. velop networks and engage internal and external One of the dissemination mechanisms em- evaluation partners to promote learning from ployed by OED is participation at knowledge OED evaluations. Major activities to implement fairs. In FY01, OED participated in ten such this strategy include the establishment of an events: the Global Public Policy Conference, evaluation Help Desk, a service that responds to CDF workshop, 2000 World Bank/IMF Annual internal and external inquiries; participation at Meetings, Social Funds Workshop, PREM Week, important evaluation conferences, sector weeks, annual meetings of the American and European and Bank-wide knowledge fairs, and events to Evaluation Associations, Rural Week, Infrastruc- mainstream OED's work within the Bank and the ture Week, and the OED annual retreat. These development community; collaboration with events provide effective diffusion of OED's eval- other Bank evaluation units in the creation of an uation findings and lessons through direct con- evaluation community of practice; support to tact with event participants. OED evaluation task teams by synthesizing re- The evaluation, monitoring, and quality en- sults from OED evaluations and other relevant hancement community of practice launched in sources; and the development of an evaluation FY00 draws evaluation practitioners from around glossary in partnership with OECD in an effort the Bank and provides a forum to develop and ex- to harmonize evaluation terminology and pro- change evaluation news, activities, products, and cedures with other development partners. OED work program information. A shared web portal also provides leadership for the aid effectiveness designed in FYO1 provides a collection of evalu- section of the Development Gateway initiative ation material generated by evaluation units within and is responsible for coordinating relevant the Bank and links to those units for visitors to content submissions of development partners, the Bank's external website. including bilaterals, multilaterals, NGOs, and The KM activities are also a valuable input into civil society. OED evaluations, through background research The Help Desk responds to inquiries on OED and synthesizing findings and lessons. These in- evaluation methodology, findings, lessons, and puts to OED evaluations and responses to exter- recommendations. Since its inception in Janu- nal inquiries provide the basis for the Lessons ary 1999, approximately 3,000 such inquiries Paper Series. Four such papers were published have been processed, of which 75 percent were online in FY01: Recurrent Lessons in India, generated from OED internal and external web- Lessons on Community-Driven Development, sites. External partners contemplating similar Utilization of Project Implementation Units, and knowledge management operations within their the World Bank's Experience with Institutional agencies have contacted OED for assistance Development. 57 ANNEX E: OUTREACH, DISSEMINATION, AND PUBLICATIONS The Outreach and Dissemination strategy of OED Foreword, electronic catalog of publica- OED, crafted in 1999, is designed to support and tions, OED Reach, Evaluation Updates, E-Posters, enhance OED's role inside and outside the Bank and E-Reach all broke new ground. OED by informing the development community about launched its first set of live, hyperlinked, interactive results on the ground. The strategy rests on five documents with the completion the CDF Pro- program pillars: (i) strategies and guidelines, in- ceedings site, offered live video webstreaming of cluding the Bank disclosure policy and internal the Poverty Study Workshop to Bank networks procedures and guidelines, (ii) corporate activities and thematic groups, and has focused on maxi- including Board, Senior Bank Management, Bank mizing the benefits of the opportunities offered staff communications, and external communi- by technology in increasing OED's outreach. As cations; (iii) electronic and web publishing; (iv) part of this effort, the OED website has been re- print publishing; and (v) dissemination and out- designed to better serve its clients. At the same reach campaigns. time, recognizing that there are large, important The objective of the strategy is to enhance the groups, including many NGOs, that cannot be Bank's development effectiveness and its battle reached through computer-based dissemination against poverty by strengthening OED's contri- strategies, OED continues to develop, innovate, bution to the Bank's knowledge base and be- and disseminate products to meet their needs. coming a better learning organization. OED The outreach and dissemination Work Pro- achieves this by mainstreaming evaluation find- gram exceeded its commitments for the fourth ings into the policy and decisionmaking process consecutive year. In the past two years, OED and by sequencing its reports to match and inform has published 24 books, 35 Working Papers, Bank country and sector policy renewal. OED's over 100 Briefs and Pr6cis, and over 250 multi- outreach and dissemination activities also extend lingual editions. It has disseminated elec- its reach beyond the Bank to the global develop- tronic/interactive publications and documents, ment community and its stakeholders, including conducted live video webstreaming, and made donors, partners, civil society, nongovernmental both interactive and static web documents avail- organizations, academia, and borrowers. This is able. Forty ARDE workshops have been held accomplished through strategic internal and ex- Bank-wide during the past two years, and cor- ternal communications and policies, interactive porate kits, brochures, briefing binders, posters and participatory evaluation processes and report and PowerPoints produced. Over a quarter of dissemination, publications, electronic dissemi- a million documents were disseminated in FY00 nation, workshops, media, and innovative infor- alone. This is accomplished through targeted mation technology applications. direct mailing, by direct request, through the In implementing this strategy, OED has fo- InfoShop, and at workshops and other events. cused on broadening it's reach to internal and ex- Writing workshops have also been held to assist ternal clients through continuous innovations. the staff in delivering OED findings clearly and The OED Country Case Series; Forestry Study; effectively. 59 ANNEX F: FYO1 OED SUMMARY WORK PLAN AND RESULTS FRAMEWORK (AS OF Q3, FY01) FY99 FY00 FY01 FYO1 FY01 Regular program Actual Actual Plana YTD Proj.0 Output indicators Project Evaluations ICR Reviews Number 279 281 280 193 280 Plan: 75% completed within 60 days from receipt in OED; actual: 80%. $ Million 0.8 1.2 1.2 0.7 1.2 Plan: 100% completed within 90 days from receipt in OED: actual: 90%. Project Assessments Number 72 70 70 29 70 Average completion elapsed time is lower than FYO but falls short of the FYO1 target of 90 days. $ Million 2.9 3.1 3.1 1.7 3.1 Project Impact Evaluations Number 3 1 Product discontinued in FY01. $ Million 0.1 0.1 Country Assistance Evaluations Number 15 10 7 5 7 All five CAEs delivered to CODE ahead of CAS review. $ Million 3.9 3.2 3.5 2.0 3.5 Sector and Thematic Evaluations Number 7 6 5 2 4 One OED SSP input delivered to CODE in advance of SSP review. $ Million 2.9 5.3 3.7 3.7 4.5 Corporate and Process Evaluations Annual Reports Number 2 2 2 1 2 ARDE workshops/seminars ongoing. $ Million 1.3 1.0 0.9 0.8 0.9 Process Evaluations Number 2 1 1 2 $ Million 1.3 0.5 Other Number 1 2 2 $ Million 1.6 0.3 0.4 0.7 0.4 (continued) 61 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION FY01 OED SUMMARY WORK PLAN AND RESULTS FRAMEWORK (AS OF Q3, FY01) (continued) FY99 FY00 FY01 FY01 FY01 Regular program Actual Actual Plana YTD8 Pro.6 Output indicators Evaluation Development Evaluation Capacity Development Number 2 6 8 7 7 ECD programs in seven countries; began work on one country-led evaluation. $ Million 0.8 0.6 1.0 0.7 1.0 KM, Outreach, and Learning $ Million 1.9 2.3 3.1 2.0 3.1 Support for OED evaluations, extraction of lessons learned from OED's evaluations, assistance to evaluation units of external agencies, responding to requests for information both internal and external to the Bank, enhancement of OED's website (including training), electronic surveys on culture and gender participation in knowledge events relating to evaluation, and taking the lead in developing the aid effectiveness section of the Bank's Global Development Gateway. The move toward electronic dissemination has been inten- sified, including a live webcast with the release of the poverty study. Outreach and dissemination outputs included five new editions of OED's studies, 4 pr6cis, 3 Fast Track Briefs, 15 Working Papers, and various kiosks with OED materials for workshops. Methods and Staff Development $ Million 0.4 0.8 1.4 0.2 0.6 New CAEs have been standardized on the three dimensional triangulation methodology adopted at the end of FY00. An OED-OPS Task Force on Country Methodology has endorsed use of the Country Information Form in the new Bank score- card. A shortened version of the CIF, suitable for use by country managers in evaluating assistance results, is nearing completion, and will be ready for piloting in the spring. Note: Amounts shown in FY01 dollars. a. Excludes FY00 carry-over amounts and other transfers to OED during the fiscal year. YTD is thru March 31, 2001. 62 ANNEX G: OED CLIENT SURVEYS The design of the 2001 OED Outcome Survey Sample Selection and Administration sought to go beyond the pilot study undertaken in The Bank staff surveys were administered through 2000 and expanded both the number and the Lotus Notes from the OED Help Desk. The bor- scope of the surveys. The purpose of the expansion rower surveys were administered by e-mail and fax. was to gain greater depth of insight into the reach All the surveys were designed to ensure that the re- and impact of OED reports. Twelve surveys were sponses would be anonymous, and all cover memos administered, as shown in table G. 1. and reminders reiterated the guarantee of anonymity. Table G.1 -1I7J JK'' .-c Target audience/ Response Number of target OED product Description rate (%) responses Executive Directors' Staff Survey of 135 advisers, assistants, and alternatives to the Executive Directors, including CODE staff. 41 56 Secretariat staff Survey of Secretariat Staff (n= 7). - 1 Quality group Survey of Regional Quality Assurance, Network Anchor, and DEC staff level GE and above (n = 549). 17 96 Bank staff (random group) Simple random sample survey of 397 Regional, Network Anchor, EXT, and WBI staff level GE and above. 11 45 Sector and Thematic Evaluations Simple random sample survey of 503 Bank staff from the relevant Thematic Groups regarding three Sector & Thematic Evaluations, representing a 50% sample of all CYO0 S&T studies. Results are aggregated across the three surveys (Gender, Information Infrastructure, and Poverty). 25 125 Performance Audit Reports Survey of 98 Bank staff engaged by OED in the course of undertaking 20 audit reports, representing a 40% random sample of all CYO0 audits (n = 51). 21 21 Country Assistance Evaluations Survey of 94 Bank staff engaged by OED during preparation of CYOO CAEs(n = 10). 28 26 CAS (without CAE) TMs Survey of 76 Bank staff engaged in CAS preparation for countries in which there has not been a CAE. 47 36 Borrower survey-Audits Survey to government officials engaged by QED in the course of undertaking CYOD Project Performance Audit Reports (same 40% sample of CYO0 audits as for Bank staff survey) (n = 56). 32 18 Borrower survey-CAEs Survey to government officials engaged by OED during preparation of CYOD CAEs (minus Kazakhstan, n = 9). 44 4 63 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION Care was taken to ensure that no Bank staff covered in the surveys targeted to client groups as member received more than one survey. To ac- one of the categories of OED products on which complish this, a sample of staff was selected (ei- opinions were sought. ther by random or purposive random sample) Six studies were candidates for the S&T stud- for each survey group. Each sample was over-se- ies survey. Three were selected. Of the rest, one lected to ensure that duplicate names could be was a follow-up to a study included in last year's eliminated. The selection of individual names for pilot survey (Rural Development). To avoid going the product-specific surveys was given prefer- back to the same staff a second time, this study ence. That is, the lists for product-specific surveys was not included. Another report is the subject were compiled first (a targeted, nonrandom group of a tracer study (Forestry), and the third study who were on the distribution lists for PARs and was deemed to have a very narrow target group CAEs). These names were then eliminated from (CFA devaluation). The distribution lists for the the random samples for other surveys. studies chosen (Poverty, Gender, and Information A primary concern in the design of the surveys Infrastructure) were compiled in consultation was to keep them short. The three product-spe- with the task managers of the studies and the rel- cific surveys from the 2000 exercise were ex- evant thematic group coordinators. Once the panded to allow for more questions. An effort was target population of staff was identified, a ran- also made to retain certain elements to allow for dom sample was chosen from the large target pop- comparison between the two time periods (Feb- ulations of the Gender and Poverty studies and ruary 2000 and February 2001). An additional the entire audience for Information Infrastruc- survey was designed and implemented for CAS ture study was chosen (because it was small and task teams for which there had not been a CAE. very specific). Other surveys were added to access the impact of OED products on different target groups in Response Rates the Bank. These included similar surveys for (a) As shown in table G.1, the response rates range the Bank's "Quality group," identified as the Re- from 11 to 47 percent, with an overall response gional Quality Assurance staff, network anchor rate of 27 percent. Multiple e-mail reminders staff and DEC, staff level GE and above; (b) the were sent to all groups, and telephone follow-up staff of the Executive Directors, made up of as- was done for selected groups. Requests were also sistants, advisers, and alternates; (c) the Secretariat made to the relevant thematic group leaders (for Department; and (d) a random sample of staff, S&T surveys) and unit managers (for the Qual- level GE and above but excluding senior managers ity group) to encourage their staff to respond to (directors and up) in the RVPs, EXT, and WBI. the surveys. From the phone calls, it is estimated The products selected for the product-related that 20-25 percent of the staff were on mission surveys (PARs, CAEs, and S&T studies) were or out of the office. About 10 percent of staff ei- completed during calendar year 2000. The sam- ther had left the Bank or were no longer work- ple technique is consistent with the previous ing in that area. In addition, the surveys included exercise, which examined outcomes of OED prod- several managers and senior managers; who are ex- ucts from calendar year 1999. However, where pos- pected to have a low response rate (primarily for sible, the sample size was larger, in keeping with lack of time). The response rates for these surveys the recommendations of last year's report. In ad- are also similar to those of other OED surveys, in- dition, a survey for Corporate and Process stud- cluding those soliciting opinions on thematic ies was considered. But with only one product, the topics and issues (e.g., for OED studies on Rural Aid Coordination Process Review, the survey was Development, Culture, Forestry, and the like), sug- deemed premature because the report has not yet gesting that the low response rates may not nec- been formally released. Similarly, a survey for the essarily represent biased results. ARDE was considered premature, because it had The borrower surveys piloted this year were just been released. These products, however, are directed at government officials in countries 64 ANNEX G where audits and CAEs had been conducted and asked where respondents were stationed, what po- finalized in the past calendar year. The same sition they had, and what Region they worked products were used as in drawing the sample of in. The results showed that 71 percent of re- Bank staff who had worked on the products. spondents to the random sample of Bank staff The surveys targeted government officials en- were located in headquarters. This figure corre- gaged by OED during the preparation of both sponds to that of 73 percent cited in the recent the sample of CYO0 audits and CYO0 CAEs. Location of Work Report (2001) for all Bank The audit survey was also sent to 56 govern- staff. Of the respondents, 23 percent were ment officials concerned with 17 audits. The re- sector specialists, 21 percent technical specialists, sponse rate was 32 percent, comprising 18 and 9 percent were economists or other respondents from 11 of the 17 countries. The specialists. Thirty-two percent were from the CAE survey had four responses from four different Africa Region, 20 percent from EAP, 14 per- countries for a 44 percent response rate. cent from LCR, and only 2 percent from ECA. To check the accuracy of the survey in re- The remaining 9 percent reported not being flecting a broad range of Bank staff, the surveys in a Region. 65 ANNEX H: THE INFLUENCE OF THREE OED EVALUATIONS: A TRACER STUDY This AROE background study explores the ways implementation of the World Bank 1991 Forestry in which OED evaluations are perceived as use- Policy; the Large Dam Evaluation is an ex-post ful for decisionmaking within the World Bank. cost/benefit analysis gauging the extent to which This investigation does not aim to quantify how Bank-financed dams comply with both the many instances of use Bank staff self-report. previous and current safeguards; and the public Rather, building on Bank task managers', senior expenditure review study reconstructs the impact managers', and Board members' perceptions and of the content, the process, and the techniques opinions, the analysis offers empirical evidence of using the public expenditure review as an on how three specific OED evaluation studies analytical tool. have contributed to change in Bank's practices With the focus on three specific OED stud- and policies. The study reconstructs the cir- ies, there is the advantage of dealing with con- cumstances under which experiences of use occur, crete and actionable variables-that is, real what change they bring about, and what factors evaluations. The characteristics of the three eval- favor or impede the perception of usefulness as- uation studies that are particularly valuable to sociated with OED studies. The three selected Bank staff can be identified to offer guidance on evaluations are (a) The World Bank Forest Strat- suitable modes of evaluation design, methodol- egy: Striking the Right Balance; (b) The World ogy, and dissemination to further enhance the use- Bank's Experience Wth Large Dams: A Prelimi- fulness of OED evaluations. And with the focus nary Review ofImpacts; and (c) a review of the on specific evaluation studies, it is possible to com- impact of public expenditure reviews.' These pare the perceptions and opinions of evaluators OED studies were selected according to the fol- and other Bank staff about the same body of ev- lowing criteria: idence. This research collects their responses to * Evaluations meet the OED standard method- the same questions of interest and learns to what ology- There is already some perceived indi- extent their perceptions converge or diverge, par- cation of utilization. The rationale for choosing ticularly about the usefulness of the work for positive experiences of "use" responds to the decisionmaking. need to understand how and under what cir- cumstances evaluation use occurred, and how Main Research Questions it relates to methodological quality. This study has addressed the following questions: * Evaluations must have been completed within 1. To what extent have the three OED studies a timeframe long enough to systematically contributed to change in Bank practices and gauge the aspects and indications of changes policies? in practices and policies (at least 1 year old, at 2. What specifically has been used of the three most 4-5 years old). OED evaluations: data, findings, generaliza- Furthermore, the three evaluations look at tions, concepts, and theories? different aspects of development programs: that 3. How has this evaluation-based information is, the Forestry Study specifically focuses on the been used: action, thinking, legitimizing? 67 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION 4. What factors favor or impede OED studies' per- * Content analysis of official documents and ceived usefulness? evaluation reports 5. According to Bank staff, who are the primary * Social science literature review, specifically, on users of OED evaluation studies? research and evaluation utilization. Interviews have been conducted with four sam- Methods of Data Collection ples of informants to be able to triangulate differ- Research data have been collected through: ent perspectives and perceptions. The following * Semi-structured interviews of Bank staf, man- table specifies the composition of the four samples agers, and evaluators of informants and indicates the number of interviews * Participant observation conducted in each sample. Interviewee samples Number of interviews Evaluators 6 Board/CODE members (and one assistant) 4 Country/project managers 19 Advisers/specialists 14 Total 43 All interview questions were tailored to the viewees during the interviews. The following table three selected evaluation studies, and the three indicates the number of interviews conducted executive summaries were presented to the inter- for each tracer study and across them. Interview focus Number of interviews conducted Large Dam Evaluation 11 Impact of Public Expenditure Review 9 Forestry Study 13 Across the three Tracer Studies 10 Total 43 68 ANNEX I: CHARIMAN'S STATEMENT, COMMITTEE ON DEVELOPMENT EFFECTIVENESS (CODE) On July 16, 2001, CODE discussed the findings * AROE Recommendations. The Committee and recommendations of the 2000-2001 An- broadly agreed with the four key recom- nual Report on Operations Evaluation mendations for enhancing the Bank's eval- (AROE)(R2001-0130), along with the Draft uation and control framework by: (a) Management Response (CODE2001-0072). The ensuring the evaluation and control frame- Committee commended OED for a compre- work addresses emerging priorities and hensive report on a broad agenda. It welcomed the initiatives; (b) accelerating the process of use of the COSO framework to assess the ade- converting and updating operational poli- quacy of the Bank's development evaluation and cies; (c) bridging the gap between corpo- control processes and the focus on areas of risk and rate sectoral priorities and country vulnerability for the Bank. The Committee programs; and (d) improving the timeliness underlined that it considered the -meeting an and rigor of OED evaluations and of the important one because the report touched Bank's evaluation methods for assessing directly on CODE's mandate of assessing new products and processes, including ad- accountability and increasing the development justment lending. effectiveness of the Bank's activities. * Strengthening Evaluation Methods and 1. The Committee members agreed on the fol- Processes. The Committee welcomed the lowing: focus on strengthening evaluation meth- * COSO Framework. The Committee agreed ods and procedures and underlined the need that the AROE had identified key areas of for further progress in looking at method- risk, and that it highlighted the challenges. ological questions in a number of areas in- Members found the use of the COSO cluding: (i) developing better methods to framework constructive; believed it was a assess the impact of Bank country programs useful tool in assisting OED and the Com- and adjustment lending; (ii) harmoniza- mittee in assessing the adequacy of the tion of evaluation methodologies across de- Bank's development risk management sys- velopment partners; and (iii) reviewing tem; and supported its continued use in OED's self-evaluation processes and how to future reports. ensure their objectivity. The Committee * Revised Management Response. The Com- also underscored the importance of the mittee felt that the draft Management Re- country assistance evaluations to Board dis- sponse reflected considerable agreement cussions of CASs and called for even shorter with the OED recommendations but re- feedback loops. quested that a revised response address them * Disclosure. The Committee supported dis- more explicitly. The Committee agreed with closure of the AROE. Process evaluations Management that the focus of the report on under the current policy can only be risk and vulnerability needed to be made ex- disclosed with Board approval. If the draft plicit. It requested that the DGO revise the disclosure policy currently under review is transmittal memo to make this clear. adopted by Executive Directors, process 69 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION reviews, including the AROE, would be lines for implementation; and (iii) reduce disclosed in the future. costs to operational staff and borrowers of developing the strategies. The Committee Issues raised during the discussion stressed the importance of implementation included: plans being an integral part of SSPs, includ- a. Comments from Management. Management ing their costing and underscored the need for underscored that it took the OED findings and a more timely, efficient and transparent pub- recommendations seriously and that it was in lic consultative process for developing both broad agreement with most of the recom- SSPs and the operational policies. The Com- mendations. Management raised a number of mittee welcomed Management's intent to concerns: (i) the report should make explicit provide Executive Directors with periodic that the focus of the AROE this year had progress reports on the status of updating the shifted from the previous "balanced scorecard Bank's operational policies, including a time approach" to risk and vulnerability; (ii) the line for completion of key policies and it balance between recasting policies and imple- looked forward to the report on managing op- mentation; (iii) the need for continuing pri- erational risk and the SSP Stocktaking paper oritization of and fewer recommendations in in early FY02. OED evaluations; and (iv) OED should extend c. Evaluation Coordination. The Committee noted the coverage of its targeted review of evaluation the increase in evaluation units within the methodology beyond adjustment lending and Bank and appreciated that this was a sign of coverage of new instruments to include in- Management's commitment to reducing cor- vestment lending. porate risk, but cautioned against duplication b. Modernization and Updating of the Bank's and evaluation overload and encouraged more Policy Framework. The Committee welcomed coordination of evaluations across the Bank and the progress in updating the Bank's policy with partners. The Committee suggested that framework and underlined a number of issues staff be trained about the role of evaluation to regarding development of sector and opera- make clear its benefits. A progress report on har- tional policies. These included the need to: monization of policies and procedures will (i) achieve balance between timeliness, clar- come to the Board in September 2001. ity and consultation in revising operational The AROE and its revised transmittal mem- policies. It cautioned that the quality of the orandum (R2001-0130/1) and the revised Man- process and of the policies should not be sac- agement Response (R2201-0132/2) will be rificed for shorter preparation times; (ii) discussed by the Board on July 31, 2001. shorten, as much as possible, the time lag be- tween the Bank's undertaking new operations and developing operational tools and guide- Pieter Stek, Chairman 70 ENDNOTES Chapter 1 ex-post independent evaluation. The OED budget 1. Evaluation findings for completed projects increased by 10 percent between FY95 (US$15.7 are presented in OED's Annual Review ofDevel- million) and FY00 (US$17.3 million). Whereas, opment Effectiveness (ARDE) even without a full accounting for all evaluation 2. 2000 Annual Review of Development Effec- and control activities, the FY00 budgets for four tiveness: From Strategy to Results (Washington, oversight units (QAG, LAD, Oversight Com- D.C.: World Bank, 2001). mittee on Fraud and Corruption, and QACU) to- taled US$18.0 million. None of these units except Chapter 2 LAD even existed in 1995 (the FY95 budget for 1. The Drive to Partnership: Aid Coordination LAD was US$ 4.4 million-about the same as in and the World Bank (Washington, D.C.: World FY00). It should be noted that there may have Bank, 2001). been offsetting savings as a result of the discon- 2. ESW expenditures went up from $56 to tinuation of some past self-evaluation activities. $72 million between FY97 and FY00, just under 7. 2000 Annual Review ofDevelopment Effec- FY95 but much below FY93 expenditures. ESWs tiveness: From Strategy to Results (Washington, share in total operating expenditures has remained D.C.: World Bank, 2001). constant at 7 percent since FY97. Lending ex- penditures declined from $159 in FY95 to $118 Chapter 3 million in FY00. 1. The sample included 11 adjustment oper- 3. The establishment of the Management Com- ations out of a total sample of 80 in QEA3 (or mittee was announced on the Bank Kiosk on about 14 percent). May 11, 2001. 2. Management notes: Based on experiences 4. Management notes: Fiduciary and safeguard from earlier rounds of Regional risk assessments, oversight of projects has been highlighted else- a risk assessment working group is currently where in the report. In addition, all other sector developing a common methodology that will boards review their pipeline of operations and be used by all Regions in the next round of provide special support for those that are compli- assessments. cated or represent significant risk. One result has 3. QSA4 indicates that in many Bank units, PSR been the increase in ex ante quality enhancement reporting is seen as purely a formality and that there reviews in recent years, led either by sector boards is little incentive to staff to spend time and effort or with strong sector board input. to fill out the reports with adequate care. 5. Management notes: Management believes 4. Management notes: Recent portfolio reviews that the country level is the appropriate one for show that over the past five years, considerable operational integration across sectors. By making progress has been made in improving the candid- it easier for staff to work across country units and ness of project ratings. Further, field supervision in Regions, the matrix has enhanced cross-country key areas is carried out by economic and socially learning. With regard to issues raised at the sustainable development specialists (especially for sector-wide level of work on water resources, the high-risk operations) and fiduciary specialists. Bank now has a water resources management 5. DGF programs with more than US$300,000 group and Regional water teams. The updated in annual financing undergo an independent eval- water strategy will refine these coordination issues uation. The FY02 budget draws on independent in the water sector and links to other sectors. evaluations focusing on output and outcomes of 6. As noted below, the increasing portfolio six programs. quality can be attributed to enhanced quality as- surance and risk management efforts by the Bank's Chapter 4 management. This growth in the quality and 1. The six were poverty reduction, adjustment volume of self-evaluation and strengthening of lending, environmental assessments, indigenous management controls complements existing peoples, involuntary resettlement, and cultural 71 2000-2001 ANNUAL REPORT ON OPERATIONS EVALUATION heritage. In 1998, the conversion of another 11 was 8. Management notes: Management agrees postponed because of evolving Bank work in these with the assessment in this report (Chapter 7, sec- areas. Of these, the policy on investment lending tion entitled "Harmonizing Evaluation Method- has been updated; others are in various stages of ology") that additional work is needed to develop preparation. consistent guidelines for the relevance and insti- 2. Three Operational Memoranda have been tutional development aspects of performance issued to provide interim instructions to staff: measurement. Progress in this area would con- Guidelines for Programmatic Adjustment Lend- tribute substantially to reducing the gap noted ing, Clarification of Current Bank Policy on above with regard to institutional development. Adjustment Lending, and Interim Guidelines on Poverty Reduction Support Credits. Chapter 5 3. Management notes: "Current" with regard 1. See, for example, "Poverty Reduction and the to OPs is used, inter alia, to indicate: that they are World Bank: Progress in Fiscal 1999" (Washing- in the OP/BP format and not the OD or OMS ton, D.C.: World Bank). format-implying that their mandatory aspects 2. The proposal was to cover five countries and have been clearly separated from the good prac- two sectors in year one of the program, and an- tice advice; that they reflect recent lessons of ex- other three countries and two sectors in year perience (including those identified by OED and two of the program. accepted by management) and cumulative inter- 3. M&E deficiency contributes to many of the pretations by the Board; and that their description issues noted in other parts of this report (e.g., risk of procedures reflects the current organizational assessment, performance measurement, realism in structure of the Bank. However, in a broader sense, reporting). M&E can also contribute to reducing any policy statement in the Operational Manual, supervision and completion reporting costs. when read together with its updates (i.e., Opera- 4. The Panel's mandate is limited to cases of al- tional Memoranda issued subsequently), reflects leged noncompliance of policies and procedures the current Bank Policy. with respect to the design, appraisal, and/or im- 4. Five of the eight OED sector and thematic plementation of projects. No procurement action evaluations completed in FY00 have raised issues is subject to inspection by the Panel. related to operational policies (aid coordination, information infrastructure, gender, forestry, Chapter 7 poverty). 1. The eight criteria are: Relevance, Achieve- 5. Management notes: Corporate risk projects ment of Objectives (efficacy), Efficiency, Sus- are identified by the Regions with assistance from tainability, Institutional Development Impact, QACU and the Legal Department. Other types Aggregate Project Performance Indicator, Bor- ofhigh-risk projects are identified by sector boards. rower Performance, and MDB Performance. 6. Management notes: The concern noted 2. Two pilot borrower surveys were also sent to above was expressed on the basis of earlier re- selected officials associated with recent PARs and views on ESW quality. More recent findings con- CAEs. The results are generally positive, and the firm that management has continued to make findings are largely similar to those of the staffsur- commendable progress in improving the quality veys. The findings are discussed in Annex G. of ESW with 86 percent of the tasks rated as 3. One explanation of the lower ratings for satisfactory. CAEs could be the lack of agreement on a stan- 7. Management notes: The criteria used by dard CAE methodology. OED staff for evaluating the section in ICRs on 4. Objectivity and relevance are rated lower by transition arrangements are not transparent to respondents from the CAE and PAR groups, but staff. It would be useful to make these criteria substantially higher by the ED staff, quality, ran- readily accessible to staff. dom, and S&T groups. 72 ENDNOTES 5. Of the FY00 CASs (17), about half (8) were Annex H "CAE-CASs," that is, they had had a CAE in the 1. Review of the World Bank's 1991 Forest past. The rest are "non-CAE CASs." Policy and Its Implementation (Washington, 6. Transcripts and "yellow sheets' for a total of D.C.: OED/World Bank, 2000). The WorldBanki 32 selected Board discussions on topics related to Experience with Large Dams: A Preliminary Review the Bank's operational work and development of Impacts, Report No. 15815, (Washington, effectiveness were reviewed. D.C.: OED, 1996). The review of public expen- diture reviews, a process evaluation, was prepared Annex A for internal use only and not disclosed according 1. The Treadway Commission, also known as to the Bank's policy at the time. the National Commission on Fraudulent Finan- cial Reporting, presented the framework in their 1992 report, titled "Internal Controls-Integrated Framework." 73