WORLD BANK OPERATIONS EVALUATION DEPARTMENT ui 19062 February 1999 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS øé I~ OPERATIONS EVALUATION DEPARTMENT ENHANCING DEVELOPMENT EFFECTIVENESS THROUGH EXCELLENCE AND INDEPENDENCE IN EVALUATION The Operations Evaluation Department (OED) is an independent unit within the World Bank; it reports directly to the Bank's Board of Executive Directors. OED assesses what works, and what does not; how a borrower plans to run and maintain a project; and the lasting contribution of the Bank to a country's overall development. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank's work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. WORLD BANK OPERATIONS EVALUATION DEPARTMENT 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS Robert Buckley 1999 The World Bank www.worldbank.org/html/oed/ardefm98.htm Washington, D.C. Copyright © 1999 The International Bank for Reconstruction and Development/THE WORLD BANK 1818 H Street, N.W. Washington, D.C. 20433, U.S.A. All rights reserved Manufactured in the United States of America First printing February 1999 The opinions expressed in this report do not necessarily represent the views of the World Bank or its member governments. The World Bank does not guarantee the accuracy of the data included in this publication and accepts no responsibility whatsoever for any consequence of their use. The boundaries, colors, denominations and other information shown on any map in this volume do not imply on the part of the World Bank Group any judgment on the legal status of any territory or the endorsement or acceptance of such boundaries. The material in this publication is copyrighted. The World Bank encourages dissemination of its work and will normally grant permission promptly. Permission to photocopy items for internal or personal use, for the internal or personal use of specific clients, or for educational classroom use is granted by the World Bank, provided that the appropriate fee is paid directly to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, U.S.A., telephone 978-750-8400, fax 978-750-4470. Please contact the Copyright Clearance Center before photocopying items. For permission to reprint individual articles or chapters, please fax your request with complete information to the Republication Department, Copyright Clearance Center, fax 978-750-4470. All other queries on rights and licenses should be addressed to the Office of the Publisher, World Bank, at the address above or faxed to 202-522-2422. ISSN 1019-4363 ISBN 0-8213-4474-9 Contents v Acknowledgments vii Foreword, Prefacio, Pr6face ix Executive Summary, Resumen, Rsume Analytique xvii Abbreviations and Acronyms 1 1. Development Effectiveness in a Volatile World 1 Institutional Weaknesses 4 Broadening the Agenda to Maximize Development Effectiveness 4 The New Primacy of the External Environment 7 2. Trends in Project Performance 7 Outcomes More than 75 Percent Satisfactory 11 Gains in Development Effectiveness 14 Institutional Development Impact-Improving but Still Weak 14 Sustainability-Low and Weakening? 15 Bank and Borrower Performance Improving-But Greater Gains from Borrowers 16 Outstanding Projects-7 Percent of the Total 17 Sector Analysis 19 3. Deveiopment Effectiveness at the Country Level 19 The Country as the Unit of Account 20 Institutions, Aid, and Growth 23 Adjusting to the External Environment 24 Outward Orientation and Growth 24 Outward Orientation and Poverty Alleviation 27 4. Thematic Evaluations and Institutional Development 27 Financial Sector 28 Capitalization and Bad Loan Exposure 28 Government Interventions 28 Accounting and Prudential Standards 28 Enforcement Capabilities 29 Governance Issues 29 Governance Institutions 33 5. mplications for tie Bank and for Evaluation 33 Global Risks 35 Prospects for Poverty Reduction 36 Implications for the Bank 36 Poverty Alleviation 36 Country Strategies 36 Institutional Development 39 Annexes 39 Annex 1: A New Approach to Evaluating Bank Projects-The Development Effectiveness Index 42 Annex 2: Glossary of Selected Terms 44 Annex 3: Data on Outcome, Sustainability, and Institutional Development Impact 48 Annex 4: Report from Committee on Development Effectiveness (CODE) iii 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 51 Endnotes 53 Bibliography Boxes 8 2.1 Institutional Development and Sustainability Help Capture Project Performance 11 2.2 Improvement in Africa 17 2.3 Two Outstanding Projects 21 3.1 Institutions and Developing Economies 22 3.2 The Long-Run Expected Benefits of Adjustment 24 3.3 Measuring Borrower Ownership and Its Relation to Adjustment Outcomes 25 3.4 Greater Poverty and Income Inequality in Transition Economies 31 4.1 Public Sector Performance Review: Integrating Public Sector Performance with a Results-Based Management System 34 5.1 What Do We Know about the Extent of Poverty? 37 5.2 Poverty Alleviation and Income Growth Figures 4 1.1 Institutions and Project Performance in Low-Income Countries 8 2.1 Demandingness, Complexity, Riskiness, and Outcome 9 2.2 Development Effectiveness Index and Outcome: Measuring Project Performance 10 2.3 Satisfactory Outcome by Region 12 2.4 Africa Region-Relative Performance Trend, Percentage Operations with Satisfactory Outcome 12 2.5 Satisfactory Outcome by Sector and Exit Fiscal Year Group 13 2.6 Institutional Development Impact 14 2.7 Sustainability 14 2.8 Borrower Performance 15 2.9 Bank Performance 16 2.10 Relative Risk and Reward, by Sector, Exit FY97-98 29 4.1 Performance of Civil Service Reform Interventions Tables 2 1.1 Traditional Crisis Indicators 3 1.2 The Washington Consensus-Not in East Asia 15 2.1 Recently Evaluated Projects: Outstanding Performers and Poor Performers 20 3.1 Performance of Bank-Supported Country Strategies and Projects 21 3.2 Trends in Socioeconomic Indicators and Income, 1970-96 28 4.1 Bank System Risk Exposure in East Asia iv ACKNOWLEDGMENTS his review was prepared by Operations Evalu- Silvana Valle, Geri Wise, Brigitte Wittel, and Barbara ation Department staff. The task manager was Yale provided administrative assistance. Robert Buckley. The contributors were Will- This study was produced in the Partnerships and iam G. Battaile, Jr., and Ruchira Banerjee-Corcoran Knowledge Group (OEDPK) by the Dissemination and (OEDCM); and Deepa Chakrapani, Federico Mini, Outreach Unit. The unit is directed by Elizabeth and Dinara Seytjaparova (OEDCR). Contributions Campbell-Pag6, Task Manager, and includes Caroline were also made by James Alm (consultant); Keith McEuen and Leo Demesmaker (editors), Kathy Strauss Bezanson (IDS, Sussex); Jose Maria Dagnino Pastore, and Lunn Lestina (desktop design and layout), and Louis Goreux, and Jane-Francis Kelly (consultants); Juicy Qureishi-Huq (administrative assistance). Anwar Shah and Navin Girishankar (OEDCR); and Sohail Malik, Kenneth Watson, and John Eriksson (consultants). Ruben Lamdany (Manager, OEDCR) Director-General, Operations Evaluation: Robert Picciotto and Roger Slade (Manager, OEDST) and their staff Director , Operations Evaluation : Robet cit r provided comments and were helpful in many ways. Director, Operations Evaluation Department: Elizabeth McAllister Wendy Jarvie (Manager, OEDCM) supervised and Manager, Corporate Evaluations and Methods: Wendy jarvie helped to structure the Review. Norma Namisato, Task Manager: Robert Buckley v Forew ord FOREWORD PREFACIO P REFACE The ongoing financial crisis La actual crisis financiera ha La crise financière en cours has raised questions about the planteado interrogantes acerca de los soulève des questions quant aux underpinnings of development fundamentos de la asistencia para el principes directeurs de l'aide au assistance and the role of desarrollo y el papel que cumplen las développement et au rôle des international financial institutions. instituciones financieras institutions financières internationales. A new development assistance internacionales. Comienza a surgir un Un nouveau cadre d'aide au framework, grounded in nuevo marco para la asistencia en pro développement axé sur des partnership, is emerging. That is del desarrollo, basado en la formación partenariats fait son apparition. C'est the backdrop for this year's Annual de asociaciones de colaboración. Este dans ce contexte que le Département Review of Development es el telón de fondo del presente de l'évaluation des opérations (OED) Effectiveness by the World Bank's Examen anual sobre la eficacia en de la Banque mondiale a réalisé cette Operations Evaluation Department términos de desarrollo, preparado por année son Examen annuel de (OED). el Departamento de Evaluación de l'efficacité du développement. As in past years, the Review Operaciones (DEO) del Banco Comme les années passées, cet tracks the Bank's operational Mundial. Examen fait le bilan des opérations de performance based on the findings Al igual que en ados anteriores, la Banque d'après les conclusions des of recent evaluations. The trends en este informe se pasa revista a los récentes évaluations. Les tendances are highly encouraging, but when resultados de las operaciones del sont particulièrement encourageantes, countries that have performed so Banco sobre la base de las mais lorsque des pays qui ont obtenu well for so long suddenly stumble conclusiones de las evaluaciones d'excellents résultats pendant très as dramatically as they did in the realizadas recientemente. Las longtemps ont soudain une défaillance past year, the meaning of project- tendencias observadas son muy aussi marquée que celle qu'ils ont level trends deserves careful alentadoras, pero cuando los paises connue l'an passé, la signification des consideration. que han registrado resultados muy tendances au niveau des projets mérite Accordingly, this Review favorables durante mucho tiempo la plus grande attention. draws on the work of scholars enfrentan de pronto graves En conséquence, le présent convened by the Institute of problemas, como ha sido el caso en el Examen s'appuie sur les travaux Development Studies of Sussex último año, es necesario analizar d'universitaires réunis par l'« Institute University to assess the cuidadosamente el significado de dicha of Development Studies » de implications of the crisis. It also evolución al nivel de los proyectos. l'université du Sussex pour évaluer les relies on a relatively new OED En consecuencia, el presente incidences de la crise. Il fait également instrument-country assistance examen se basa en la labor de appel à un instrument de l'OED evaluations-to place the lessons expertos convocados pot el Institute relativement nouveau-les évaluations from the Bank's project experience of Development Studies de la de l'aide aux pays-pour replacer in a broader context. Universidad de Sussex con el fin de dans un contexte plus large les The Review complements the evaluar las repercusiones de la crisis. enseignements tirés des projets de la Annual Report on Portfolio También se apoya en un instrumento Banque. Performance, which documents the relativamente nuevo del DEO-las Cet Examen complète le Rapport Quality Assurance Group's findings evaluaciones de la asistencia a los annuel sur la performance du about active operations, and the paises-con el objeto de poner las portefeuille, qui rend compte des vii 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS Annual Report on Operations ensefñanzas derivadas de los conclusions du Groupe pour le Evaluation, which presents proyectos del Banco en un contrôle de la qualité sur les OED's assessment of the status contexto más amplio. opérations en cours et le and prospects of internal El examen complementa el Rapport annuel sur l'évaluation evaluation processes. Informe anual sobre el des opérations, dans lequel desempeiio de la cartera, en el l'OED fait le point de la cual se exponen las situation en ce qui concerne les conclusiones del Grupo de garantia de processus d'évaluation interne et les calidad acerca de las operaciones perspectives qu'ils offrent. activas, y el Informe anual sobre evaluaciôn de operaciones, en el que se presenta el análisis que hace el DEO de la situación y las perspectivas de los procesos de evaluación interna. Robert Picciotto Director-General, Operations Evaluation viii Executive Summary EXECUTIVE RESUMEN RESUME SUMMARY 1 ANALYTIOUE This Review of development El presente Examen sobre la Cet Examen de l'efficacité du effectiveness comes at a time of eficacia en términos de desarrollo se da a développement coïncide avec une crisis. In East Asia, about conocer en un momento de crisis. En période de crise. En Asie de l'Est, une 20 million people have fallen back Asia oriental, aproximadamente 20 vingtaine de millions de personnes into poverty in the last year. millones de personas han vuelto a caer sont retombées dans la pauvreté l'an Russia has been beset by political en la pobreza en el último afño. La dernier. La Russie est assaillie de and economic upheaval. Japan is Federación de Rusia se ha visto afectada graves problèmes politiques et in recession, with profound por conmociones politicas y économiques et le Japon connaît une récession qui a de profondes implica- tions pour l'économie mondiale. Des catastrophes naturelles telles que des inondations au Bangladesh, en Chine et en Amérique centrale ne font qu'aggraver les problèmes économiques, et les chances de réaliser les objectifs de réduction de la pauvreté fixés par l'OCDE sont maintenant réduites. Cette crise est riche d'enseignements aussi bien pour les praticiens que pour les analystes du développement. Les pays en développement se trouvent maintenant implications for the world económicas. El Japôn atraviesa por una dans une situation beaucoup moins economy. Economic problems have recesión que ha tenido profundas favorable, comme en témoignent les been compounded by natural repercusiones en la economia mundial. effets de flux privés incontrôlés et de disasters, such as floods in A las dificultades económicas se han l'interdépendance mondiale, et Bangladesh, in China, and in agregado los desastres naturales, como l'influence croissante des facteurs Central America. The prospects for las inundaciones en Bangladesh, China y exogènes sur les impacts du achieving the OECD poverty Centroamérica. Las posibilidades de développement. reduction targets have dimmed. alcanzar las metas de reducción de la The crisis is rich in lessons for pobreza fijadas pot la OCDE se han La stabilité macroéconomique both development practitioners and desvanecido. ne suffit pas evaluators. Developing countries La crisis deja numerosas Les facteurs macroéconomiques ne now confront a severe deterioration ensefñanzas para los que se dedican a suffisent pas à assurer une croissance in the enabling environment, la tarea del desarrollo y para quienes équitable. À la différence de la crise de highlighting the effects of la evalúan. Los paises en desarrollo la dette des années 80, la crise unregulated private flows and global se enfrentan ahora al grave deterioro financière actuelle a commencé dans interdependence and the growing de las condiciones que hacen posible des pays dont la position budgétaire influence of exogenous factors in el progreso económico, destacándose était relativement forte et qui étaient determining development impacts. en particular los efectos de la caractérisés par de saines politiques ausencia de regulación de los flujos monétaires et des régimes du com- privados y de la interdependencia a merce ouverts sur l'extérieur. Lorsque ix 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS A stable macro economy nivel mundial, asi como la la crise s'est produite, les bud- îs not enough - influencia creciente de los gets publics de la plupart des Sound macroeconomic factores exógenos en la pays touchés étaient équilibrés conditions are not enough to determinación del impacto en ou excédentaires, l'inflation sustain equitable growth. el desarrollo. maîtrisée, les taux d'intérêt en Unlike the debt crisis of the baisse et le taux de chômage 1980s, the present financial La estabilidad macroeconómica déclaré faible. crisis started in countries with no es suficiente relatively strong fiscal situations, Las condiciones macroeconômicas no Importance des institutions sound monetary policies, and bastan para mantener un crecimiento La crise a montré combien les outward-oriented trade regimes. con equidad. A diferencia de lo ocurrido faiblesses institutionnelles peuvent être When the crisis hit, government durante la crisis de la deuda de los años coûteuses-en particulier dans les budgets in most crisis-affected ochenta, la actual crisis financiera se secteurs financier et social. En fait, il countries were balanced or moving originô en paises con una situación est maintenant évident que de solides into surplus, inflation was contained, financiera relativamente sôlida, una institutions sont indispensables à la interest rates were going down, and politica monetaria acertada y sistemas stabilité économique et sociale. Dans le recorded unemployment was low. de comercio orientados al exterior. Al cas contraire, les économies en producirse la crisis, los presupuestos développement et en transition sont Institutions matter púiblicos de la mayoría de los paises d'autant plus vulnérables à une perte The crisis showed just how costly afectados estaban equilibrados o incluso de confiance de la part des weaknesses in institutions can be- mostraban superàvit, la inflación estaba investisseurs privés. Le développement especially in the financial and social controlada, las tasas de interés estaban institutionnel est important, pas sectors. Indeed, it is now clear that bajando y el desempleo registrado era seulement pour éviter des crises strong institutions are essential for reducido. economic and social stability. Poor • Pour les projets financés par la institutions increase the vulnerability Las instituciones revisten importancia Banque, la qualité des institutions of developing and transition La crisis mostrô claramente el costo que peut influer considérablement sur economies to shifts in private pueden significar las deficiencias de las l'efficacité du développement. investor confidence. The importance instituciones, sobre todo en los sectores Leurs effets sont particulièrement of institutional development goes far financiero y social. En efecto, ahora no prononcés dans les pays à faible beyond avoiding crises: cabe duda de bo importante que es revenu. contar con instituciones sólidas para • D'une façon générale, lorsque les • For Bank-supported projects, lograr la estabilidad económica y social. institutions laissent à désirer, les the quality of institutions can Cuando las instituciones son deficientes projets ont des taux de rentabilité have important effects on las economias en desarrollo y en plus faibles et comportent development effectiveness. transición se vuelven màs vulnerables davantage de risques. These effects are particularly ante los cambios en la confianza de los • Un pays doté de bonnes institu- pronounced in low-income inversionistas privados. El desarrollo tions est mieux à même de countries. institucional no sólo es esencial para s'adapter et a deux fois plus de • Where institutions are evitar las crisis: chances de pouvoir maintenir le systematically weak, projects cap s'il est engagé dans une yield lower returns and entail • Para los proyectos respaldados por politique d'ajustement. higher risk. el Banco, la calidad de las • Better institutions strengthen instituciones puede tener efectos de Une analyse de 41 pays à faible a country's ability to adjust. consideraciôn en la eficacia en revenu montre que la qualité des insti- They can more than double términos de desarrollo. Estas tutions n'a été jugée satisfaisante que the likelihood that a country repercusiones son especialmente no- dans un seul d'entre eux. Quarante undergoing adjustment can tables en los paises de ingreso bajo. pour cent seulement des projets stay the course. • Cuando las instituciones son soutenus par la Banque ont un impact x Executive Summary An analysis of 41 low- sistemáticamente deficientes, los marqué sur le développement income countries shows that - proyectos arrojan una menor institutionnel ; les réformes de la only one was rated rentabilidad y entraôan fonction publique entreprises à satisfactory on institutional mayores riesgos. titre de composantes de prêts à quality. Only 40 percent of • Cuando un pais cuenta con l'ajustement structurel ont des Bank-supported projects have instituciones más eficientes, tiene résultats mitigés ; les projets substantial impact on màs capacidad de adaptación. relatifs à la gestion du secteur institutional development; civil Gracias a éstas, al emprender ajustes, public, tout en s'améliorant, donnent service reforms undertaken as las probabilidades del pais de depuis toujours des résultats components of structural mantener el rumbo son doblemente inférieurs à la moyenne des projets de adjustment loans have mixed mayores. la Banque. Une évaluation de l'OED a outcomes; and public sector montré que les projets soutenus par la management projects, while Un análisis de 41 paises de ingreso Banque dans le secteur financier ne improving, have historically bajo muestra que tan sôlo uno de ellos donnaient des résultats satisfaisants et performed below the Bank obtuvo una calificación satisfactoria en soutenus que dans 50 % des pays. Le average. An OED evaluation lo que respecta a la calidad de las développement institutionnel est lent showed that Bank-supported instituciones. Apenas el 40% de los et difficile à assurer dans un financial sector projects had proyectos respaldados por el Banco environnement institutionnel fragile et satisfactory and sustained tienen un efecto considerable en el il nécessite une excellente coordination outcomes in just 50 percent of desarrollo institucional; los resultados de l'aide et la mise en place des countries. Institutional de las reformas de la administración capacités voulues pour absorber l'aide development is slow and difficult pública que se han emprendido como et réduire les risques de surcharge. to achieve in a fragile institutional componentes de los préstamos para environment and requires strong ajuste estructural han sido variados; los Réduction de la pauvreté et filets aid coordination and the proyectos de gestiôn del sector público, de protection sociale development of capacity to absorb si bien están mejorando, generalmente Un autre enseignement à tirer est le aid and reduce the risks of han registrado resultados por debajo suivant: le développement social overload. del promedio para el Banco. Una devrait jouer un rôle central, au niveau evaluaciôn realizada por el DEO reveló aussi bien de l'évaluation de l'efficacité Poverty reduction and social que los proyectos del sector financiero du développement que du financement safety nets respaldados por el Banco arrojaban des programmes d'aide aux pays. A corollary lesson is that social resultados satisfactorios y duraderos On estime qu'en Indonésie et en development should come center tan sôlo en la mitad de los países Thaïlande, les suppressions d'emplois stage-both in assessing estudiados. El desarrollo institucional sont de l'ordre de 10 à 15 %. Du fait development effectiveness and in avanza lentamente y es dificil de lograr des dévaluations et de l'élimination des financing country assistance cuando las condiciones de las subventions, les nouveaux chômeurs programs. instituciones son precarias; se requiere subiront les effets d'une énorme perte Serious reductions in una buena coordinaciôn de la ayuda y de revenu et d'une forte hausse des employment of 10-15 percent are es preciso crear la capacidad para prix. L'environnement mondial étant estimated for Indonesia and absorberla y para reducir los riesgos de de plus en plus intégré, les pays Thailand. With devaluations and sobrecarga. resteront vulnérables aux chocs. Il the removal of subsidies, the newly faut accorder beaucoup plus unemployed will suffer from Reducción de la pobreza y redes d'attention aux filets de sécurité drastic losses in income and sharp de protección social destinés à empêcher les pauvres et rises in prices. The increasingly Un corolario es que el desarrollo social ceux qui sont à la limite de la pauvreté integrated global environment deberia pasar al primer plano a la hora de faire les frais de ces chocs de façon means that country susceptibility to de evaluar la eficacia en términos de disproportionnée. shocks will not disappear. Much desarrollo y de financiar los programas Les pays en crise ne sont pas les greater attention must be given to de asistencia a los paises. seuls à connaître une aggravation des x i 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS safety nets in helping to Se prevé una fuerte inégalités. Les données relatives insulate the poor and the - reducción del empleo, de entre à 74 pays révèlent que ce near-poor from 10% y 15%, en Indonesia y phénomène a été très marqué disproportionately bearing Tailandia. Con las devaluaciones dans les années 90. Les the costs of shocks. y la eliminación de subsidios, los inégalités se sont en effet The crisis countries are trabajadores que pierdan su aggravées dans 49 pays et n'ont not the only ones empleo sufrirán las consecuencias diminué que dans 10. Cela experiencing increasing de las graves pérdidas de ingresos y del montre combien il est nécessaire de inequality. Data for 74 countries fuerte aumento de los precios. Con un mettre l'accent sur la lutte contre show an overwhelming increase entorno mundial cada vez más l'exclusion, le développement social et in inequality within countries in integrado se mantendrá la les filets de sécurité lorsque l'on the 1990s-49 countries susceptibilidad de los paises a las crisis. conçoit et met en oeuvre des stratégies experienced increasing Se debe prestar mucha mis atención a de réforme et des programmes de inequality, while only 10 had las redes de protección social a fin de développement. decreasing inequality. This evitar que la población pobre y la que confirms the need to emphasize está cercana a la pobreza tenga que Une aide aux pays fondée sur inclusion, social development, soportar excesivamente el costo que des partenariats and safety nets in the design and entrafñan las conmociones. Les facteurs financiers, institutionnels implementation of reform Los países en crisis no son los et sociaux doivent être considérés strategies and development únicos afectados pot el aumento de la simultanément. Pour que la croissance programs. desigualdad. Datos correspondientes a débouche sur un développement du- 74 países revelan un incremento rable, les stratégies d'aide aux pays A country focus based on extraordinario de la desigualdad en la doivent accorder un poids suffisant partnership década de 1990: ésta ha aumentado en aux facteurs structurels, au Financial, institutional, and 49 paises, y ha disminuido tan sólo en renforcement des capacités et à la jus- social factors must be considered diez. Lo anterior confirma la necesidad tice sociale, et identifier les lacunes together. For growth to result in de insistir en la inclusión, el desarrollo potentielles d'ordre structurel qui sustainable development requires social y las redes de protección social a risquent de réduire à néant les acquis country assistance strategies that la hora de disefñar y aplicar las du développement. give adequate weight to estrategias de reforma y los La Banque doit opérer dans le cadre structural factors, capacity programas de desarrollo. de projets efficaces. Cela veut dire que les building, and social equity, and opérations doivent être liées au contexte that identify potential holes in Centrar la atención en los paises social, civil et économique dans son en- the boat-where structural faults recurriendo a las asociaciones semble. Pour être plus efficace, la might cause development gains de colaboración Banque doit opérer en partenariat avec to unravel. Los aspectos financieros, institucionales les emprunteurs, les bailleurs de fonds et A credible Bank role begins y sociales deben considerarse en les autres parties prenantes afin de with effective projects. This conjunto. Para que el crecimiento maximiser l'impact de son action sur le implies operations linked to the conduzca al desarrollo sostenible, en las développement à l'échelon des pays. broader social, civil, and estrategias de asistencia a los países se Pour ce faire, la Banque doit examiner economic environment. To scale- debe dar la debida importancia a los les importants effets secondaires qu'ont up successes, the Bank must work aspectos estructurales, el fortalecimiento des activités liées entre elles sur les in partnership with borrowers, de la capacidad y la equidad social, y se politiques et institutions des pays. Il faut donors, and other stakeholders to deben determinar las posibles également que tous les participants focus on maximizing deficiencias estructurales que podrian reconnaissent leurs points faibles et development impact at the hacer desaparecer los avances en materia points forts respectifs et soient en même country level. To do so the Bank de desarrollo. temps désireux de définir et de partager must consider the important side El Banco debe participar en les responsabilités en jeu. Une stratégie effects that interrelated activities proyectos eficaces; es decir, las fondée sur un partenariat est une bonne xii Executive Sunmary can have on country policies operaciones deben estar chose du point de vue non and institutions. It also - relacionadas con las condiciones seulement du développement, requires recognition by al sociales, civiles y econômicas gen- mais aussi de la situation participants of their relative erales. Para ampliar la financière de l'institution. strengths and weaknesses, proporción de proyectos Il reste beaucoup à faire together with a willingness to satisfactorios, el Banco debe pour améliorer la qualité des define and share account- trabajar en asociación con los stratégies d'aide aux pays. L'OED ability. A partnership-based prestatarios, los donantes y otras partes estime que celles-ci n'ont été strategy is good policy from a interesadas, concentrándose en satisfaisantes que dans 68 % des cas. development perspective, and good maximizar el impacto en el desarrollo al Une analyse confirme que les résultats corporate finance. nivel de los países. Para ello, debe tener des projets dépendent largement de la Much remains to be done to en cuenta los importantes efectos stratégie dont un pays fait l'objet, et enhance the quality of country secundarios que pueden producir las lorsque celle-ci est bien conçue, les assistance strategies. Where actividades interrelacionadas en las projets exécutés dans le pays en ques- country assistance evaluations políticas e instituciones nacionales. tion ne donnent jamais de mauvais have been undertaken, OED También es necesario que todos los résultats. estimates that assistance strategies participantes reconozcan sus fortalezas have been satisfactory only 68 y debilidades relativas, y expresen su Nette amélioration des résultats percent of the time. Analysis disposición a definir y asumir su cuota des projets confirms that project outcomes are de responsabilidad. Una estrategia Le pourcentage des projets soutenus par highly dependent on the country basada en una relación de colaboración la Banque et donnant des résultats strategy. For example, no country es acertada desde el punto de vista del satisfaisants une fois les décaissements that had a satisfactory country desarrollo y de las finanzas de prêts effectués est passé d'une strategy demonstrated weak project institucionales. moyenne de 65-70 % durant la période performance. Es mucho bo que queda por hacer de 1990 à 1996 à au moins 75 % para elevar la calidad de las estrategias (chiffre prévisionnel) en 1997-98, 7 % Project performance has improved de asistencia a los paises. El DEO estima des projets ayant même donné substantially que éstas han sido satisfactorias d'excellents résultats. Ce progrès The percentage of Bank-supported solamente en el 68% de los casos. Los remarquable témoigne de la volonté projects with a satisfactory analistas confirman que los resultados qu'ont la Banque et les emprunteurs de outcome at the end of loan de los proyectos dependen en gran rendre plus efficace l'impact de leur ac- disbursement increased from an medida de la estrategia aplicada. Pot tion sur le développement. average of 65-70 percent in the ejemplo, en ningdn pais en que la Des progrès considérables ont été 1990-96 period to an expected 75 estrategia de asistencia del Banco ha sido constatés dans deux des secteurs où les percent or higher in 1997-98, satisfactoria se han obtenido resultados résultats laissaient le plus à désirer including 7 percent with deficientes en los proyectos. (finances et gestion du secteur public) outstanding outcomes. This ainsi qu'en Afrique, en particulier dans remarkable improvement Mejora sustancial de los resultados de le secteur agricole. Ces progrès tiennent demonstrated Bank and borrower los proyectos à une meilleure performance des commitment to improving El porcentaje de proyectos financiados emprunteurs, à la conception plus development effectiveness. por el Banco que registran resultados réaliste des projets et à une meilleure There have been major satisfactorios al término de los gestion du portefeuille. Toutefois, la quality improvements in two of desembolsos del préstamo aumentó de durabilité des projets et leur impact sur the poorest-performing sectors un promedio de 65% a 70% en el le développement institutionnel sont (finance and public sector período de 1990-96 a un 75% o màs deux points qui laissent encore management) and in Africa, en 1997-98, incluido un 7% de beaucoup à désirer. particularly in agriculture. Better resultados sobresalientes. Esta notable borrower performance, more mejora ha demostrado el empeio del Une perspective mondiale realistic project designs, and Banco y de los prestatarios en L'Examen de l'an passé concluait que xiii 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS better portfolio management aumentar la eficacia en términos « le problème consiste à explain the improved - de desarrollo. parvenir à faire coïncider la outcomes. But sustain- Se han observado politique du pays et les ability and institutional « importantes mejoras de la calidad stratégies et facteurs development impact both de los proyectos en dos de los institutionnels pour tenter de remain considerably below sectores con resultados menos créer des conditions plus those levels. satisfactorios (el financiero y el de favorables à une croissance plus gestiôn del sector público), y en Âfrica, forte et à un développement plus A global perspective sobre todo en la agricultura. Este efficace ». Dans un environnement Last year's Review concluded that progreso se atribuye al mejor beaucoup plus complexe et "the challenge is to find the right desempefño de los prestatarios, al diseñio défavorable, l'Examen de cette année fit between country policy and más realista de los proyectos y a una arrive à une conclusion similaire. Il institutional factors and strategies gestión màs acertada de las carteras. est maintenant plus clair encore que to try to improve conditions Con todo, tanto desde el punto de vista l'amélioration des résultats des favorable to improved growth and de la sostenibilidad como de su impacto projets-si importante qu'elle soit- development." In a much more en el desarrollo institucional, los n'est pas suffisante. complex and hostile environment, resultados de los proyectos siguen La façon nouvelle dont la this year's Review reaches a siendo muy inferiores a estos niveles. Banque envisage d'apporter une aide similar conclusion. It is now even au développement a été mise à clearer that improvements in Una perspectiva global l'épreuve des événements de l'an project performance-important En el examen del año pasado se concluía passé. Il est certain que la stratégie though they are-are not enough. que el desafio consiste en encontrar la doit être ajustée et perfectionnée et The architecture of the Bank's justa medida entre la politica para el pais que les risques de l'environnement new approach to providing y los aspectos institucionales, y las extérieur doivent être reconnus et development assistance has been estrategias para tratar de mejorar las internalisés. La place accrue accordée tested by the events of the past condiciones propicias para un mayor au partenariat et à la lutte contre la year. To be sure, adjustments and crecimiento y desarrollo. Ante un pauvreté, thèmes majeurs du Pacte refinements in strategy must be entorno mucho màs complejo y hostil, stratégique, et l'appel lancé par le made, and the risks of the external en el examen de este afio se llega a una président Wolfensohn pour aller environment must be recognized conclusión similar. Ahora es incluso màs « au-delà des projets » dans and internalized. Nevertheless, the evidente que no basta mejorar los l'allocution qu'il a prononcée lors de Bank's new strategy for resultados de los proyectos, por muy l'Assemblée annuelle de 1998, maximizing development importante que sea este aspecto. revêtent la plus haute importance si effectiveness in a volatile global La arquitectura del nuevo enfoque l'on veut que les progrès réalisés ces environment appears well del Banco para proporcionar asistencia deux dernières années soient conceived. The increased emphasis para el desarrollo ha sido puesta a durables. on partnership and poverty prueba por los acontecimientos del año alleviation stressed in the Strategic pasado. No cabe duda de que hay que Incidences Compact, and President ajustar y afinar las estrategias, y que es Le diagnostic qui précède a les inci- Wolfensohn's call to move "beyond preciso reconocer e internalizar los dences suivantes du point de vue de projects" in his 1998 Annual riesgos que plantean las condiciones la mesure et de l'évaluation des per- Meetings speech, are key to externas. El mayor énfasis en las formances: maintaining the performance relaciones de colaboración y en el alivio improvements that have been de la pobreza que se destaca en el Pacto • Le suivi et l'évaluation des realized in the past two years. Estratégico, y el llamado que ha résultats doivent être plus formulado el Presidente Wolfensohn en transparents, la gouvernance et la Implications su discurso durante las Reuniones performance institutionnelle The above diagnosis has the Anuales de 1998 en el sentido de ir jouant un rôle de premier plan. Il following implications for "màs allà de los proyectos", son faut accorder davantage xiv Executive Summary performance measurement and fundamentales para continuar las d'attention au suivi des evaluation: - mejoras del desempefño que se indicateurs structurels et han conseguido en los últimos sociaux ainsi que des Performance monitoring dos aios. indicateurs de pauvreté. and assessment need greater • L'évaluation doit s'effectuer à transparency, with 1 Repercusiones un niveau plus élevé et porter governance and institutional El diagnóstico antes expuesto sur un pays ou un secteur, ou performance at center stage. tiene las siguientes repercusiones para la bien avoir un caractère global. More attention must be paid to medición y la evaluación del desempefño: * Les systèmes de notation des monitoring structural, social, évaluations doivent accorder un and poverty indicators. * Para hacer el seguimiento y la poids plus explicite à l'impact so- • Evaluation has to move to a evaluación del desempeio se cial des projets et des programmes higher plane, focusing on the requiere mayor transparencia, et aux effets considérables que des country, sector, and global poniendo en primer plano la chocs extérieurs peuvent avoir sur levels. función de gestiôn y la actuación les pauvres. • Evaluation rating systems have de las instituciones. Se debe to give more explicit weight to prestar más atención al En ce qui concerne les opérations de the social impact of projects seguimiento de los indicadores la Banque, il faut: and programs and to the estructurales, sociales y de important effects that external pobreza. • Consolider les succès, en shocks can have on the poor. • La evaluación debe pasar a un examinant les importants effets plano más elevado y concentrarse secondaires que peuvent avoir des Bank operations need to: a nivel de pais, sectorial y global. activités liées entre elles sur les • Los sistemas de calificación que se politiques et institutions d'un pays. • Scale-up successes, considering aplican en la evaluaciones deben * Renforcer le soutien au the important side effects that dar màs importancia en forma développement institutionnel, en interrelated activities can have explicita a las repercusiones sociales particulier pour les institutions on country policies and de los proyectos y programas, asi financières et la protection sociale. institutions. como a los graves efectos que las * Passer du stade de projets isolés à • Strengthen support for conmociones externas pueden tener une approche à long terme des institutional development, sobre la población pobre. pays au niveau aussi bien de la particularly for financial conception que de l'exécution des institutions and social Con respecto a las operaciones stratégies. protection. del Banco, es preciso: * Shift from a project to a long- term country focus in both the * Ampliar la proporción de proyectos design and implementation of satisfactorios, teniendo en cuenta operational strategies. los importantes efectos secundarios que pueden tener las actividades interrelacionadas en las politicas e instituciones de un pais. * Aumentar el apoyo para el desarrollo institucional, sobre todo de las instituciones financieras y de las redes de protección social. * Dejar de concentrarse en el proyecto y centrar la atenciôn en el pais y en el largo plazo a la hora de disefñar y aplicar las estrategias operacionales. x v ABBREVIATIONS AND ACRONYMS ARDE Annual Review of Development Effectiveness ARPP Annual Report on Portfolio Performance CAE Country Assistance Evaluation CAN Country Assistance Note CAR Country Assistance Review CAS Country Assistance Strategy CBO Community-Based Organization CGAP Consultative Group for Assistance to the Poor CSR Civil Service Reform DAC Development Assistance Committee DEI Development Effectiveness Index EAP East Asia and Pacific Region ECA Europe and Central Asia Region GDP Gross Domestic Product IBRD International Bank for Reconstruction and Development ICRG International Country Risk Guide ID Institutional Development IDA International Development Association IDF Institutional Development Fund IDS Institute of Development Studies at Sussex University IMF International Monetary Fund LCR Latin America and the Caribbean Region MNA Middle East and North Africa Region NGO Nongovernmental Organization OECD Organization for Economic Cooperation and Development OED Operations Evaluation Department OEDCM OED Corporate Evaluation and Methods OEDCR OED Country Evaluation and Regional Relations Group PER Public Expenditure Review QAG Quality Assurance Group SAR Staff Appraisal Report WDI World Development Indicators WDR World Development Report xvii 1 DEVELOPMENT EFFECTIVENESS IN A VOLATILE WORLD The current crisis confirms two main lessons for development effectiveness-and provides a new one. First, good macroeconomic fundamentals are necessary but insufficient for stable and sustainable growth. In today's global economy, sound institutions, especially in the financial and social sectors, are essential to economic and social stability. Second, projects are no longer the appropriate vehicles for development assistance unless they are connected to balanced country assistance strategies focused on structural reform and capacity development and owned by borrowers and other partners. An consensus in the development community about what important new lesson is that exogenous factors are far constitutes sound economic policy? more influential in determining development impacts than had been previously thought. Institutional weaknesses The past year produced a severe deterioration in With the crisis having lasted more than a year, one the economic conditions of the developing world. fact has become evident: the costs of unregulated Russia's economic reform collapsed in a year when movements of private capital must be balanced against transition economies were finally growing after seven the risks. Indonesia, the Republic of Korea, Malaysia, years of decline.' The sharp reversal for some of the the Philippines, and Thailand received net private world's fastest-growing economies and the greater inflows worth almost 7 percent of their combined GDP caution of private investors have chilled global eco- in 1995-96. The reversal from 1996 to 1997 involved a nomic growth. The perceived risk of investing in swing of 11 percent of their combined GDP.' Changing emerging markets made a very large one-year jump.2 risk perceptions by commercial banks, and particularly The crisis has induced concern about the ability of portfolio investors (rather than foreign direct inves- the global system to contain the contagion. The tors), explain the reversal. The sudden downgrading of economic turbulence and uncertainty call into question country credit ratings sparked panic and flight among the consensus reported in last year's Annual Review of private investors. Development Effectiveness. Is good macroeconomic Unlike the debt crisis of the 1980s, the current crisis policy a sufficient foundation for development effec- started in countries with relatively strong fiscal situa- tiveness? Does the crisis invalidate the broadly based tions, sound monetary policies, and outwardly oriented 1 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS trade regimes (table 1.1). Except in Thailand, govern- spot than macroeconomic weaknesses. Relying as they ment budgets were balanced or moving into surplus did on macroeconomic indicators, decisionmakers in when the crisis hit. Inflation was contained, interest the private sector and international financial institu- rates were going down, and recorded unemployment tions found it difficult to argue with success. In East was low. Taking into account foreign direct invest- Asia, as in Chile and Mexico before, credible domestic ment, current account deficits were not excessive. reforms, low interest rates, and good growth prospects Weaknesses in economic management helped trig- contributed to an explosion of private flows. Attracting ger the crisis. In all cases the capital account was the those flows were exchange rate pegs, profitable interest main vulnerability. Imbalances between short-term rate spreads, and liberalizations of the capital account. debt and official reserves-combined with premature Given the "halo effect" typical of investment booms, financial liberalization and weak financial discipline decisionmakers overlooked the failure of Asian coun- in domestic banking systems-created situations vul- tries to comply with basic tenets of the much-abused nerable to speculative pressures. Washington consensus, a listing of sound economic Many fundamentals were sound in crisis-affected practices on which most analysts agree. countries. The financial panic would not have spread Using the benchmarks of this consensus, Rodrik without weaknesses in domestic institutions. As in the (1996) notes that the policies of Korea and Taiwan Southern Cone crisis 15 years earlier, banking disci- (China) have long been well below par. We extend pline was weak and links among economic conglomer- Rodrik's analysis to Indonesia and Thailand in light of ates, banks, and governments were too close. This led OED's recent country assistance evaluations (table to excessive borrowing, disproportionate real estate 1.2). Cumulatively, the analysis confirms that East booms, poor private investments, and escalating levels Asian countries were following policies consistent with of nonperforming loans. Why did not policymakers and only 6 or 7 of the 10 tenets of the consensus. Caprio international financial institutions give these weak- (1998) finds that banking sectors were extremely weak nesses appropriate weight? Because the lessons of the in Indonesia and Thailand. In contrast, according to general debt crisis were guiding them, not the more Rodrik, many Latin American economies-in particu- relevant institutional lessons of Chile's 1982 crisis and lar, Argentina, Bolivia, and Mexico-fulfilled most of Mexico's 1994-95 crisis. OED's (1990) audit report on the consensus conditions. The conclusion: Washington Chile's structural adjustment loans highlighted the lack consensus policies were neither the cause of high of prudential supervision of financial institutions in growth, nor the cause of the crisis. increasing the economy's vulnerability to the point of Macroeconomic policy weaknesses were linked to collapse. A key lesson of that audit was: "prudential competition policies and financial liberalization-the rules and surveillance are necessary safeguards for the sequencing of which needed to be made coherent with operation of domestic financial markets, rather than prior institutional development and structural policy unnecessary restrictions" (p. 12). reforms. It is significant that their neglect featured Microeconomic dysfunctions are more difficult to prominently in the Chile and Mexico crises. In the TABLE 1.1: TRADITIONAL CRISIS INDICATORS Indicator Indonesia Rep. of Korea Malaysia Philippines Thailand Governnment budget deficit (perceitage of GDP) Average, 1990-94 0.4 -0.4 -0.7 -1.4 3.2 Average, 1995-96 1.7 0.1 0.8 0.4 2.6 Inflation rate change in the consumer price index) Average, 1990-94 8.8 5.3 4.1 11.1 4.6 Average, 1995-96 8.7 4.7 4.4 8.3 5.8 Currentaccount (Percentage otGDP),, Average, 1990-94 -2.7 -1.5 -7.4 -4.5 -7.5 Average, 1995-96 -3.8 -3.4 -9.7 -5.5 -9.1 Source: Reisen (1998). 2 Development Effectiveness in a Volatile World TABLE 1.2: THE WASHINGTON CONSENSUS-NOT IN EAST ASIA ticrnlb *4i [il \ .ahneron t_n,;cnu% Indonesia Rep. ot Korea Tai%%an IChinai Thailand 7 11 JRplind. tdii, p . lI Y( . L nIIE llJ YeLiY it. l Id. dii ir ri. jInd 'i l liIir IsYe 'YC 'YCs, in thre l.IrcI j, 'Jr -I iiiId L ii i n 4 i th- .. !!- I nj qii' p,iA! tiij ri.. cc.r[I 'rt 'c IS iii L l13rted h ii .Il1t in anid cht%icin u I~ 1 r r L1 i..i [ n1i d I r11[ed L.nilled I r1med . I l h Jr , 'in.. n Lirn Ecd Linitn d Li t J Ys Until rle I LEintil the I [Fu1 i lt 1 I.1; . i r;. itI.. . i In!im .-, No. ( c. riiTc Im N,.. ( Iverinlkil In ri ed. bL t IIl 1i IL c r.bhl hdlle r1iar1 . u Es ll-.1i el d i. . but lit .1r1 i-Lc pLNIC nrerphi w Public cn 11pfr JUIn* ici- I lfid dulin p 1ird Li i Intl iti.*nIi f ' Ii.: 1,t bi.reigri dilrLL Fuirernl direcet Lirntcd Ii%e;EnficInt h,~ In_%L1IIIrIer ;sIm`jt%e to foirci,_n dmrrcu libe5[FllentI hemr- iti i r in L L ircL t:t i t iCstriLcd [ii .z:l Ile (illtill i[[* ri\c'tfli-i F,n-i r, -j hhi )hIjr.. )ei LIrn Irrcd Lijr r ud 'Is, Untl li. l L.Ic . * un d trhe lJi ..In thc1, Source: Data on Korea and Taiwan are from Rodrik (1996); other data, OED, World Bank. words of Claessens and Glaessner (1997, p. 8), "liber- edented volume and reversibility of short-term capital alization is inexpensive, fast, and easy to implement; flows, weak banking institutions and ineffective regula- building institutional capacity is expensive, slow, and tory systems proved a lethal combination. complex." In sum, mistakes in macroeconomic policy The reversal of capital flows was especially deep played a part in the East Asian downturn, and an even and disruptive where domestic interest rates were bigger part in Russia, but the more critical dysfunction higher than in the international markets. Financial was institutional. Financial sectors, governance stan- supervision, corporate governance, and corruption-let dards, and corporate investment regimes (and in alone social safety nets-were too often given minimal Russia, fiscal regimes) seemed adequate as long as the emphasis in the "metrics" of performance monitoring booms lasted.4 But they proved fatally flawed once and assessment. As stressed in last year's Review, external conditions deteriorated. Given the unprec- institutional development lies at the core of develop- 3 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS ment effectiveness. The "silent crisis" of poverty and pacts turned them into targets of public criticism in the destitution that affects low-income countries is deeply 1970s and 1980s. As a result, new policies and rooted in capacity constraints. How important is the standards emerged that scaled-up the scope of the quality of the institutional environment for Bank- World Bank's intervention to include avoidance or supported projects in low-in- mitigation of the adverse environmental and social r icome countries? Very impor- consequences of large dams and, by extension, of all Iore inclusive, tant indeed, because stronger projects with significant potential adverse side effects. participatory institutions are associated To move beyond isolated success stories, the full array approaches with a 20 percentage point of factors affecting development results must be exam- must be increase in the likelihood of a ined, and more inclusive, participatory approaches deve loped. project's outcome being rated must be developed. satisfactory (figure 1.1). For Such enhanced participation is essential if the the Bank's lending to low- complementarities across sectors and activities are to income countries over the past two years, this improve- be fully exploited. However, besides developing ment would translate into a more than $1 billion broader, more inclusive approaches, the Bank and its increase in effective Bank support. partners must identify "holes in the boat." Donors cannot simply focus on projects or sectors that they Broadening the agenda to maximize development know will perform well. They must also identify points effectiveness of stress-such as financial weaknesses-that can cause The need to scale up from a strictly project-specific gains to unravel. To achieve development effectiveness focus to a broader, more inclusive orientation is now at the country level, Bank interventions must begin by well recognized. This perspective has been realized by designing effective projects. But these must link indi- hard-learned lessons-for example, from experience vidual operations to the broader social, civil, eco- with large dams. As OED (1996d) shows, large dams nomic, and international environment. They cannot were viewed as being synonymous with modernization neglect existing weaknesses that could more than offset and development in the 1950s and 1960s; but growing the development effectiveness gains from a project. To evidence of their adverse indirect and secondary im- be successful, the country assistance program must be firmly rooted in the borrower's ownership of reform FIGURE 1.1: INSTITUTIONS AND PROJECT objectives. PERFORMANCE IN LOW-INCOME COUNTRIES Another lesson of the crisis is that social develop- ment should take center stage in the financing of Average project performance (percent satisfactory) recovery or development programs. The shocks of 80 _ High institutional quality recent years have plunged millions into absolute 70 -i ni poverty. The lack of formal social safety nets for the 60 - unemployed is being felt severely, partly because of so - the weakening of traditional systems that once sup- 40 _ ported the poor. Cuts in public spending for the social 30 _ sectors and rises in prices associated with devaluations 20 _ and the removal of subsidies add to the burden on the 10 _ middle class and the poor. If the global objectives for 0 poverty alleviation of the OECD's Development Assis- Low-income countries Other countries tance Committee (DAC) are to be attained, more Note: The institutional quality index is composed of three vari- attention must be paid to social development and ables based on data from the International Country Risk Guide social safety nets. (ICRG) covering 1982-98. These variables are corruption, rule of law, and bureaucratic quality. Countries are assigned income The new primacy of the external environment groups based on classifications in World Development Indica- The central emerging lesson of the crisis is that a risky tors 1998. The average project performance for the country groups are based on project-level outcome data. external environment can strangle development pros- Source: ICRG; OED, World Bank. pects. Developing countries face the prospects of 4 Development Effectiveness in a Volatile World continuing reductions in aid; they also confront a A series of recent reports on international financial more uncertain environment than the developed architecture laid out a range of actions to strengthen countries do (see Chapter 5). Private investors will the international financial system.s The recommenda- give even greater scrutiny to the country and institu- tions call for enhanced transparency and greater tional conditions that allow entry to global financial discipline in balancing market incentives and public markets. That is why development effectiveness control. They also stress the need for improved vigi- requires a perspective that goes beyond country-level lance by international organizations. The novelty of concerns, especially for small countries, these reports lies not in the proposals themselves, but in Particularly important for the external environ- their urging the practice of what has long been ment are donor efforts to bolster financial systems preached. and to help restore the confidence that is needed for The following chapters review recent evidence the capital flows vital to restart growth. The World about the Bank's development effectiveness. Chapter 2 Bank and the International Monetary Fund (IMF) describes project and sector performance trends. Chap- now lead in the provision of information on regional ter 3 considers recent evaluation lessons at the country and global trends-certain to be an important global level. It uses OED's country assistance evaluations to public good. The Bank is especially well placed to help draw out the lessons of the ongoing crisis. Chapter assist in developing, processing, and understanding 4 draws lessons that can be inferred from OED's the often important fragments of information with thematic studies. The final chapter discusses the impli- broader implications. cations for Bank operations and evaluation. 5 2 TRENDS IN PROJECT PERFORMANCE MA mong evaluated projects exiting the Bank's portfolio in fiscal 1997 and 1998, more than 75 percent had satisfactory project outcomes. Thus the improvement in perfor- mance presaged by last year's Review has been sustained. There has also been a conver- gence in performance across regions and sectors, the result of major advances in Africa and in two of the poorest-performing sectors (finance and public sector management). Better borrower performance and more realistic project designs, as well as better portfolio management, explain most of the improvement. But sustainability and insti- The figure for exits in the first half of fiscal 1998 is 80 tutional development remain sorely neglected in percent, exceeding the target of the Strategic Compact.' project design and portfolio management. While this preliminary figure is likely to be biased OED evaluates all closed projects, assessing results upward, it represents a likely to be achieved in each project's operational substantial improvement. There has been a phase. Since last year's Review, OED has evaluated We estimate the percent- substantial 298 operations across all regions and sectors.' Africa age of satisfactory projects has the largest regional share of the evaluated cohort for fiscal 1998 exits to be improvement in project (28 percent), followed by Latin America and the 76 percent, only slightly outcomes in the last Caribbean and East Asia and the Pacific, with shares of above the level for fiscal two years. roughly 20 percent each. Nearly half the projects are in 1997.' But it is now clear the finance, private sector, and infrastructure network, that the Bank has moved in the past two years above the led by the transportation sector (30 projects). Agricul- plateau in fiscal 1990-96, when the percentage satisfac- ture projects make up an additional 21 percent. tory remained in the 65-70 percent range. Some qualifications are in order. First, the unusu- Outcomes more than 75 percent satisfactory ally large risks in the international economy call for The percentage of satisfactory project outcomes has substantial discounts in the long-term effects likely to continued to improve in recent years. The proportion of be reaped from many completed projects. Second, more satisfactory projects for fiscal 1997 exits-the latest year than 60 percent of the gains were in Africa, which has for which complete results are available-is 75 percent. aggressively implemented a portfolio restructuring 7 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS FIGURE 2.1: DEMANDINGNESS, COMPLEXITY, A third reason for caution is that these buoyant RISKINESS, AND OUTCOME outcome ratings are goal-sensitive-relating largely to planned project goals. Thus their significance is con- Percent substantial nected to the demand, complexity, and risk of project 100 - objectives. In previous Reviews outcome trends were steady or only slowly improving as the portfolio Outcome became increasingly ambitious. But there has been a 80 break in the upward-sloping curves of demandingness, -. riskiness, and complexity from fiscal 1996 to fiscal 1997-98 (figure 2.1). This shift is admittedly modest. 60 Demandingness * Still, it could indicate that the improvement in outcome .e ratings was achieved, at least in part, through the achievement of more modest project goals. Complexity Other notable characteristics of the performance 40 1 I I I I I I trends include: 1990 1991 1992 1993 1994 1995 1996 1997 1998 Note: By exit fiscal year. Broken lines (exit fiscal year 1998) in- dicate preliminary results, with less than 50 percent coverage of * Strong improvements among the previously exited operations. poorest-performing groups, including finance Source: OED, World Bank. and public sector management. plan, but remains the region with the highest share of * Continued improvement in the outcome ratings projects at risk of not achieving their development of adjustment loans, with performance levels objectives.4 But a significant part of the improvement remaining above those of investment projects. in Africa is in agriculture, which in 1993 developed an * Sustained progress in borrower performance, action plan emphasizing simpler models of delivery now at par with Bank performance. and greater sector coherence. So the recent gains may * Recent improvements in Bank performance, not be ephemeral. especially in the quality of project supervision. BOX 2.1: INSTITUTIONAL DEVELOPMENT AND SUSTAINABILITY HELP CAPTURE PROJECT PERFORMANCE B eyond the unsatisfactory outcome. wells. But the project was water-users group worked assessment of The project sought to rated as having substan- well in coordinating the goal-oriented help farmers to invest in tial institutional decisions of nearly 4,000 performance captured their own small wells in development because it independent pump own- by the outcome rating, areas where there was closed down the loss- ers; no depletion of water two examples show fresh groundwater- making public institution resources was expected. how explicit consider- directly useful for irriga- formerly responsible for Contrast this with the ation of institutional tion. Behind the outcome fresh groundwater pump- outcome of the Rehabilita- development and rating were two factors. ing, replacing it with a tion Project of the port of sustainability can First, the estimated eco- market-based institution. Dar-es-Salaam in Tanza- provide a richer nomic return fell short of Sustainability was rated nia, rated as marginally description of the the 10 percent minimum as likely for two reasons. satisfactory. Although development impact of threshold for satisfactory First, the project used there were delays in Bank-financed projects. outcomes. Second, subsi- diesel pumps, which do implementation and part In 1995 an irriga- dies and free hook-ups to not depend on the highly of the project was not un- tion project in the power grid resulted subsidized and unreliable dertaken, the port facilities Pakistan was rated as in the installation of too electric distribution sys- were converted to contain- having a marginally many private electric tem. Second, the erized operations, the 8 Trends in Project Performance FIGURE 2.2: DEVELOPMENT EFFECTIVENESS INDEX AND OUTCOME: MEASURING PROJECT PERFORMANCE Development effectiveness index 'Outcome (percent satisfactory) 9 - 90 - 8 80o 7B Byidisbursement By projects 6 - By projects 60 - 5 I I I 1 1 5 0 11 1 1 I I 1992 1993 1994 1995 1996 1997 1998 1992 1993 1994 1995 1996 1997 1998 Note: By exit fiscal year. Broken lines (exit fiscal year 1998) indicate preliminary results, with less than 50 percent coverage of exited operations. Source: OED, World Bank. In aggregate, this year's evaluation results confirm and even greater use of evaluation findings. With that improvements in development outcomes have been demand for lending boosted by the crisis, concessional sustained and broadly based. But in the meantime, the resources declining, and Bank loans more costly to overall development environment has shifted in dra- borrowers, concern with development effectiveness matically challenging ways. This shift underscores the should not be allowed to flag. need for more reliable and timely evaluation measures main project objective. As considered a critical short- development effective- second, and that is the a result, ship waiting coming. It appeared ness index, the project in information the devel- times and berth times unlikely that the weak Pakistan scores a 6.75, opment effectiveness decreased. The institu- management of the well above the Bank index conveys. A crude tional development Harbours Authority could average of 6.38 for fiscal binary analysis based component, however, was remedy the problem with- 1994-97. The project in strictly on goal-based rated as negligible be- out additional external Tanzania receives a evaluation (outcome) cause of poorly planned support. 5.25. An analysis based would have reversed training arrangements for Now consider how the solely on a binary classi- the ranking in favor of employees of the Tanza- two projects would com- fication of outcome does the project in Tanza- nian Harbours Authority. pare with one using an not take into account the nia. These issues are The sustainability of aggregate measure of contrasting performance discussed more fully in project benefits was also performance that of sustainability and in- Annex 1. disappointing, and rated included sustainability stitutional development. unlikely. Given the size and institutional develop- The descriptions of the and complexity of the ment, and one that two projects indicate that facilities installed, the lack considered only project the first had better over- of local basic skills was outcome. Using the all performance than the 9 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS FIGURE 2.3: SATISFACTORY OUTCOME BY REGION Africa (AFR) East Asia and Pacific (EAP) Percent satisfactory Percent satisfactory 100 - 100 - East Asia and Pacific 80 - 80 - 60 60 4 0 I I I II I I4 0I I I I I I I I 1990 1991 1992 1993 1994 1995 1996 1997 1998 1990 1991 1992 1993 1994 1995 1996 1997 1998 Europe and Central Asia (ECA) Middle East and North Africa (MENA) Percent satisfactory Percent satisfactory 100 - 100 Middle East and 80 - 80 North Africa Bank 60 - 60 Bank Europe and Central Asia 40 II I I I I 40 I I I I I 1990 1991 1992 1993 1994 1995 1996 1997 1998 1990 1991 1992 1993 1994 1995 1996 1997 1998 Latin America and Caribbean (LAC) South Asia (SA) Percent satisfactory Percent satisfactory 100 100 80 Latin America and Caribbean 80 'OABank * 60 Bank 60 - A South Asia 40 1 I 1 1I I 40 I 1990 1991 1992 1993 1994 1995 1996 1997 1998 1990 1991 1992 1993 1994 1995 1996 1997 1998 Note: By exit fiscal year. Broken lines (exit fiscal year 1998) indicate preliminary results, with less than 50 percent coverage of exited operations. Source: OED, World Bank. 10 Trends in Project Performance Gains in development effectiveness analysis is under way to confirm the robustness of the As part of its search for a more comprehensive measure development effectiveness index, as well as its consistency of project performance than the outcome rating, OED and complementarity to measures used by the Networks has piloted the development effectiveness index for this and the Quality Assur- Review. It integrates current OED measures of out- ance Group. Annex 1 comes, sustainability, and institutional development summarizes the index's Gains in outcome impact. The measure ranges from 2 (for a project with construction. ratings have not been a highly unsatisfactory outcome, which also has While also showing matched by gains in unlikely sustainability and negligible institutional de- improvement over the velopment impact), to 10 (where high achievements on past two years, the devel- iustinal all three measures are realized). opment effectiveness in- The index improves the presentation of performance dex trend is less dramatic development. trends in three ways (box 2.1). First, it uses the spectrum than the trend in out- of outcome assessments made by OED rather than the comes (figure 2.2). Outcome ratings gains have not been binary assignment to satisfactory or unsatisfactory out- matched by gains in project sustainability or institutional come. Second, it qualifies a project's outcome judgment development. by rewarding the robustness of achievements into the Weighted by disbursements, the data show a similar future, in some cases recognizing the lasting benefits of pattern of improvement, with the share of outcomes rated significant achievements that fall short of expectations. satisfactorily reaching 78 percent in fiscal 1997.s For Third, institutional development impact is given special fiscal 1998 exits, the preliminary satisfactory outcome emphasis. Together these aspects of the index provide a achievement rate is 84 percent, although using the more complete picture of trends. The new index facilitates development effectiveness index, the fiscal 1998 figure performance analysis of sectors and countries. Additional shows no gain over fiscal 1997, holding steady at 7.02. BOX 2.2: IMPROVEMENT IN AFRICA erformance improvement is the suc- trally planned to a mar- a financial sector gains in Africa cess of capacity-building ket-oriented economy, project more than have been driven work. There has been a and illustrates well the achieved its objectives by improvements in 16 percentage point improvement in African through better fiscal agriculture, public improvement in the num- public sector manage- and monetary manage- sector management, and ber of projects having ment projects. The pace ment and privatization finance. Of the projects substantial institutional of institutional develop- of the banking and exiting in fiscal development impact in ment in Mozambique industrial sector. This 1997-98, agriculture Africa. For example, the was rapid, with key result contributed to a dominates the Africa National Agricultural economic institutions sharp drop in inflation, portfolio (26 percent Services project in Cte such as the Ministry of and increased private share). Satisfactory d'Ivoire strengthened the Finance and the central investment and higher outcome ratings for monitoring, evaluation, bank significantly economic growth. agriculture projects and cooperative support strengthened, while Expansion of social increased from 54 activities of the Ministry progress was steady in spending also resulted percent in fiscal of Agriculture. The Tech- improving the quality of in a noticeable 1993-96 to 76 percent nical Assistance project financial reporting in improvement in health in fiscal 1997-98. in Mozambique sup- both the public and and education. One explanation of ported the country's private sectors. this extraordinary transition from a cen- Again in Mozambique, 11 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS FIGURE 2.4: AFRICA REGION-RELATIVE PERFORMANCE FIGURE 2.5: SATISFACTORY OUTCOME BY SECTOR TREND, OPERATIONS WITH SATISFACTORY OUTCOME AND EXIT FISCAL YEAR GROUP Africa (AFR) E 1990-96 EE 1997-98 Percent 100 By projects (percent satisfactory) Industry Public sector management 80- Other regions 0* Finance Water supply & sanitation 60 BakAgriculture_______________ AfricaPopulation, health & nutrition Urban 40 I I I1I_I_I development 1990 1991 1992 1993 1994 1995 1996 1997 1998 Electric power & other energy Note: By exit fiscal year. Broken lines (exit fiscal year 1998) in- dicate preliminary results, with less than 50 percent coverage of exited operations. Multisector Source: OED, World Bank. Mutior Education - Social sector But regional analysis shows the disbursement-weighted I I I I I index to have fallen in most regions, with the largest drop 0 20 40 60 80 100 in East Asia and the Pacific (1 point, for 27 percent of By disbursements (percent satisfactory) Bankwide disbursements). This was offset by a large Industry increase in Latin America and the Caribbean (1.15 points, for 23 percent of Bankwide disbursements). The drop in Public sector East Asia reflects movement in China and Indonesia management especially, which together account for roughly 80 percent Finance of the region's disbursements. Water supply & sanitation All regions except East Asia and the Pacific show improvement in fiscal 1997-98 compared with long- Agriculture term averages for fiscal 1990-96 (figure 2.3). Improve- Population, health & nutrition ments in project outcomes were largest in Africa (box Urban 2.2) and Latin America and the Caribbean, with 14 and development 15 percentage point gains in the share of satisfactory Electric power & other energy outcomes, respectively. These improvements-particu- larly in Africa, where performance has historically Transportation lagged behind other regions-have raised the global Multisector average and reduced regional disparities (figure 2.4).6 Improvements in project performance were notice- Education able in most sectors (figure 2.5). The most notable Social sector decline, among sectors with significant numbers of recently evaluated projects, was in industry. In that 0 20 40 60 80 100 sector there was a fall from 54 percent satisfactory Note: Only sectors with at least 10 exiting projects in fiscal performance in fiscal 1990-96 to a dismal 36 percent 1997-98 are included. satisfactory in fiscal 1997-98, as well as a 7 percent Source: OED, World Bank. 12 Trends in Project Performance drop in the average development effectiveness index. portfolio data do not support the probability of sus- This represents the continuation of a downward trend tained gains in the finance sector. in a sector with declining emphasis in the Bank. Adjustment loans continue to have higher average The fastest-improving sectors were concentrated in outcomes and sustainability than investment loans, the previously poorest-performing groups. Public sector although the gap has narrowed. The share of satisfac- management, finance, and industry were the worst- tory outcomes for adjustment loans rose from 74 performing sectors in fiscal 1990-96 (by projects), with percent in fiscal 1990-96 to 82 percent in fiscal 1997- development effectiveness and project outcomes well 98, compared with a rise below the Bank average. Public sector management from 66 to 76 percent for and finance now show more than a 10 percentage point investment loans. Institu- Improvements in increase in the share of projects with satisfactory tional development perfor- project performance outcomes and a significant increase in the average mance has evened to were especially development effectiveness index in fiscal 1997-98. roughly 38 percent sub- Significant improvements (by projects) were also evi- stantial for both types of noticeable in dent in agriculture; urban; transportation; and popula- loans. previously poory tion, health, and nutrition. Last year's perfor- performing groups. Care should be taken in interpreting these sector mance results suggested changes as potentially lasting shifts in performance only minor progress in trends. A comparison with performance assessments of IDA and blend-financed projects. This year's results the active portfolio currently under supervision pro- display much stronger improvement-with IDA and vides a complementary gauge.7 This provides modest blend projects among the fiscal 1997-98 exits perform- support for the view that sector improvements are ing at par with IBRD-financed projects in outcome, likely to be sustained. For example, active agriculture sustainability, and institutional development impact. projects have improved to above-average performance Considering outcomes alone, the share of satisfactory in fiscal 1998, with lending scheduled to increase in the projects among IDA loans and blends was actually near future as the Rural Action Plan is implemented. above that for IBRD-financed projects: 79 percent were The assessments indicate declining performance for the judged satisfactory in fiscal 1997-98 compared with shrinking number of industry projects. The current 74 percent for IBRD loans. FIGURE 2.6: INSTITUTIONAL DEVELOPMENT IMPACT By projects (percent) By disbursements (percent) 60 - 60 - 50 - Modest 50 L Modest 40 4 Substantial 30 30 Sustntial 20 - 20 10 - Negligible 1 - Negligib1 - 10 1 1 1 0 - 1 n 1990 1991 1992 1993 1994 1995 1996 1997 1998 1990 1991 1992 1993 1994 1995 1996 1997 1998 Note: By exit fiscal year. Broken lines (exit fiscal year 1998) indicate preliminary results, with less than 50 percent coverage of exited operations. Source: OED, World Bank. 13 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS FIGURE 2.7: SUSTAINABILITY By projects (percent) By disbursements (percent) 80 - 80 60 - 60 Le 40 - Uncertain 40 - 20 -20 Unlikely Unlikely 0 Il 0 III 1990 1991 1992 1993 1994 1995 1996 1997 1998 1990 1991 1992 1993 1994 1995 1996 1997 1998 Note: By exit fiscal year. Broken lines (exit fiscal year 1998) indicate preliminary results, with less than 50 percent coverage of exited operations. Source: OED, World Bank. Institutional development impact-improving Sustainability-low and weakening? but still weak The fiscal 1997-98 data send a mixed signal on The first two objectives of a five-point development sustainability'o that may well be a precursor of future framework offered by President Wolfensohn in his declines in performance attributable to a deteriorating address at the 1998 Annual Meetings relate to institu- external climate (figure 2.7). The proportion of projects tional development.' This is particularly important in judged as having likely sustainability that exited in low-income countries, where the potential gains are the fiscal 1997 maintained an upward trend, increasing to greatest and, as Chapter 3 illustrates, the quality of 54 percent from 46 percent in fiscal 1990-96. But these institutions the lowest. projects closed at the latest in June 1997, well before Recent evaluations show that the institutional devel- the East Asian crisis. The partial results for fiscal 1998 opment impacts of Bank projects are improving, but there is enormous scope for further improvement (figure 2.6).' FIGURE 2.8: BORROWER PERFORMANCE Current exits show historical highs of only 40 percent of By projects (percent) operations with substantial institutional development. Between fiscal 1991 and 1995 the share of projects with 1 negligible institutional development rose from roughly 20 percent to nearly 30 percent. That share dropped to the lowest on record, of just 15 percent, in fiscal 1997. By 80 _ Preparation disbursements, however, the historical trend of larger 04 projects having greater institutional development impacts Avrage shows signs of weakening. The share of disbursements 60 with substantial impact was below the project-weighted Implementation average in two of the past three years. In Chapters 3 and 4 Compliance we show the fundamental importance of institutional development in development effectiveness. The main 4990 1991 1992 1993 1994 1995 1996 1997 1998 finding: the spillover effects from better monitoring and closer attention to institutional development have been Note: By exit fiscal year. Broken lines (exit fiscal year 1998) in- dicate preliminary results, with less than 50 percent coverage of neglected, and need much greater emphasis in Bank exited operations. operations. Source: OED, World Bank. 14 Trends in Project Performance projects, exiting in the unfolding of the Asian crisis, In fiscal 1997-98 borrower performance improved show a decline to 50 percent of projects judged to have to 75 percent satisfactory from 66 percent satisfactory likely sustainability. This decrease is largely because of in fiscal 1990-96 (figure 2.8).12 This increase is the drop in East Asia and the Pacific projects-those pronounced in IDA countries, particularly in Africa most directly affected by the crisis-from 66 percent and Latin America and the Caribbean. Borrower inputs likely sustainability in fiscal 1997 to 43 percent in have improved dramatically in finance and public fiscal 1998. Similarly, there has been an almost sector management projects-rising from satisfactory doubling of the share of active projects in the Region at performance in 57 percent of operations exiting in risk of not achieving their development objectives. fiscal 1990-96 to 84 percent exiting in fiscal 1997-98. These sectors enjoyed large gains in outcomes over the Bank and borrower performance improving-but same period, confirming the importance of borrower greater gains from borrowers inputs in project performance. Improvements of more Analysis in past Reviews on the determinants of than 10 percentage points were found in urban and successful project outcomes found project-specific bor- transport operations. rower performance to be the most important determi- The increase in borrower performance reflects nant of project success. Bank performance and the improvements in project preparation and in compli- country macroeconomic policy environment were ance with legal covenants. Implementation perfor- found to be less significant." mance remains the poorest-performing dimension, with TABLE 2.1: RECENTLY EVALUATED PROJECTS: OUTSTANDING PERFORMERS AND POOR PERFORMERS 0 ou1Lr% Project name Loanicredit number Outstamdiug perjbnneis Benin Urbin Rdi.ibhiIi.u..r -nd NIluii,cnicln C2338 Brazil Land Nflancmla I - Par.in. L3018 Brazil P.irana MutinicIpal ieC1e.p.ic [t L3100 Chile Sc.,Jrid ['uhlc )cctr Nan:.tmenr L3411 China Eran Hidr irleciric L3387 China In ,d. J R HLi..nal HIaldh Dich-pmnt C2009 China N.rinaI Aterc st,ui-rn C2145 China N[r',rn iiflmo - [ari A C1885 China ship \N .t, Dip.ol C2391 Dominican Republic Prnuar% EJ10CanIonn Deelh,pnen[ L3351 Estonia Htzhs as NI.in1cn ancc L3731 Hungary Hu1inin RcOiircc. I e.cI.pmnt L3313 Lao PDR SCL nd Tccl.nmmunimnt,uun C2101 Latvia Agniulriir.fl Declopiucmn L3695 Mexico ( 'nt1-mr.)tal in;-- L)e:I.paitu Pr*rim L4123 Morocco TelCC ninici in 5ctr.r L3557 Mozambique 5.c:nd Ic:n:nu: kc...acri C2628 Peru Dcbi ind DLhb scr%lLu RLJudt.,n L4133 Tunisia PpoPlar.n .nd Familmd Hcalh L3307 Vietnam 5tru.tiril .\JiW,imen[ C2657 Poor performers Ecuador Send \.110nar Supph L2774 Morocco Rur il Primnir. Ed..n0-1 L3026 Papua New Guinea Land NI.-bib non L3051 Rwanda NcLr-r.a and Pre-lImensnen )vudic. C1796 Note: Covers the 298 projects evaluated since the last ARDE. Source: OED, World Bank. 15 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS an unsatisfactory assessment in more than a third of FIGURE 2.9: BANK PERFORMANCE evaluated projects. The overall increase in perfor- mance puts the overall quality of borrower inputs at By projects (percent) par with Bank inputs. Three of every four evaluated 100 projects now show satisfactory Bank and borrower inputs.Identification inputs. Bank performance, by contrast, improved modestly 80 Average for fiscal 1997-98 exits, with only a 1 percent increase in the overall average over the 75 percent satisfactory Supervision average for fiscal 1990-96 (figure 2.9).1 Lower-quality 60 project identification has been offset by better supervision, Appraisal while project appraisal improved slightly.14 These results, however, conceal some areas of particular improvement. Latin America and the Caribbean showed improvement 40 1 1 1 I I i 1990 1991 1992 1993 1994 1995 1996 1997 1998 in both appraisal and su- In fiscal 1997-98 pervision, raising the Note: By exit fiscal year. Broken lines (exit fiscal year 1998) in- overall Bank perfor- dicate preliminary results, with less than 50 percent coverage of borrower performance exited operations. improved to 75 mance average to 80 per- Source: OED, World Bank. cent. Conversely, bor- percent satisfactory. rower performance in East Asia and the Pacific Sector analysis projects dropped in all three dimensions in fiscal 1997- For a successful development strategy, assessing risk (as 98, with appraisal performance showing the largest measured by the variability of rewards) is as critical as decrease. Finance and public sector management and assessing the expected development impact. Risks and population, health, and nutrition projects show improve- rewards in the 12 sectors with more than 10 projects ments in Bank performance. exiting in fiscal 1997-98 are shown in figure 2.10. The origin of the axes corresponds to a reward equal to the Outstanding projects-7 percent of the total average development effectiveness index in the 1997-98 Of the 298 operations evaluated in the past year, OED portfolio, and risk is equal to the standard deviation of the assessed 20 (7 percent) as outstanding and 4 (1 percent) as index in that portfolio. The axes measure differences for exceptionally poor (table 2.1). The 20 outstanding opera- specific sectors relative to the Bankwide average and the tions all had highly satisfactory outcomes, likely standard deviation of the development effectiveness sustainability, and substantial institutional development index. Thus each quadrant, starting from the upper-left impact. These projects met or exceeded their main goals, and moving clockwise, corresponds to one of the four had highly innovative designs, and are replicable in other classes: high risk-low reward, high risk-high reward, countries or sectors. They featured strong borrower low risk-high reward, and low risk-low reward. The ownership, enjoyed highly satisfactory or satisfactory coordinates corresponding to each sector measure the quality at entry, and were well supervised. sector's risk-reward combination relative to the average The success of the outstanding projects can be project. traced to flexibility in responding to changing condi- Figure 2.10 allows a comparison of average tions-the result of consistent monitoring, good super- project results for each sector and the dispersion of vision, and partnership building. Even projects with these results around the corresponding Bankwide val- complex designs succeeded because of extensive and ues. However, these comparisons are not meant to rank effective Bank staff involvement and judicious techni- sectors. Instead they suggest that the relative risk- cal assistance (box 2.3). In contrast, only one poor reward framework may be useful for the Bank Net- performer had satisfactory quality at entry. The char- works in considering potential weaknesses and acteristic features of poor performers are a general lack strengths of Bank interventions in specific sectors, and of supervision and low borrower ownership. external challenges that the sectors may face. Take the 16 Trends in Project Performance FIGURE 2.10: RELATIVE RISK AND REWARD BY SECTOR, EXIT FY97-FY98 High risk- Risk High risk- Low reward 0.8-- High reward 0.7-- 0.6-- Industry 0.5-- 0.4-- 0.3-- Electric power and other energy sources 0.2 - - 0aMultisector Reward Public sector management 0 Education -1.5 -1.2 -0.9 -0.6 -0.3 -0.1 0.3 0.5 Water supply and sanitation Population, health, and nutrition 0 -0.2 Finance Urban development m -0.3 Agriculture Transportation Low risk- Social sector U -0.4-- Low risk- Low reward -0.5 -- High reward Note: Only sectors with at least 10 exiting projects in fiscal years 1997-98 are included. While the development effectiveness index has desirable mathematical properties for this kind of analysis, a similar performance distribution occurs when measurement relies on project outcome measures. Source: OED, World Bank. industry sector, where the average development effec- Similarly, the relatively low rewards of water and tiveness index is far below the Bankwide average, and sanitation projects-and the relatively high confidence volatility is much higher. This raises questions. Does that performance in this sector will be weak-raise the Bank have comparative advantages in supporting several questions. Why do this sector's results stand in industrial projects relative to the private sector? Or can such contrast to the rest of the Bank's portfolio? Do the below-par results for this sector be attributed to a measurement problems drive the results? Or is a sector concentration of these projects in countries lacking with such important potential effects on health and good infrastructure? How much of the volatility in poverty really a relatively weak performer? We do not industry projects is the result of poor selectivity by the try to answer these questions here. The point is to Bank? How much can be explained by exogenous present a framework that the Bank Networks can use to shocks (such as contagion effects in regional crises), compare and contrast their sector performance with the which are likely to affect industry projects more than results in other sectors. The framework itself should be less market-oriented interventions? scrutinized and tested for robustness over time. 17 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS BOX 2.3: TWO OUTSTANDING PROJECTS he importance legislature. These objec- ship is similarly illus- Finance Company of borrower tives were well exceeded trated by Latvia's Agri- were important ownership, because of the dedicated, cultural Development contributors to project commitment of Bank competent action of the Project-the first Bank- success. Firmly under- staff, and good supervi- borrower, supported by an financed investment pinning these sion is illustrated by excellent Bank team. project in Latvia. Its successes was the Chile's Public Sector Capacity improvements overall development government's commit- Management Loan. and the beginning of objectives were to ment to the project, The loan's main objec- cultural change were enhance the privatization which was strongly tives were to increase achieved in several key of agriculture, agropro- influenced by the ef- the efficiency and effec- policymaking agencies cessing, and forest fectiveness and staff- tiveness of key public and in the Library of industries through finan- ing continuity of the management agencies, Congress-above-average cial and technical project management remedy the govern- achievements for a assistance. Well-focused unit and by Bank ment's inadequate freestanding technical objectives, a simple supervision. economic analysis and assistance loan. design, and such innova- coordination capabili- The crucial role of tive features as the ties, and improve borrower commitment mobile credit officers of policymaking in the and strong Bank leader- the new Agricultural 18 3 DEVELOPMENT EFFECTIVENESS AT THE COUNTRY LEVEL Country evaluations confirm that weak institutional development has been a key prob- lem in improving development effectiveness. Risk-bearing institutions-particularly the financial system and social safety nets-have been neglected by the development process. In many low-income countries, channeling aid through isolated, uncoordinated enclave projects has left capacity inadequate. Where the enabling environment is weak, projects should be justified largely for their policy reform and capacity development impacts, with the attendant risks reduced through judicious testing of borrower owner-* When graded as projects have traditionally been ship. Country evaluations show that institution build- evaluated by OED, CAEs rate the overall out- ing is needed to ensure that a country's outward come of the Bank's country strategy as satisfac- orientation can safely reap the benefits of globaliza- tory 68 percent of the time.2 tion-and shield the poor from its shocks. * The rating of the Bank's country strategy from the In 1995 OED inaugurated country assistance CAEs is a relatively strong predictor of the average evaluations to assist the Board deliberations on country performance of Bank-supported projects. assistance strategies. By now,' 17 such country assis- * The most important development issues identi- tance evaluations (CAEs) have been produced. They fied by CAEs are consistent with those that would assess the relevance, efficacy, efficiency, sustainability, be inferred from other empirical analyses of and institutional development of assistance strategies. growth and poverty alleviation.3 Using the insights of independent professionals, they are case studies of aid effectiveness in the tradition of The country as the unit of account Cassen and associates (1994) and Mosley, Harrigan, For this Review, OED undertook a pilot analysis of the and Toye (1991). They identify lessons of experience linkages between country strategy and project perfor- and draw the implications for future strategies. More mance. As a first step, all completed CAEs were than 90 percent of operational staff preparing country subjected to a formal rating process consistent in assistance strategies found them helpful. The following structure with OED project ratings, which are based on findings emerge from an overview of CAEs: evaluative conclusions about the design, outcome, and 19 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS impact of the Bank's assistance strategy in a country.4 Bank country assistance strategies. In what follows we OED staff preparing CAEs were asked to summarize consider some of the common lessons from CAEs. their views on the outcome, sustainability, and institu- tional development of Bank country strategies as they Institutions, aid, and growth would on projects (table 3.1). Development assistance has achieved much (table 3.2). Project performance is strongly correlated with the For low-income countries the rate of improvement on quality of the country assistance strategy. None of the most measures of deprivation is considerably better countries that had a satisfactory country strategy than that for high-income countries. But improvements experienced weak project performance. In only 2 of 25 in growth have been less propitious (Ingram 1992). For periods rated did an unsatisfactory Bank country example, the weighted average per capita growth rate strategy result in satisfactory performance on projects. for low-income countries for 1980-96 (outside China CAEs' judgments about key strategic issues are and India) has been negative. So low-income countries similar to those that would be suggested by empirical do not-as economic convergence models predict- studies of growth and poverty.' However, the relatively catch up with high-income economies. Instead they fall low aggregate outcome measure, 68 percent satisfac- farther behind. tory, suggests room for considerable improvement in Poor policies have a lot to do with disappointing TABLE 3.1: PERFORMANCE OF BANK-SUPPORTED COUNTRY STRATEGIES AND PROJECTS Average project outcome performance Satisfactory Unsatisfactory Th.n a 1% -% Dlsn Ikia 7, Im a I I)9N - 2 .5 M ~irk ?cconirt, P,'ll zutIqu I 08-X-'- I Th,iil.mj, ' 7irnbi I. Ii ,94-Y6f Note: Average Bank-supported project outcomes over 50 percent are categorized as "satisfactory." Country strategy ratings of 4 and above are rated "satisfactory." Other approaches to categorizing relatively strong and weak performance, using central tendencies such as mean and median, produce similar results. Source: OED, World Bank. 20 Development Effectiveness at the Country Level TABLE 3.2: TRENDS IN SOCIOECONOMIC INDICATORS AND INCOME, 1970-96 (PERCENTAGE CHANGE) Crude Infant A.-ess to death ratc mortality rai Life ,ale water GNP per capita rfr1.111 1 (per 1,000 expectanc> of i-r- Country group pe. pl live births) iw' at bi-f/ *i alation) cc'.'6ta.t. 5 High-income 11I 72 .a. (.I Low-income 3 40 1 :60 4- Low-income excluding China and India 37 21 55 -4 Source: World Development Indicators 1998, World Bank. income growth. Yet even the low-income African coun- enjoy an institutional environment rated as marginally tries described as sustained policy reformers (IMF 1995) satisfactory-less than half the level for middle-income have had average growth of only 0.5 percent a year. The countries (box 3.1). Only 1 of 41 low-income countries growth rate among this group is only slightly higher than scores a satisfactory rating on institutional quality, that realized by Europe over the 400-year pre-capitalist while more than 30 percent of middle-income countries period before 1820 (Maddison 1997). So something other do.6 By contrast, OECD countries (except Korea) boast than weak macroeconomic policies is impeding growth. satisfactory institutional quality ratings. Among low- There is no simple explanation for such poor income countries, a low institutional rating is more performance. But one factor, which recurs in OED's common than a weak policy environment: 40 percent country evaluations, is the weakness of institutions. For of low-income countries and 77 percent of middle- instance, only 30 percent of low-income countries income countries have good policy regimes.7 BOX 3.1: INSTITUTIONS AND DEVELOPING ECONOMIES S ince the 1980s a ensuring the sustain- ment challenges through if institutional analysis new development ability of development evaluative techniques. is to become opera- perspective has programs. Yet the links The papers show that: tional, it will have to emerged. It holds that between development provide greater clarity institutions and eco- practice and academics * Variable combina- in the area of incen- nomic organizations are not strong. In addi- tions of competition, tives. Institutions mat- are the key determi- tion, the evaluation cooperation, and ter because incentives nants of economic, profession has been slow hierarchy are needed trigger motivation and social, and political to adapt its methods and to achieve positive action in both the progress. Six Nobel processes to the new societal outcomes in public and private prizes have been development consensus. specific country sectors. Incentives are awarded to scholars The papers presented circumstances. thus the first building who made pioneering at a recent OED confer- * Getting the incen- block for policy design contributions to ence (1998) illustrate that tives right is crucial and implementation, neo-institutional eco- institutions matter. The to overcoming the and the evaluation of nomics. In parallel, papers explore not only restrictions that results. The difficulty development evalua- how to get the institu- arise from the neo- lies in aligning the tors have established tions right but also how classical model. incentives structure the crucial role of to assess the fit between with the collective capacity building in institutions and develop- The papers argue that interest. Source: Picciotto and Wiesner (1998). 21 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS BOX 3.2: THE LONG-RUN EXPECTED BENEFITS OF ADJUSTMENT C an a country use the Burnside and Dol- durable adjusters are yet durable adjusters, with a weak lar (1997) measure of those that have and 11 oscillators. policy envi- macropolicy, based on adjusted for at least Durable adjusters are ronment adjust? Can the financial deficit, the past four years further divided into coun- it shift from a bad to a inflation, and openness but have not main- tries that adjusted and good regime and for 1975-96. We consid- tained adjustment maintained it at the first sustain it over a ered 43 countries for long enough (nine attempt, and countries decade or more? Or, which data were avail- years) to be classified where the policy index as a number of ana- able. The Burnside and as durable adjusters. oscillated between good lysts have observed, Dollar policy index was * Oscillators are those and bad before the do countries that constructed annually, that do not adjust but adjustment to a good attempt to adjust not and the changes in it continue to oscillate policy environment was make it-and adjust were used to categorize between weak and sustained. Eleven of the over and over again? countries: strong policy envi- 12 countries successful at If durable adjustment ronments. Their maintaining a good is possible, do adjuster * Durable adjusters are policy index remains policy index, according countries grow more those that main- volatile over time. to the Burnside and rapidly? tained a good index Dollar measures, did so To consider some of for at least nine The sample has 12 on the first attempt. these questions, we years. Not yet durable adjusters, 20 not The oscillators have Policies and institutions are weaker than they might Malawi, Mozambique, and Zambia-as well as recent otherwise be because of inadequate donor coordination. work by African policymakers on the African Capacity Zambia's country assistance strategy provides a stark Building Study (World Bank 1996a), argue that a example. In 1987 the Bank concluded that Zambia's central problem has been lack of capacity to absorb the policy regime was not appropriate for the Bank to provide volumes of aid provided. Rather than directly address- further lending. The Bank ing this, however, many development agencies, includ- maintained this position un- ing the Bank, have established parallel methods to Lack of capacity to til 1991, when it resumed channel financial assistance, ignoring the adverse absorb the aid lending. In the 12 years effects on capacity creation. This finding is not new, provided has been a before the Bank's with- but it bears repeating. For example, Johnston and Van drawal of assistance, aid de Walle (1996, p. 66) argue that "aid has rarely p averaged more than 9 per- contributed to effective institution building as it has cent of GDP a year. During bypassed local institutions in project implementation the hiatus in Bank lending, donor assistance as a share of and design. The preference for enclave projects and GDP rose to more than 15 percent of GDP a year. So parallel management structures to ministerial adminis- during a period when the Bank found the policy environ- trations has been particularly destructive." ment such that aid was unlikely to promote development A Danish government report on development effectiveness, donor assistance increased-and exceeded cooperation issues in Tanzania (Helleiner and others total investment. This pattern of donor support was 1995) reaches similar conclusions about how donor motivated by donors' humanitarian concern with the practices often undermine ownership. For example, in problems of a very poor country. But it did little to primary education, the report finds that agencies increase borrower ownership, to strengthen institutions, frequently manipulate the choice of government de- or to reduce poverty. partments they work with in order to achieve their CAEs of low-income countries-Albania, Ghana, objectives. The report contends that where the govern- 22 Development Effectiveness at the Country Level had a weak policy index, measure depends funda- one in six. And as mea- associated with success- on average, for about 13 mentally (and arbitrarily) sured by OED, the ful adjustment can be years and for more than on the period chosen to outcome of adjustment illustrated by the differ- 60 percent of the period qualify as a successful loans in not yet durable ences in average for which data were avail- adjuster. It also depends adjusters would be strong growth, per capita able. They also perform on the quality of the and very similar (if income, inflation, and badly on Bank adjustment country's institutional en- slightly higher, 85 per- volatility of these vari- loans-with only 49 vironment. For example, cent satisfactory) to that ables in the different percent satisfactory in countries that have of durable adjusters. country classes. In outcomes. The average satisfactory institutional But regardless of the successful adjusters, per number of oscillations environments, the prob- threshold, it is clear that capita GNP grew at between strong and weak ability of successfully the probability of success almost three times the policy regimes for this adjusting increases to 30 is not high. So, for adjust- rate of countries that group is four. If we add up percent. ment to have a lasting, have not yet achieved the times all three catego- The threshold of nine high payoff, the gains durable adjustment, ries of countries have years is arbitrary-and from adjustment must be and six times faster attempted to adjust, the conservative. If we substantial. Are they? than oscillators. The probability for a country reduce it to seven years, Does this turnaround to adjusters increased their to adjust durably is about the odds of success good policy make a growth rates more than 12 percent. Of course, this increase to 16 percent- difference? The payoff sixfold. ment is reluctant to agree to a donor's project, there highly innovative projects overall, "IDA's strategy ... have been implicit threats of a reduction in general did not fully appreciate the risks of overload inherent in donor support. The report suggests that it is common a rapidly growing and diverse portfolio and a fragile practice for donors to pay "incentives" to government institutional framework" (p. 12). Donor assistance in officials working on their projects. Albania reached more than 50 percent of GDP. Berg (1993) points out that, in such situations, In contrast, in middle-income countries, external "technical cooperation takes on a role different than its assistance flows tend to be too small to have much traditional one: it substitutes for and subsidizes govern- impact. For instance, shortly after the 1994 devalua- ment operating budgets. It does this directly by tion in the 13 West African countries belonging to the payments to government staff on projects, and indi- Communaute Financiare Africaine, the Bank lent the rectly by financing experts to do operational work largest borrower in the zone (C6te d'Ivoire, a low- normally done by government employees. This is income economy) an amount equivalent to 1.5 percent disadvantageous in two ways. It misuses the technical of its GDP, or less than 2 percent of Bank commitments assistance personnel resource, reducing its effectiveness that year. But to provide similar support to the three for institution building. And it is extremely costly; East Asian economies-Indonesia, Korea, and Thai- high-cost expatriates are hired in posts that nationals land-that ran into difficulties in 1997 would have could fill more cheaply" (pp. 213-14). required lending more than three times the scale of net The effects of lack of donor coordination on transfers that actually took place, equivalent to more institutional development can be particularly acute in than 60 percent of total Bank lending for 1997. small countries, where the sheer volume of external assistance (and the associated absorptive capacity Adjusting to the external environment constraints) can hinder development effectiveness. The quality of lending and the support for institution Albania's country assistance review (OED 1998a) building are only parts of the equation. Equally important found that despite supporting and helping to design for development effectiveness is how well a country 23 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS BOX 3.3: MEASURING BORROWER OWNERSHIP AND ITS RELATION TO ADJUSTMENT OUTCOMES orrower owner- Wasty (1993) highlights * Is there observable The evaluation ship of adjust- the symbiotic relation- consensus among key shows that measures ment programs ship between borrower ministries and of ownership are is often cited as vital ownership and program decisionmakers on strong predictors of for making policies outcomes. It presents the the nature of the outcomes. And credible, safeguarding following performance crisis and the neces- through selected coun- against policy rever- measurement criteria for sary actions? try case studies, it sals, and ensuring ownership: * Have specific ascertains the impor- that the benefits are up-front actions been tant factors that sustainable. An OED * Is the initiative for initiated before the account for differences evaluation of nearly formulating and program? in the intensity of 100 adjustment pro- implementing the ad- * Has participation borrower ownership. grams in 42 countries justment plan the taken place within by Johnson and borrower's? the society? adjusts to the external environment, so that opportunities policy environment? In the cases considered, only for growth can be exploited and the poor can be insulated about 12 percent of adjuster countries realized a from adverse shocks. CAEs provide detailed and useful permanent and major improvement in their macroeco- examples of how countries adjust, or fail to do so. nomic policy environment. But for them, per capita income growth rates have been almost three times Outward orientation and growth those for unsuccessful countries, and more than six To nurture sustained ownership, reform requires a clear times higher than their pre-adjustment growth rates. understanding of borrower interests in the light of Thus a review of the broader adjustment experience political economy considerations. Where the seeds of supports a key lesson of CAEs: realization of high-payoff borrower ownership are in place, lending can be a successes requires careful analysis and cooperation, useful instrument of reform. But as CAEs show, devel- rather than the use of simple rules to determine whether opment assistance is not science-it is art. In unstable support should be provided. This complexity does not policy environments, there is no substitute for case-by- mean that indicators are not available. For instance, case assessments, framed to distinguish risks worth information on seeming intangibles-including concepts taking (C6te d'Ivoire and the Philippines) from risks such as borrower ownership-can be made operational, that are inappropriate (Kenya). and obvious tests can be used to sort out the likely The expected gain from assistance is the product of sustainability of investments in policy change (box 3.3). the probability of success times the rewards of adjust- By relying on such characteristics to guide Bank ing. The enormous gains that can be realized from support for adjustment lending, the probability of success adjusting are often overlooked (box 3.2). Many studies can be improved. Indeed, the probability of success in focus only on whether adjustment took place-not on Bank-supported adjustment lending has increased signifi- the payoff. A perspective that considers only the risk of cantly in recent years-because the Bank has become failure and not the gains from success can be mislead- more conscious of the importance of borrower ownership. ing. It is the perspective that might be used by a lender Combining this insistence on ownership with a greater facing roughly the same level of loss with each failure, concern for institutional quality could improve perfor- and a payoff that does not increase with the gains from mance even more. It could also increase the legitimacy of success. This kind of decisionmaking is inappropriate subsequent reform efforts. for an equity investor-or for the kind of development partner described in the Strategic Compact. Outward orientation and poverty alleviation What happens if a high standard is set for what is In low-income countries, capacity weaknesses are judged to be a permanent, lasting improvement in the pervasive. In middle-income countries they tend to be 24 Development Effectiveness at the Country Level specific. The country evaluation of Poland (OED equality within countries.' Increases in inequality were 1997e) suggests that reform of the social safety net 5 times more frequent than decreases: 49 countries remains one of the most important items on the experienced increasing inequality, 15 appear to have unfinished policy agenda. Recent reports on Thailand no trend, and 10 had decreasing inequality. Inequality and Indonesia (World Bank 1998) and a growing body is particularly acute in transition economies (box 3.4). of empirical work summarized in Levine (1997) show But it is also increasing in countries that previously had that in middle-income countries the institutions that no trend or a decreasing trend. specialize in bearing risk-financial intermediaries To sum up, growth and an outward orientation are and social safety nets-have been neglected by the keys to reducing poverty. But alone, they are not development process. Such oversight has had serious enough. For sustainable progress, better safety nets and economic and social consequences, as recently demon- better-targeted expenditures on those aspects of poverty strated in Indonesia, the Republic of Korea, Russia, and for which markets do not work are essential. As Sala-i- Thailand. Martin (1997) documents, such expenditures can have Over the long term, openness helps poor countries strong positive effects on growth. Similarly, OED's grow faster, and assists in reducing inequalities Social Dimensions of Adjustment (1996e) found that no (Edwards 1997). But openness makes it more important country has achieved sustainable poverty reduction to put in place well-functioning risk-bearing institu- without growth. It also showed that the quality of tions. To estimate the effects of shocks on income growth is critical to the distribution of benefits. Much distribution, OED updated data from Deininger and greater emphasis on safety nets-and expenditures on Squire (1996) and added estimates for other countries. sectors not adequately funded by market processes- The data indicate an overwhelming increase in in- are needed if poverty is to be alleviated. BOX 3.4: GREATER POVERTY AND INCOME INEQUALITY IN TRANSITION ECONOMIES overty has in- and there were declines in Republic, and Slovenia- unequal developing creased across 14 of 17 countries for inequality was historically countries. Eastern Europe which there are data. The low and has risen moder- Poverty and income and Central Asia. Life income and health dimen- ately. In the second inequality have expectancy at birth has sions of poverty have also group-Poland, Romania, increased in all transi- declined precipitously in deteriorated. Nine of the and the Baltics-inequal- tion economies since the several countries, most 17 countries with data ity was slightly higher late 1980s. Some in- notably Russia, where experienced increases in than in the first group at crease in income in- the average life expect- infant mortality, and 10 of the start of the transition equality in the region, ancy for men in 1995 16 countries with data and has since increased to even in the long run, is (58.3 years) was three experienced a deteriora- levels at or above the probably an years below that in tion in secondary school OECD average. In the unavoidable conse- India and a stunning six enrollment, third group-primarily quence of the introduc- years lower than at the The close relationship Russia and Bulgaria, and tion of market-based start of transition. The between poverty and perhaps including other rewards. But aside from drop in life expectancy income inequality distin- Commonwealth of Inde- this, structural change in other countries, such guishes three groups of pendent States countries- and economic disloca- as Ukraine and the countries. In the first inequality has shot up tion have introduced Baltics, is similarly con- group-the Czech Repub- from moderate to levels additional inequality centrated among men, lic, Hungary, the Slovak typical of the more and poverty. Source: Based on EBRD (1997), annex 2.2. 25 4 THEMATIC EVALUATIONS AND INSTITUTIONAL DEVELOPMENT The current financial crisis has far-reaching implications for development practi- tioners, and for evaluators. A higher priority must go to monitoring financial sector performance-and to the wide range of institutions involved in improving gover- nance. Emphasis must also be given to the institutions-such as NGOs and civil society-that help those not served by formal institutions. And to improve the effectiveness of public expenditures, practitioners and evaluators should help to introduce results-based manage- ment in the administration of development programs. evaluations for the wide range of institutional devel- Structural and social constraints to development opment issues involved in improving governance. need far more scrutiny. OED's Process Review of World Bank Grant Programs (OED 1998j) shows that Financial sector progress has been made on broadening the Bank's The financial institutions in crisis countries violated agenda and developing instruments to nurture the virtually all the institutional norms recommended by a many kinds of institutions that can address these recent OED study on Financial Sector Reform: A constraints. For example, the Institutional Develop- Review of World Bank Assistance (OED 1998f). The ment Fund (IDF) and the Consultative Group to Assist OED analysis of financial sector interventions focused the Poorest (CGAP) promote institutional capacity- on analyzing how the elements of the Bank's evolving building and donor coordination. Similarly, the financial sector policy were reflected in Bank-sup- mainstreaming of new lending instruments-such as ported projects. In examining 23 countries, the study Learning and Innovation Loans and Adaptable Pro- found a satisfactory and sustainable outcome in only gram Lending-represents tangible progress in the 12. The recommendations of the study-especially on development of stronger, more sustainable institutions. the timing, sequencing, and scope of regulatory inter- But much more needs to be done. vention-are more relevant than ever in light of the Although it is well known that institutional past year's events. According to Reisen (1998), careful factors are essential ingredients of economic growth monitoring of financial institution conditions in the and social stability, these factors remain neglected. crisis countries would have revealed serious weak- This chapter considers the lessons of OED thematic nesses in their financial systems. Consider four key 27 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS indicators of financial system strength developed in the by the uncertain fiscal implications of honoring these OED study in the crisis countries. explicit and implicit government guarantees. Capitalization and bad loan exposure Accounting and prudential standards At the outbreak of the crisis, nonperforming loans were The weaknesses of accounting standards in crisis coun- highest in the Republic of Korea (16 percent of total tries are common in many emerging markets: inconsistent assets), but similarly high in Thailand (15 percent) and financial reporting; limited power of auditors to examine Indonesia (11 percent; table 4.1). These figures are much company records; lax auditing and accounting standards, higher than the 9 percent in Mexico in early 1995, where out of line with international good practice; lack of the cost of rescuing banks has been estimated at about 15 penalties for incorrect reporting of information; and percent of GDP (Caprio and Klingebiel 1996). In Indone- tolerance of multiple accounts. In such environments, sia, Korea, and Thailand, capital-asset ratios were 6-10 even detailed examinations by supervisors and regulators percent. So even before the crisis, nonperforming loans far may not reveal the information needed to regulate outweighed (on average) bank equity capital. properly or to ensure prudential soundness. Government interventions Enforcement capabilities Banks in crisis countries may have been affected by Although some of the crisis countries had strengthened certain kinds of government intervention in bank their supervisory and regulatory infrastructure during lending and corporate finance. The governments often the 1980s and 1990s, partly in response to costly directed lending toward particular sectors, both for- banking crises in Indonesia and Malaysia a decade mally and informally. In addition to explicit guaran- earlier, enforcement capabilities remained weak tees, there were implicit guarantees that led to pre- (Fischer and Reisen 1993). Bank regulators had im- sumptions of government bailout for nonperforming posed limits on bank lending, including liquidity loans in favored sectors. This encouraged excessive requirements, exposure limits, and risk-based capital investment and risky lending. Once the bubble burst, requirements. But according to Reisen (1998), these the shaken investor confidence was further undermined standards and ratios were poorly enforced. TABLE 4.1: BANK SYSTEM RISK EXPOSURE IN EAST ASIA Indicator Indonesia Rep. of Korea Malaysia Philippines Thailand Bank system exposure to risk (percentage of assets, end-1997) Nonperforming loans 11 16 8 6 15 Capital ratio 8-10 6-10 8-14 15-18 6-10 Real estate exposure 25-30 15-25 30-40 15-20 30-40 Collateral valuation 80-100 80-100 80-100 70-80 80-100 Regulatory features during tbe 1990s Bank lending to connected firms High High Governme,r.rJrcrJ I. t lIndil.C YC Yes Yes Yes Yes Bank dep. - r iTermT N'S No No Yes No Importance .. iun ci d boink- Hi-h High Accountir.- r w.J di Wak Weak Weak Weak Enforcemu.r .V C.uu; H.t Weak Weak Weak Weak Incentives .a pa,l r,tv, Short-term inflows Limited Limrd Promoted Ipr._,ri, , ri Long-term inflows Limited Lirr- icJ Pr. rno-c[,d Outflows Fruv Linr1rcd Linircd Source: Reisen (1998). 28 Thematic Evaluations and Institutional Development The crisis makes it urgently necessary to adopt a yes. As measured by the development effectiveness central recommendation of the OED study on financial index, Bank-supported projects in countries with an sector reform: "more resources should be allocated to inadequate bureaucracy are on average the weakest monitor and evaluate countries' financial sector pro- performers.2 In countries grams, with performance indicators" (p. 83). The crisis with a well-functioning Do better governance also is the reason for Bank management's seeking special bureaucracy, projects per- budget authority from the Board to support financial form much better, with and lower corruption sector work. But above all, it confirms the serious adverse significantly lower risks improve the effects of neglecting institutional development. than average. development Bank-supported effectiveness Governance issues projects in high-corruption of projects? The crisis has reinforced the already-strong evidence economies have had sig- that growth in per capita income is enhanced by strong nificantly lower returns Unambiguously, YES. property rights, sound legal foundations, and capable with significantly higher civil servants-all operating in an effectively managed risk. Corruption is almost always associated with low institutional system. There is clear evidence that bureaucratic quality, so that public sector management corruption in these institutions hampers growth.' Do projects in corrupt countries are particularly likely to be better governance and lower corruption improve the low-return/high-risk projects. Only education projects development effectiveness of projects? Unambiguously, have a high return and a low risk in countries with high corruption. Projects in low-corruption economies nearly FIGURE 4.1: PERFORMANCE OF CIVIL SERVICE always have a higher return and a lower risk than REFORM INTERVENTIONS comparable projects in other countries. OED evaluated projects *Perhaps the most difficult governance issues Percent arise where the state has collapsed or failed, particu- 0 20 40 60 80 100 larly in countries that have recently emerged from I I I I conflict. In general, projects in these countries tend Downsizing to have lower rewards and higher risks, reflecting the Organizations and structures turbulence that conflict engenders. But risk-reward Quality of human resources __performance varies significantly across sectors and, as might be expected, many types of infrastructure Principal agent relations projects perform well in societies that have experi- Reforming i o ___________enced destruction and civil conflict. Another message of OED's (1998k) study of the Recently completed and active projects Bank's experience with post-conflict countries is that the Percent Bank can assist best if it avoids the overzealous pursuit of 0 20 40 60 80 100 fiscal rectitude. The circumstances of these countries Downsizing require that the first emphasis be on support for rebuilding Downszingthe institutions of government and civil society. Organizations and structures Governance institutions Quality of human resources The Bank has long supported efforts to improve the Principal agent relations workings of the state. In the 1980s, Bank assistance Reforming institutions _primarily sought to make governments "leaner" through downsizing. In the 1990s the Bank sought a l Unsatisfactory g Lack of monitoring &evaluation "clean state" for its clients by strengthening the a Satisfactory credibility of governance institutions through institu- Note: The data reflect OED evaluations through July 1998. tional reforms-that is, intra-public sector regula- Source: Civil Service Reform Study, forthcoming, OED, tory reform and establishment of checks on arbitrary World Bank. action. For both approaches, the results have been 29 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS meager. OED recently evaluated more than 300 civil had only a modest effect on Bank lending strategies, service reform interventions supported by 124 lend- client expenditure policies, and aid coordination. The ing operations approved during 1980-97. A signifi- study argued that such reviews could become signifi- cant percentage of completed and ongoing opera- cantly more effective if they were more demand- tions lacked the adequate monitoring and evaluation responsive, if they were better synchronized with information needed to authorities' budget cycles, and if they gave due NGOs and CBOs can be make meaningful assess- recognition to institutional constraints. particularly important ments of performance. In many societies, nongovernmental organizations Only 33 percent of the (NGOs) and community-based organizations (CBOs) in projects for interventions that could provide a closer link to the poor than public sector improving gender be evaluated achieved institutions. According to an OED (1998h) study of equality, protecting the satisfactory outcomes NGOs in Bank-Supported Projects, 38 percent of environment, and (figure 4.1). Institutional Bank-supported projects include NGOs or CBOs in development impact was their plans. This involvement increased to 46 percent alleviating poverty, rated as substantial in of projects in 1997, more than doubling from 20 only 16 percent of com- percent in 1989. The study found that these institutions pleted operations-and in only 10 percent of com- can be particularly important in projects targeted at pleted stand-alone projects. improving gender equality (80 percent), the environ- Overall project performance in the public sector ment (54 percent), and poverty alleviation (48 percent). management sector is improving, but civil service But their capacity is often limited by erratic funding reform components remain among the weakest-per- and a lack of financial independence. forming interventions in the Bank's portfolio. As with Low government capacity to work effectively with financial institutions, a central recommendation of the NGOs and CBOs is also important. While there are OED analysis is that far greater priority should be some outstanding examples of government agencies given to integrating the use of performance indicators that have shown a strong ability to work with NGOs, to monitor and support more effective public adminis- these are exceptions. The Bank's capacity to encourage tration efforts. Results-based management systems can NGO and CBO involvement in projects remains be an effective way to focus public sector performance limited. And as for financial institutions and public on outcome measures (box 4.1). sector institutions, the Bank's database and statistics on Public expenditure analysis is central to the Bank's NGOs and CBOs do not provide a reliable picture of policy dialogue with member countries. It is a rapidly their involvement in Bank-supported projects. Nor do expanding aspect of the Bank's economic and sector work, they describe results. The database mainly records having grown from 3 reviews before 1979 to 39 in fiscal whether provision was made for NGO or CBO 1998. Public expenditure reviews are a means for the involvement, not the actual involvement. external evaluation of a borrower's fiscal policies and Much remains to be done to develop a better sector reform efforts. They provide a framework for understanding of the task of institutions in assessing and coordinating external assistance and assessing its effec- encouraging development effectiveness. As measured by tiveness, and they can provide a micro foundation for the the performance of Bank projects, there are significant, IMF's macroeconomic framework. IDA's deputies have broad gains to be realized from developing and maintain- stressed the importance of public expenditure reviews as ing well-run and effectively managed public institutions. instruments for client capacity development as well as for Increased vigilance on corruption-and increased reli- enhancing development effectiveness by integrating re- ance on new public sector management techniques, such view results with country assistance strategies. as results-based management-could have positive OED's (1998g) study of The Impact of Public spillover effects on the overall quality of Bank assistance. Expenditure Reviews found that quality has improved The Bank could more systematically engage the institu- in recent years. But it also found that public expendi- tions of civil society in addressing gender equality and ture reviews provided good (but often dated) analyses poverty alleviation. The Institutional Development Fund, of spending policies with little concern for cost effi- the Consultative Group to Assist the Poorest, and the new ciency or the quality of public services. Then reviews lending instruments are promising vehicles for doing this. 30 Thematic Evaluations and Institutional Development BOX 4.1: PUBLIC SECTOR PERFORMANCE REVIEW: INTEGRATING PUBLIC SECTOR PERFORMANCE WITH A RESULTS-BASED MANAGEMENT SYSTEM A n OED study be tabled in Parliament tries, institutional norms are expected to be yes; of Public two to three months are strictly adhered to, in developing coun- Expenditure before the budget state- and there are severe tries, this will not be Reviews (1998g) ment. In contrast, public moral, legal, voter, and the case. discusses how this sector values are rarely market sanctions against The analysis and analysis might give addressed in developing noncompliance. In devel- recommendations in greater emphasis to countries, because the oping countries, non- such a review must be output measures rather orientation is to "com- compliance often is consistent with (and than traditional input mand and control" rather neither monitored nor recognize) any incon- measures. It says that than to serve the citizenry. subject to sanctions. sistencies among a to provide relevant Authorizing environ- Operational capacity country's mission and analysis of public ment. This includes and constraints. What is values, its authorizing expenditures, a public formal (budget processes authorized is not neces- environment, and its sector performance and institutions) and sarily what will get operational capacity. If review must begin by informal institutions of done: available opera- so, the review will developing an under- participation and tional capacity may not enable the client and standing of three accountability. Do these be consistent with the external partners, contextual dimensions institutions and processes task at hand. Even the including the World of the country's public provide an enabling envi- operational capacity that Bank, to understand sector. ronment for the public is available may be better how to improve Public sector mission sector to meet its goals? circumvented by the public sector perfor- and values. Societal Do the various levels of bureaucratic culture or mance. It will further values and norms-as government act in the by incentives that reward serve as a catalyst to embodied in the consti- spirit of the constitution command and control- introduce results-based tution or in annual bud- in exercising their and corruption. Some management in devel- get policy statements- responsibilities? What key questions: Do the oping countries. Such may be useful points of are the checks and bal- agencies responsible for an approach to public reference for public ances against deviant various tasks have the sector management sector mandates and the behavior? Are there capacity to undertake would help to change values inherent in those formal rules to ensure them? Are there binding bureaucratic culture mandates. In industrial fiscal discipline? Is public contracts on public man- from its emphasis on countries, the mission sector borrowing subject agers for output perfor- command and control, and values of the public to financial market disci- mance? Does with arbitrary and sector are spelled out in pline? How is govern- participation by civil oppressive rules, to one a medium-term policy ment performance society help alleviate focused on serving its framework. For measured? Are output some of these con- citizens, earning their example, in New and outcome indicators straints? In industrial trust, achieving results, Zealand a policy state- for public services moni- countries, answers to and working better for ment of this type must tored? In industrial coun- most of these questions less money. 31 5 IMPLICATIONS FOR THE BANK AND FOR EVALUATION The implications of the current financial crisis are sobering for the Bank and for the evaluation profession. It has become amply clear that the value added by development assistance programs and by evaluations would be substantially enhanced by more explicit attention to exogenous factors and long-term structural constraints. There should be a sharper focus on the measurement of poverty reduction as the acid test of development, and better methods of assessing institutional development at both project and country levels. The 1997 Annual Review of Development Effective- Global risks ness concluded that "the challenge is to find the right fit How hostile the current environment is to developing between country policy and institutional factors and countries is shown by Euromoney's country risk rat- strategies to try to improve conditions favorable to ings.1 For this Review, OED calculated a GDP- improved growth and development" (p. 51). In a much weighted measure of country more complex-and hostile-external environment, this risk for developing and in- In a much more year's Review reaches similar conclusions. It is now even dustrial countries. The mea- clearer that improvements in project performance are not sure for developing countries complex-and enough. Broader structural and social constraints impede shows a deterioration in the hostile-external project effectiveness, but so too does the riskiness of the past year to the riskiest level environment, global environment for developing countries. since the Latin American improvements in Before considering what these constraints mean for debt crisis-and one of the Bank operations and their evaluation, longer-term biggest adverse shifts since trends in Bank performance and development effective- World War II. The measure are not enough. ness need to be evaluated and placed in the context of also shows that in the after- the unprecedented events of the past year. It is particu- math of the crisis, the external environment for devel- larly important to consider the implications of these oping countries remains, unlike that for industrial events for the prospects of sustainable growth and countries, at a high level of uncertainty-again, the poverty reduction. highest since the debt crisis. For industrial countries, 33 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS BOX 5.1: WHAT DO WE KNOW ABOUT THE EXTENT OF POVERTY? he Bank is completed in fiscal 1996 its poor. The quality and count index-counts considered the (17 countries and 5 availability of household people below the poverty largest reposi- updates) and 10 (8 coun- survey data for developing line but ignores what is tory of information on tries and 2 updates) in countries have improved happening to them and poverty. But it has fiscal 1997. By 2000 the considerably since then. whether they are becom- systematically focused plan is to complete the Today 138 surveys are ing poorer. In the extreme, on data collection remaining 22 assessments available for 69 countries, the measure actually only since 1991, when and 9 scheduled updates. The timeliness of data has improves if the poor die it issued a directive When the Bank's World also improved from an av- from poverty (Sen 1976). mandating poverty Development Report on erage lag of 11 years in World Development assessments in bor- poverty was published in the mid-1980s to about 5 Report 1990 recognized rowing countries. So 1990, poverty measures years now. Even so, World that for any given increase far, 94 assessments could be calculated for Development Indicators in the incomes of the poor, have been done (83 only 11 countries. These (World Bank 1997d) the reduction in poverty countries and 11 surveys covered the 10 reports estimates of the depended on where the updates) covering years leading to 1990 and population living below poor were relative to the about 90 percent of together accounted for 40 $1 a day per person for poverty line. If they were the world's poor. percent of the total popu- only 60 countries. concentrated just below Twenty-two poverty lation of the developing The most commonly the line, the increase in assessments were world and 50 percent of used measure-the head their incomes would have the aggregate trend is the opposite: toward less volatil- likely performance-expectations often not realized. ity. Developing countries are now perceived as very The measures can change after investments have taken risky investment environments. Does this matter? place and been evaluated, either to reflect the changes Perhaps a great deal. While the relationship among that occurred or to correct expectations. Countries can such aggregate measures as country risk, investment do much better than was expected-as many did in the flows, and project perfor- years before the East Asian crisis-and much worse mance is not simple, the than expected-as in East Asia and Russia over the past Performance broad dimensions of the rela- year. How did changes in country risk affect the improved tionship are clear. As the debt performance of Bank projects exiting in fiscal 1997 and because it was crisis took hold in the mid- 1998? Consider three types of economies: those where 1980s, risk measures in- country risk during the implementation of Bank being swept creased, and Bank perfor- projects was stable, those where the country risk rating along by a mance deteriorated to below improved significantly during project implementation, rising tide. 70 percent satisfactory out- and those where it deteriorated. comes. Then, as the crisis was Countries whose circumstances improved account resolved and the environment for more than 60 percent of the exiting projects. In improved, private capital flows increased sharply and these countries, risk perceptions improved consider- project performance improved, particularly in the past ably during implementation, as the world moved to few years. These relationships are hardly precise. But one of the highest-ever growth rates for developing they cannot be ascribed to coincidence. In disentan- countries. Not surprisingly, project performance in gling some of the possible effects for development these economies was good-the likelihood of a project effectiveness, examining the relationship to Bank per- having a satisfactory outcome in those countries was formance is instructive. 85 percent. This is 11 percent higher than in economies Measures of country risk are expectations about that performed as expected when the project was 34 Implications for the Bank and for Evaluation a bigger effect on poverty are often dissimilar. measure of welfare the poverty lines too than if they were spread * Designed to yield a involves setting poverty high to include, say, more evenly.4 wide variety of often lines and denoting the countries in Eastern It remains difficult to different types of household cost of the Europe in the com- compare rates of information. level of welfare needed parison raises the poverty in different to escape poverty. Best estimates of poverty countries. There are Some obtain informa- practice involves adjust- for other countries. conceptual and practical tion only on incomes, ing for differences over If this vast heteroge- problems.s The surveys while others gather time or space and neity in the underlying from which the poverty information only on con- household demograph- data is not carefully data are drawn are: sumption. Most differ in ics. But the data needed controlled for, aggre- the depth and detail to do this consistently gation to obtain * Taken at different about consumption. are inadequate and regional or global points in time. Methods of valuation generally variable. The estimates will not be * Based on different vary considerably, with problems of making valid, and comparisons sample designs that some surveys using prices purchasing power parity across countries and may not be nation- at the nearest market, currency adjustments for time are, at best, ally representative. while others use farm- international compari- spurious. * Conducted under gate prices. sons add to the lack of methodologies that The money-metric comparability. Setting initiated. Fortunately, outcomes appear to be more percentage points.3 Clearly then, evaluation must give robust with respect to unexpected deterioration in greater prominence to the effects of the external country risk. Countries that experienced an increase in environment, particularly for sustainability. risk, even those that had a substantial increase, did not experience a large reduction in project performance. Prospects for poverty reduction Overall, in deterioring economies, the likelihood of a The events of the past year confirm that the assessment of satisfactory outcome was 73 percent.2 So at least from development effectiveness should give pride of place to a preliminary analysis of performance, the deteriora- poverty alleviation. Chapter 3 reviewed the effects that tion in conditions should not be devastating for the the recent financial crisis and the ongoing transition from existing portfolio. But the effects on future projects may socialism have had on income distribution. What these be more serious. shocks mean for poverty alleviation can be seen by In some respects the recent improvement in overall considering what they mean for the OECD Development performance on Bank-supported projects has been the Assistance Committee's (DAC) goals in poverty allevia- result of projects being implemented in economies tion, which call for a reduction in the number of people in undergoing improvements in broad fundamentals. In absolute poverty of 1 billion by 2015. To do this means this light, performance improved because it was being that about 50 million people must be raised from poverty swept along by a rising tide. But what will happen if each year for the next 20 years.' The East Asian crisis has this tide has crested, and future performance is no already put the DAC program nearly a year behind longer buoyed by a continually improving external schedule. But this was not the only poverty-increasing environment? If current forecasts of country risk are shock. The increase in poverty brought about by the accurate and a large portion of Bank borrowers do not collapse of safety nets in transition economies is larger experience an improving external environment during than the East Asian effect: to offset these increases will implementation, overall satisfactory performance of require another three to four years of successful effort in future projects could be reduced by as much as 5 poverty reduction for the DAC goals to be realized. 35 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS Together these two shocks have moved the goal contribute to development effectiveness without a greater posts four to five years farther away than when the understanding of how it fits into the broader country and targets were established only two years ago. Much international environment. more must be done in improving poverty measures if For evaluation, OED will reemphasize and inten- the Bank and the donor community are to come to grips sify its shift in focus to the country as the appropriate with the full dimensions of global poverty (box 5.1). unit for evaluation. It will also give more attention to While the goals may be a useful structure, greater sector performance. These new emphases will build detail is needed to flesh out all the important dimen- on-not substitute for-OED's traditional concern with sions of poverty reduction. Better data are needed so project performance. that the poor truly become visible in our evaluations and goals. More than safety nets and income growth Institutional development are needed to achieve DAC's poverty goals (box 5.2). The events of the past year have revealed how costly weak institutions can be. The importance of sound Implications for the Bank institutions has long been known. Not known was how What does all this mean for Bank operations and costly a systemic institutional failure can be. Bank evaluations? Beyond doubt, three issues require greater operations have made significant progress in focusing emphasis: poverty alleviation, country strategies, and resources on this important issue. And as discussed in institutional development. Chapter 1, a major dimension of this work will be on financial institutions: recent events underscore that Poverty alleviation much more needs to be done. In recent years poverty alleviation and social concerns For evaluation, greater emphasis should be placed have been afforded more attention in Bank strategies, on the metrics of institutional development. The devel- as mandated by the Strategic Compact. This year's opment effectiveness index is but a first step on this Review finds that much closer donor coordination is journey, and further work should be done toward essential, particularly in low-income countries. In these giving institutional development more emphasis in countries, declining volumes of external assistance evaluation measures. Simpler, concrete steps are must be better coordinated and be more sharply focused needed. For example, today's project supervision and on poverty reduction. completion reports do not require a rating of institu- OED evaluation efforts should give more emphasis tional development impact. This shortcoming should to social aspects of Bank operations. Rating systems be rectified. OED should make sure that its new should give more weight to the social impact of country evaluation instrument gives adequate attention projects and programs. Recent shocks have had a to institutional development issues. profound negative effect on poverty, and we cannot The past year has tested the Bank's new framework exaggerate the importance of measures to address these for providing development assistance. So far the results concerns systematically. Safety nets should be at the are promising. The Bank's general strategy appears to be forefront of poverty alleviation concerns, rather than well conceived. With continued adjustment and refine- being comfortably assumed away as they have been in ment, it should permit the Bank to help developing many analyses of world poverty. countries confront a much tougher external environment. Evaluation is central to adjusting the Bank's Country strategies approach. For example, systematically addressing pov- The Bank has made progress on broadening the develop- erty alleviation requires making the poor visible ment effectiveness spectrum-for example, it has already through better data and monitoring systems. Output moved beyond projects in its strategic perspective. But as targets must be linked more tangibly to policy inputs discussion of the external environment showed, continual and must recognize the increased risks of the external improvements in Bank and borrower performance may environment. With the help of evaluators, those targets not be enough to maintain the performance of recent must be embedded in the scorecards of country and years, much less continue the steady improvements. In sector assistance strategies. such a context, even the best-designed project will not 36 Implications for the Bank and for Evaluation BOX 5.2: POVERTY ALLEVIATION AND INCOME GROWTH In assessing and economic historians market institutions ap- control. human welfare, is skepticism, because pear to have generated So higher life advocates of they view improved life the new technology of expectancy cannot be human development expectancy as a by- disease control. taken as simply a by- would place indi- product of economic But perhaps economic product of economic cators of social development. growth has been neces- growth or the free conditions-notably But new techniques of sary for increasing life market conditions that life expectancy and disease control, using expectancy-by provid- foster it. Indeed, pub- educational achieve- new knowledge of ing the resources needed lic policy initiatives ment-on an equal disease, have been to fund public spending have been essential to footing with such sources of improved life on the new technology, the improvement of traditional economic expectancy. And public either directly or through life expectancy, and measures as GDP per intervention has been international aid, or to these can be-and capita and a poverty crucial for implementing fund the research respon- have been-under- index. Some would them. The free market sible for the advance in taken in the absence of include indicators of institutions commonly knowledge. At most, economic growth. political and civil considered to be behind economic growth may Thus, broader liberties. To the economic growth have have been helpful, but it measures of develop- broader measure of not been responsible for was not required to ment are important. human development, adopting the new finance the advance in a common reaction techniques of disease knowledge that brought among economists control. Nor do free infectious disease under Source: Easterlin (1998). 37 ANNEX 1. A NEW APPROACH TO EVALUATING BANK PROJECTS-THE DEVELOPMENT EFFECTIVENESS INDEX The current Bank evaluation system assesses project assigning cardinal weights to the ratings of each of the results through a set of three ordinal ratings-on three results-oriented counts, then combining them in a project outcome, sustainability, and institutional devel- simple way. The index formula is: opment impact: DEVELOPMENT EFFECTIVENESS INDEX = OUTCOME WEIGHT + * Outcome is established by answering the following SUSTAINABILITY WEIGHT + INSTITUTIONAL DEVELOPMENT WEIGHT. question: Did the project achieve satisfactory develop- ment results considering the relevance of its main stated Outcom wigt objectives, and the associated costs and benefits? The Highly satisfactory 7.75 outcome rating takes into account relevance (to check S atisfactory 6.00 whether the project's objectives were consistent with the Marginally satisfactory 5.5 . . argnaly unatifacory4.50 country's development strategy), efficacy (to examine Mainaly a ct whether the operation achieved its stated goals), and Unsatisfactory 3.75 . - ighy usatifacory2.00 efficiency (to assess results relative to inputs by costs, implementation times, and economic and financial Sustainability weight returns). Outcome is rated on a six-point ordinal scale: Likely 0.75 highly satisfactory, satisfactory, marginally satisfac- Uncertain 0.25 tory, marginally unsatisfactory, unsatisfactory, and Unlikely 0.00 highly unsatisfactory. Institutional development impact weight * Sustainability is defined as the likelihood, at the time Substantial 1.50 of evaluation, that the project will maintain its Modest 0.50 results. In assessing sustainability, evaluators focus Negligible 0.00 on features (country conditions, government and economic policies, the political situation, and condi- tions specific to the operations, such as availability Thus, the development effectiveness index ranges of funds for maintenance) that determine whether the from 2-for a project with highly unsatisfactory out- operation will last over its intended useful life, come, unlikely sustainability, and negligible institu- Sustainability is rated on a three-point ordinal scale: tional development impact-to 10-for a project with likely, uncertain, and unlikely. highly satisfactory outcome, likely sustainability, and * Institutional development impact is defined as the substantial institutional development impact. It is easy extent to which a project has improved an agency's to see that outcome is the main force behind the index. or country's ability to use its human and financial Note also how the index separates between satisfactory resources effectively and to efficiently organize eco- and unsatisfactory outcomes, where an index measure nomic and social activities. Institutional develop- of 6 represents such a "divide." A project with an ment impact is rated on a three-point ordinal scale: unsatisfactory outcome will never score higher than 6, substantial, modest, and negligible. no matter what ratings it receives on the other two dimensions. As in previous years, we report trends on each The average development effectiveness index in the evaluation dimension separately (see Chapter 2). In this fiscal 1990-98 portfolio is 6.47. The standard devia- year's Review, however, we have introduced a new tion is 1.85. The contribution of outcome to the measure of overall project results-the development average index in the portfolio is about 80 percent; the effectiveness index is based on the three ratings. This remaining 20 percent is almost evenly split between the instrument allows us to take the analysis of portfolio other two evaluation components. trends a step forward, making it possible to compare A similar cardinal measure of overall project perfor- overall average results-and their variability-across mance was presented in last year's Review. In building different groups of projects. that index, the aim was to establish, using subjective The development effectiveness index is defined by assessment, the relative importance of the three results- 39 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS oriented counts, and then combine them in an intuitively establishing the ranking of observed projects on which appealing way. But this approach, by ignoring the the index is built, we used (although parsimoniously) information embedded in the historically observed portfo- outcome as a tie-breaking rule. Second, the fact that lio, tended to excessively penalize underperforming outcome was rated on a six-point basis, as opposed to projects and over-reward overperforming ones. It induced the three-point basis used for the other two counts, a double-counting effect that duplicated a satisfactory- resulted in outcome ratings conveying more informa- unsatisfactory dichotomy, rather than conveying the extra tion, and thus receiving more weight. That outcome information contained in the six-point classification of evaluation is based on a finer scale is a testimony to its outcome results and the data about institutional develop- prominence among the ratings. We welcome this ment impact and sustainability. asymmetry, because we did not have to impose any The new measure improves on that index, making subjective mechanism to give outcome more weight in the inferences based on such an instrument more the determination of the index. robust. It does so by explicitly taking into account the Although we report the percentage that each result- information coming from the historically evaluated based count contributed to the index, this was done only portfolio, and by keeping to a minimum-and transpar- for completeness. After all, the index's purpose is to ently stating-the set of subjective judgments needed to summarize information as far as possible: researchers choose a specific index among the many ways of interested in examining performance on each count will defining one. find it optimal to look only at the ratings in the chosen Weights are assigned, using the historically evaluated dimension (rather than look at the components of the portfolio as a benchmark, and taking into account the weights, which are intimately connected to one another). strong positive association among evaluation dimensions, Another point worth stressing about the index is the to avoid double-counting effects and to extract the most nature of its cardinality. This should be understood as information from the empirical observations. interval-scale cardinality: because it is impossible to The fact that good (bad) ratings on one dimension locate an absolute, nonarbitrary, zero point for the are associated with good (bad) ratings on the other scale, distances between index values, although mean- dimensions allows us to unambiguously rank more ingful, are not ratios. We choose a 2-10 range because than 70 percent of the projects in the observed portfolio this makes the index readily comparable with other using only the ordinal information conveyed by the indexes used within the Bank, while allowing us to three sets of ratings. Using only two clear assump- space index values in a way that may be appealing for tions-premised on the Bankwide consensus about the practitioners and nonpractitioners alike. importance of outcome ratings and on the most Although it is not of the ratio-scale nature (ratios of efficient way to use information embedded in the indexes are not meaningful, given the arbitrary range), observed portfolio-we are able to increase the total of the cardinality of the development effectiveness index projects that can be ranked to more than 81 percent. makes it a valuable instrument for analyzing Bank Such rankings are then used to assign scores to project performance. The index makes it possible to each dimension separately. Choosing a formula that compare overall performance across, for instance, Sec- defines the index as the sum of three scores makes it tors, Networks, and Regions, using the information easy to understand and calculate within the Bank, as contained in the project ratings. This represents a step well as outside, and allows us to readily calculate forward with respect to standard comparisons based changes in the overall index associated with changes in simply on binary classification (satisfactory or unsatisfac- the distribution of ratings in a given group of projects tory) of outcome, which leave out the sustainability and (by sector, region, and so on). institutional development components of project results, Caution should be exercised in interpreting the and ignore the nuances embedded in the six-point scale on scores on each separate dimension, since these are outcome. This is the use of the index we are mainly derived by looking at the additional informational concerned with in this report (see Chapter 2). content that each result-based count contained relative But the usefulness of the index is not limited to what to the other two in the historically evaluated portfolio. we have suggested here. Using the index, means and The fact that the ratings on outcome appear to be variances for different groups of projects-representing driving the index is the result of two forces. First, in different types of investment-can be calculated, and the 40 Annex 1: A New Approach to Evaluating Bank Projects-The Dcvelopment Effectiveness Index portfolio can be evaluated, using the standard tools of for direct cardinal measures of project performance, portfolio theory. like the rate of return approach to project evaluation. The richer information embedded in the index can But given the shift of development economics toward be used as an alternative dependent variable in greater attention to policy and institutions, the rate of regression studies that seek to identify the factors return methods are ill-fitted to capture the policy behind the Bank's intervention successes and failures reform and institution-building components of Bank (see Burnside and Dollar 1997). The index could also projects. This discrepancy motivated the adoption of be used as an explanatory variable in the right-hand an evaluation system applicable to all projects (for side of regression analyses linking Bank efforts to about a third of Bank projects, rate of return is still development results (reduced poverty or inequality, calculated) that sacrifices the advantages of cardinality increased growth, and so on). to capture the many facets of project results. Thus the The guiding rationale of translating into a cardinal development effectiveness index represents an attempt measure an ordinal system of performance ratings is to to solve the tradeoff between these two conflicting come up with an evaluation instrument with desirable factors, and should not be construed as an attempt to properties, to be used to trace out the major trends, "faithfully and precisely" measure in a cardinal way factors, and effects of Bank investments. The develop- the underlying reality. ment effectiveness index is not meant to be a substitute 41 1998 ANNUAL REV[EW OF DEVELOPMENT EFFECTIVENESS ANNEX 2. GLOSSARY OF SELECTED TERMS Adjustment loans: Financing aimed at promoting Development Assistance Committee in a 1996 policy policy reform. Disbursement of these funds, directed at paper (OECD 1996). alleviating the costs of the transition to a different policy and institutional environment, is contingent on Demandingness, complexity, and riskiness: Demand- the fulfillment of a set of conditions by the recipient ingness refers to the extent to which the project could country (usually based on macroeconomic indicators). be expected to strain the economic, institutional, and human resources of the government/implementing Bank performance: The quality of service delivered by agency. Complexity refers to such factors as the range the Bank, especially in tasks for which it has primary of policy and institutional improvements contem- responsibility, such as appraisal and supervision. plated, the number of institutions involved, the number of project components and their geographic dispersion, Borrower ownership: The extent to which the recipient and the number of cofinanciers. Riskiness refers to the country is involved in and committed to a project's likelihood that the project, as designed, would be strategy and goals. Ownership is greater when the expected to fail to meet relevant project objectives borrower initiates the formulation and implementation efficiently. In determining project riskiness, evaluators of a project, when there is clear consensus among consider the extent to which the project could reason- government officials and other decisionmakers on the ably have been expected at the time of project course of action, and when there is broad public preparation and appraisal to face known risk factors, support for the initiative. such as lack of borrower commitment, inadequate counterpart funding, and war or civil disorder. Borrower performance: Defined as the assumption of ownership rights and responsibilities and delivery of Development effectiveness: A demonstrable contribu- the inputs needed to prepare and implement the project. tion to economically sound, socially equitable, and environmentally sustainable growth. Contagion: Transmission of destabilizing conditions from one open economy to others closely connected to Development effectiveness index: A measure of overall it, resulting in regional crisis. project-specific results using the index aggregation of three OED project ratings: outcome, sustainability, and Country assistance strategy: The main vehicle for institutional development. Board review of the Bank Group's assistance to IDA and IBRD borrowers. The strategy document describes Efficacy: A measure of whether an operation achieved the Bank Group's strategy, which is derived from an its physical, financial, and institutional objectives. assessment of country priorities and indicates the level and composition of assistance to be provided consistent Efficiency: An assessment of results in relation to with the strategy and the country's portfolio perfor- inputs, including costs, implementation times, and mance. The heart of the country assistance strategy is economic and financial returns. the ongoing Bank-country dialogue and joint efforts in preparing and implementing the strategy. Strong coun- Global public goods: Goods that are available for the try ownership and consultation with key stakeholders- benefit of all countries, and for which one country's use pursued with sensitivity and the general agreement of does not reduce another's consumption. Yet no single the government-are crucial features of a successful country could or would invest in these goods because country assistance strategy. the costs generally outweigh the aggregate benefits. DAC goals: A set of six internationally accepted Institutional development: Improvement in the ability development goals for the twenty-first century in the of an agency or country to make effective use of human areas of poverty, gender, education, environment, and and financial resources and to efficiently organize health. The goals were published by the OECD's economic and social activities. 42 Annex 2: Glossary of Selected Terms Macroeconomic policy: Government actions designed to Strategic Compact: The Strategic Compact between affect the entire economy rather than specific sectors or the Bank's management and Executive Board provides markets, especially with respect to the general level of a long-term framework for guiding the Bank's renewal income, employment, prices, interest rates, and balance and calculating the associated resource needs. The of payments. Policy measures are usually categorized as Compact, approved in March 1997, adds $250 million fiscal or monetary, depending on which instruments- to the Bank's $1.2 billion administrative budget, to be taxes, public spending and debt, control of money supply, used over 30 months to improve how the Bank does and central bank discount rates-are used. business. The Compact is complemented by substantial redeployments and savings throughout the Bank, iden- Outcome: In project ratings, outcome refers to the tified through an ongoing review of cost-effectiveness. extent to which a project achieved its major objectives The Compact focuses on refueling current business in a cost-efficient way. Under the results-based manage- activities, refocusing the development agenda, retool- ment framework, outcomes are the immediate effects ing the Bank's knowledge base, and revamping institu- and changes that result from a project's outputs (their tional capacities. services and products). For example, the outcome of a health publicity campaign might be a 5 percent Sustainability: The likelihood, at the time of evaluation, increase in awareness among those targeted. that a project will maintain its results in the future. Relevance: The consistency of goals with the country's Washington Consensus: An internationally agreed set overall development strategy and the Bank's assistance of 10 measures that are typically implemented during strategy for the country. policy reform, including fiscal discipline, financial and trade liberalization, deregulation, taxation and public Risk-bearing institutions: Organizations designed to expenditure adjustments, and privatization. share the risks and costs of unpredictable events among large groups. Examples include unemployment support programs, health programs, and life insurance. Safety nets: Mechanisms that aim to alleviate the burden on the vulnerable of an unfavorable economic situation (for example, by unemployment insurance). 43 j TABLE 1: OUTCOME, SUSTAINABILITY, AND INSTITUTIONAL DEVELOPMENT (ID) IMPACT FOR EXIT FISCAL YEARS 1990-96, ' - 1997, AND 1998, BY SECTOR, NETWORK, LENDING TYPE AND SOURCE, REGION AND INCOME GROUP (BY PROJECTS) m a Exir FN 1990-96 [-,iI 1N 199- EXit 11 19). z Suit. I1) Suq[. IE) SU;. IE) Project Ourcome I . Impaci Projec[ Outcome Impact Project Outcome i.. Impact > No. Share .. .r I klivl i . hlc. I No. Share ,'. at. I,kI v ('.. hk,.f No. Share (", 4?.' l ik - , ... Sector 3 Agriculture 4 1 1 1 .1 .4 N 2 23 1 54 2 o Education 15 S -i 4 -4 -h 14 4-sS )S- Electric Pom ro 0 &OtherEircrLc 148 4 5jq 32 1 i 4 i1 - 6 S. 33 m Environment 2 0 liii 50 . - .3 1 2t I I 1-.1. 1.1 . < Finance S9 5 1 43 2- 14 i'a 4, - 6 CS ''D Industry .5 S 54 41 ', - -1 4 4 p _ 0 M ining I I ' 'I 5il ' I 6 i ili ' ' S I So liii > Multisector I 1S 5 32 21 10 si 4 24 9 8 - 2 Oil&Gas 49 si 4 2 5 24 2 2 In Sn So Population, HealthkS NInI.d S1 4 ¾' 1. t3 "3 & -U2C 12 II hS 4- ),1 n Public Sectot Managemoti 811 i 4 '. 12 6 6 - 9 C l)1i n " SocialSecto! Iii I .i 4ii ill 3 Ijl 14 1 - a -I ii r Telecommun mI- irii -- I t 0 2ill)i ii Iii Q Transporta--l*n 18 12 -4 S2 2" 21 10' -is n 5 -4 4ii% SI' 4i' Urban Dev1l..pricuu -- 1, in 2I lb 44 P 6 5 1iii1 6i 4(1 r Water Suppl m & SanitatidI6 4 hi1. 4 1, IT!i S) ' Network Environmerr, m Socially Sr ir.111dl, Developm(ni 433 2 '- 1 55 27 55 2 2-1 in r" Finance, Pri'.ac sc,t-, & Infrastriil'c TN 4- 4 1 XX 4I 53 -1i 41 36 4 54 4 Human De',-loinclui l' 1.2 --I 5l 1i 318 IS I2 15 45 2. 2 -11 22 Poverty Red.',ni-i C Economic\'l. c.cii 218 14 61' 5ui ; .1 jS a 5. I1 10 .'- 44 Note: Sust.= sustainability; sat.= satisfactory; sub.= substantial. Income group categories are derived from the World Development Indicators 1998. A full set of supplemental statistical tables providing further project evaluation results is available electronically at http://www.worldbank.org/html/oed TABLE 1: OUTCOME, SUSTAINABILITY, AND INSTITUTIONAL DEVELOPMENT (ID) IMPACT FOR EXIT FISCAL YEARS 1990-96, 1997, AND 1998, BY SECTOR, NETWORK, LENDING TYPE AND SOURCE, REGION AND INCOME GROUP (BY PROJECTS) (CONTINUED) Exit I-N 1990-th [ut FN 199-' Fxit FY 1998 SuNt. 1) S st. ID Su5i. 11) Projec 1 Ou come ii . m paci Projec Ou rico . Im pac Projeci Ou ome 1 ni pac N". Shiare , , a.' l. lu ' i . . No. Share i s*r.' it/ v l c No. Share c.. :,i.' ihl ' ,ii, ;IL',rl s2 1 S 2 -j ¾n li.i 45 ¯4 oss 41 4 4 [ ,t i l 0- 4 18 i 11) Å ,0 i k. i i L kr' i 4 N1i44dI,1-4 1 li 2'LJ, -I. 44 2 ' S 5 2 8 814 2 ald/atra get1.5-9 100 46 31 21-' 100 -5 5411414 100 80 51 4) Note: Sust.= sustainability; sat.= satisfactory; sub.= substantial. Income group categories are derived from the World Development Indicators 1998. A full set of supplemental statistical tables providing further project evaluation results is available clectronically at http://www.worldbank.org/html/oed 4. o TABLE 2: OUTCOME, SUSTAINABILITY, AND INSTITUTIONAL DEVELOPMENT (ID) IMPACT FOR EXIT FISCAL YEARS 1990-96, 1997, AND 1998, BY SECTOR, NETWORK, LENDING TYPE AND SOURCE, REGION AND INCOME GROUP (BY DISBURSEMENTS) 00 Exit FY 1990-96 Lxr FY 199' Exit FY 1998 z L) ID [F) Disburw. Outcome Suit. Impact Disburse. Outcome Suit. Impacr Disburse. Outcome sust. Im1pacr .SM11ILo 1 I-... "dr. ' . l/kele S . 0 . 5 /s n l I'.s t l r o lk l a *. s b ' mllih01S 1 ("1' ' Jt.I hk./r 111 '.:1al Sector Agricil[tIrV 26,660 -4 - 3,1)t N( 1o 42 1,056 8- -1 Educa[n 5,993 80 .1 4" 1.438 -I 43 l62 95 '' Electric P .%,. 0 & 11 O vnw f ~r.... 19,3SO 61 Ii AS 1,28 C 4'' 5-2 14(j9 -I Environmnt .5 I r2 '1 11 232 -8 ". .4i 34 11) III.) Financ, 9,-6- 61. 1A3S 2.336 St) 45 i S4 -85 -4 1,1 Indus-r% 11,890 t4 5 ' 1.338 5 N1 4W ( 0 I* o Minim 1,.271 1, il 41 lb6 ) 4 lilli "O 16 - 14i -14 Multisct..r 21,91). 86 45 -I 1.488 911 1 25 60 99 - Oil &c C'b 14x3 83 42 '4(, Liq QI As 34- 11, ) 47 Populzicl-m, H-Icilclh &Nic* 1,62 i v. f, s'im 94 1 22-41 N) ~ . Public scc.r n Sociali nr-r t-4 9 1" IN 43I 111 . NA 181 1 Cl Telecco nn111L . [(1-1.N 1,.30f, N ' 44 382 0 111. s 91 111') jI111 lill Trans.-.rE..c[in 1.198 83 jq )3 2,04- 84 44 tIl1 6-5 11.(1 s hii Urban rh- lpintc 6.8 5'In 20 6 -0 58 In 544 1) _ - Water bWppi% & Saiuir .-n 4,.5- 5 2" 's 1,18 v. l II 564 9 .1 24 Netwoi k Environii,n.ilIk Socialli I.ncnibk Developnnr 26,-4 4 4- V 3,328 -~ 41 1.090 -1 Finamc, PNi ic Stc.hi A link 1 crruci'L 4.99. i SS' A; 11-i.C 9. 85 44 4,318 81 5 42 Human Dc;clopiieir 8.'4) 81 t'4 42 2I 30 9(1 4" 1,285 82 -o 10 Poverti RCluc'.n & Ecoroii, N I Il'cInt 25.87- 84 (,4 41, 1.93 10 SS 2l 1,0;3 A' Note: Disburse.= disbursements; sust.= sustainability; sat.= satisfactory; sub.= substantial. Disbursements are measured in real terms, deflated to fiscal 1996 US dollars. Income group categories are derived from the World Development Indicators 1998. A full set of supplemental statistical tables providing further project evaluation results is available electronically at http://www.worldbank.org/html/oed TABLE 2: OUTCOME, SUSTAINABILITY, AND INSTITUTIONAL DEVELOPMENT (ID) IMPACT FOR EXIT FISCAL YEARS 1990-96, 1997, AND 1998, BY SECTOR, NETWORK, LENDING TYPE AND SOURCE, REGION AND INCOME GROUP (BY DISBURSEMENTS) (CONTINUED) F%is FY 1990-' Exit FN l09- Ftir FY 191)s 11) ED ID Disburse. Outcome Sust. Impaci Disburse. Out,:ome war. Impact Disburse. Outcome SLIS1. Inpact 4 1j, Ii'' L) -S(4 \J1in[inenr 4I.blo 5'-4 43 3.'1 on 5 Ti L.42 9-4.191 42 Si 4 14,11S N, 1 l --4 S.S4 NI I khri. L. lIIIL 1f;RI' --ih 4x'6 19 -S o 4 4'' 1I. i 4 4.9 ' ~ 4 41 IP ' -d b1 ' 4 1" 4 -11 1 41 1.3. t k ~ 1 1d[cl. 2,44t,' 14 - 4~ iI 4 1 "C' I I l.ilIll k ,ii N N I rb Al'I 1.1.11 I 44 - 1. . I ' 4 -Is 0 1 I".S .1 4- 31 :S,h4n s'42 1, Icl I h i4 L. 1 77--I1 JI1. 4.',9 l 1 i 4,04 4 '41 lu 0 -I 11111 3 1 I 11,1"i nl'dic " .'Il - ' 4 4, 4.1 li,I - 4 " 43 LJ 4 .r.13.8 11 -4 i F7 1.834 4' -2 .-2(,S- _S _4 i4 Note: Disburse.= disbursements; sust.= sustainability; sat.= satisfactory; sub.= substantial. Disbursements arc measured in real terms, deflated to fiscal 1996 US dollars. Income group categories are derived from the World Development Indicators 1998. A full set of supplemental statistical tables providing further project evaluation results is available electronically at http://www.worldbank.org/html/ocd a -4O 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS ANNEX 4. REPORT FROM COMMITTEE ON DEVELOPMENT EFFECTIVENESS (CODE) On December 9, 1998, the Committee on Development can use in developing goals and targets for the Effectiveness (CODE) reviewed the annual summary "Corporate Scorecard." The Committee welcomes the entitled Managing Development Effectiveness: An launching of a new program of sector strategy papers Overview of FY98 Apex Evaluation Report (SecM98- to assist in this regard. Furthermore, it emphasized 941). This "chapeau" document is prepared jointly by strongly that the Networks must become much more OED and management to facilitate Board discussion by involved in evaluation and must contribute more to the highlighting the main conclusions and priority con- dissemination and communication of information on cerns arising from the FY98 Annual Review of Devel- results and to the promotion of learning. At the country opment Effectiveness (ARDE), the 1998 Report on level, the Committee noted that significant progress Operations Evaluation (AROE), and the Annual Re- has been made in enhancing the strategic focus of view of Portfolio Performance (ARPP). CASs, but said that CASs should routinely analyze In general, the Committee commended OED and portfolio experience, OED lessons, and QAG findings, management for the excellent quality of the reports. It and the link between portfolio performance and new welcomed the continued improvement in the overall lending should be made more explicit. quality of the Bank's portfolio, but recognized that the A key element of RBM is the "Corporate improvements should be interpreted with caution and Scorecard," which is designed to monitor progress that there are major challenges ahead. The Committee toward agreed targets in the Bank's processes, outputs, endorsed the overall recommendations of the reports and strategies-and progress in those development and stressed the need to communicate general recom- outcomes and goals in borrowing countries in which mendations widely and to convert them into definitive the Bank is engaged. Much remains to be done to follow-through that will lead to poverty reduction. make the Scorecard consistent with results-based man- First, one of the main messages emerging from the agement. The Committee noted management's concur- three reports is that we are continuing to enhance rence with OED's recommendation in this regard. It evaluation processes, and that the Bank is at an early looks forward to further discussions with management stage of practicing results-based management (RBM), on issues such as the integration of assessment indica- which is both a management system and a performance tors across corporate goals and strategies and Bank reporting system. RBM provides a coherent framework performance, and the development of procedures to for learning and accountability in a decentralized link more explicitly the performance assessments of environment. CODE endorsed OED's strong recom- managers to the quality of the work produced in their mendation for implementing RBM, which was first units. made in the 1997 AROE. According to the 1998 Second, the Committee believes that a closer link AROE, progress has been made in enhancing perfor- among strategic planning, budgeting, and evaluation is mance management and the use of lessons learned for needed. This requires more systematic, timely, and development effectiveness in the Bank. However, the user- friendly evaluation processes, as well as a tighter reports conclude that the Bank needs to accelerate coordination among the Strategy and Resource Man- progress toward the implementation of full-fledged agement Vice Presidency, self-evaluation activities in RBM. This would entail linking corporate resource Operations, and independent evaluation in OED. The allocation processes to results achieved and incorporat- Committee urged management to accelerate progress ing evaluation information into management perfor- toward developing an integrated work program com- mance assessment. A clear statement of goals and bining independent and self-evaluation, as recom- objectives at the corporate level is the essential precur- mended by the Evaluation Learning Group and en- sor for this. The Strategic Compact has given the dorsed by CODE and the Board. It looks forward to institution as a whole a clear sense of momentum and reviewing a progress report on monitoring and self- direction, and the new development framework will evaluation in FY99. help sharpen it. At the sectoral level, the Committee Third, in view of the shift toward the country level stressed the urgent need for sectors to develop strategies in the new development framework, the Committee with clear goals, targets, and indicators the Networks emphasized the need to strengthen partnerships with 48 Annex 4: Report from Committee on Development Effectiveness (CODE; borrowers and with donors. It stressed the critical concerns fully across all Bank work. It also endorsed the importance of stakeholder participation in setting recommendation that social development should become development goals and in the design and implementa- central to the Bank's work. The Committee looks forward tion of appropriate monitoring and evaluation (M&E) to hearing from management about how these recommen- systems at the project, sector, and country levels. dations will be implemented. Although progress has been made and participation has Conclusion. These reports clearly identify the brought substantial benefits in enhanced stakeholder major challenges the World Bank faces and make buy-in, better implementation, and reduced risk, the strong, well-founded recommendations for how these Committee was concerned that M&E and evaluation challenges might be met. We must now translate these capacity development are receiving too little attention. recommendations into action. The Board and manage- Evaluation in the Bank cannot substitute for an ment need to work together to ensure that, as an effective system of evaluation in borrowing countries. institution, we accelerate the pace at which we imple- Evaluation capacity development must be a priority ment RBM and all the attendant systemic changes for Bank operations. This has been a recurring theme required. The Committee recognizes that the broad in CODE statements in the last few years, and we urge systemic changes needed to achieve a full-fledged RBM management to assure action on this priority. The system in the Bank will take a sustained effort over a Committee looks forward to seeing the results of a number of years. However, we urge management to be review of the implementation of the 1994 Evaluation more explicit about its commitment to achieving this Capacity Development Task Force. It also urged goal and how it plans to address these issues. management to make evaluation capacity develop- Working closely with management, CODE will ment a major focus of partnership pilots, and a key develop a work program for 1999 that will, in large theme in the Bank's forthcoming sector strategy for part, be derived from the conclusions and recommenda- assisting public sector reform. tions of the apex evaluation reports. We will discuss the Fourth, the Committee stressed that poverty reduc- most appropriate and effective ways for CODE to tion must continue to be the ultimate goal of all Bank monitor progress in key areas. An indicative work activities. Strengthening partnerships must be a key program for CODE will be circulated shortly. element of the Bank's poverty reduction strategy. The crisis in East Asia has highlighted the importance of giving considerable attention to the strength and resil- ience of institutions, particularly in the financial and social protection areas. The Committee concurred with Jan Piercy the recommendation that the Bank should move institu- Chairperson, CODE tion building to center stage and mainstream institutional 49 ENDNOTES Chapter 1 to be achieved. Potential problem projects are rated satisfactory 1. World Bank 1998e. on implementation progress/development objectives, but have other risk factors historically associated with unsatisfactory 2. Based on Euromoney's ratings for country risk. outcomes. 3. IDS 1998, Chapter 1, p. 1. 8. "The Other Crisis," delivered in Washington, D.C., on 4. The glossary (Annex 2) defines terms related to institu- October 6, 1998, emphasized the essentials of good governance tional development. In addition, box 3.1 illustrates the general and the need to specify the regulatory and institutional funda- perspective taken by analysts who focus on institutional issues. mentals essential to a workable market economy. 5. In April 1998, finance ministers and central bank gover- 9. Institutional development impact measures the extent to nors from a number of large economies formed three working which a project has improved the ability of an agency or country groups-on transparency and accountability, on strengthening to make effective use of its human and financial resources. This financial systems, and on international financial crises-to dis- impact, possible even in the absence of explicit institutional cuss the policy issues raised by the financial crisis. In October development objectives, includes both traditional impacts, 1998, each working group presented its findings and recommen- through new or improved organizations, and the impact projects dations in a separate report. have on the rules of the game governing public and private sector behavior. Chapter 2 10. Sustainability is defined as the likelihood, at the time of 1. The terms "operation" and "project" refer to both IDA evaluation, that a project will maintain its results in the future. To and IBRD lending and are used interchangeably. Unless otherwise judge the sustainability of an operation, evaluators take into stated, all time period references relate to the fiscal year in which account country conditions, government policies, and other con- evaluated operations exited the portfolio. ditions specific to the operation, such as availability of funds for 2. The results for fiscal 1998 are based on a sample of 114 operation and maintenance. Basic factors behind the original evaluated operations-40 percent of the 283 exiting operations- project appraisal are also considered-such as technical, finan- for which regional staff have prepared completion reports. A cial, and economic viability; susceptibility to external shocks; and complete set of data by region and sector will be provided on the social, environmental, and governance environment. OED's Web site for reference and follow-up. 11. Looking at shifts in country macroeconomic environ- 3. The likely bias arises from problems of sample representa- ments, the fiscal 1997-98 evaluated exits have slightly more tion for the preliminary fiscal 1998 results. According to project projects in the poorly performing country group than do those data from the Quality Assurance Group (QAG) at exit, the cohort exiting in fiscal 1990-96 (12 percent, compared with 9 percent), evaluated so far includes far fewer problem projects (16 percent) with a slight decrease in projects in the lower-performing group than the projects remaining to be evaluated (26 percent). Of the (from 17 percent to 14 percent). The three other groupings of evaluated cohort, those rated as problem projects by QAG at exit high, lower-medium, and upper-medium performing countries were only 18 percent satisfactory, while the rest were more than show even less variation when analyzed for fiscal 1990-96 and 90 percent satisfactory. Assuming the same relationship between fiscal 1997-98. While slight, the movement toward poorer- QAG and OED assessments for those fiscal 1998 exits yet to be performing economic environments makes it unlikely that coun- reviewed by OED yields an estimate of 73 percent satisfactory for try selectivity affected the improved performance results of fiscal the part of the cohort not reviewed here. Combining this group 1997-98. with the exits reviewed implies an overall 76 percent satisfactory 12. Borrower performance is assessed for three project rating for the entire group of fiscal 1998 exits. processes-preparation, implementation, and compliance. Imple- 4. Data from the Annual Report on Portfolio Performance, mentation is the broadest of these three project measures, with Fiscal Year 1998 (ARPP), prepared by the Quality Assurance dimensions under the government's control-such as broad Group. project commitment, appointment of key staff, and counterpart 5. Disbursements are measured in real terms, deflated to funding-as well as implementing agency factors such as man- fiscal 1996 US dollars. agement, staffing, cost changes, and beneficiary participation. . 13. Bank performance is assessed for three project pro- 6. The standard deviation of the regional shares of satisfac- 1. .an pefrac sassefrtrepoetpo 6. Te sandrd eviaionof he egioal hars o stfa- cesses-identification, appraisal, and supervision. Assessments of tory projects has dropped 40 percent, from 10 percentage pomnts ces -intfaio,prialadsurvin.Aesmtsf for fiscal 1990-96 to 6 percentage points for fiscal 1997-98. the Bank's performance during project identification include for fisca on06 toe 6ctiv pertagefolinfr fisl 1997-98.nlooking at the involvement of the government and beneficiary, 7. Data on the active portfolio under supervision are taken whether the project is consistent with the Bank's country assis- from the Annual Report on Port folio Performance, Fiscal Year tance strategy, and whether there is a grounding in economic and 1998 (ARPP), prepared by the Quality Assurance Group. The sector work. The quality of appraisal includes the following basic measure of performance used in the ARPP is the number of dimensions: technical and financial analysis, cost benefit analysis, projects at risk of not achieving their development objectives. institutional capacity analysis, and environmental and social Projects at risk consist of actual and potential problem projects. analysis. Dimensions taken into account for assessing project Actual problem projects are those for which implementation supervision include progress reporting, identification/assessment progress is unsatisfactory or development objectives are not likely of problems, use of performance indicators, advice to the imple- 51 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS menting agency, and flexibility in suggesting/approving modifi- aspect of the changing pattern. But even in the 45 countries cations. considered by Deininger and Squire, an updating of their data and 14. Findings that supervision improved are consistent with a more short-term focus shows that about five times as many the latest Quality Assurance Group self-evaluation assessments of countries (24) have an increasing trend as have a decreasing trend supervision in Supervision Quality in FY98: A QAG Assessment (5 countries). Of course, given the limited amount of observa- (RSA2). tions, our classification was necessarily based on a heuristic approach. Chapter 3 1. Up until August 1998. Chapter 4 2. OEDCR staff were asked to rate country assistance strate- 1. See Clague (1997) and Knack and Keefer (1995). For gies (quality at entry, implementation, outcome, sustainability, and literature on corruption, see Bardhan (1997), World Bank institutional development) from their evaluations in CAEs. Where (1997b), Mauro (1995), and Rose-Ackerman (1998). the evaluator thought appropriate, the country strategy was rated 2. To consider the effects that country characteristics might separately for different periods. For the 17 countries reviewed, CAEs have on Bank performance, we examined performance in a wide identify 25 time slices-that is, different and distinct strategies for range of countries. For example, we grouped countries by such the country within the time period covered by the evaluations. characteristics or whether they were post-conflict societies, tran- Project ratings corresponding to the same countries and periods show sition economies, Sub-Saharan Africa economies, had high or low that project outcome was satisfactory 68 percent of the time. The levels of corruption, good or weak bureaucracies, and persistently latter figure is similar to the average project performance during the poor policy environments. period of the CAE analysis. 3. Derived from background work on country assistance Chapter 5 strategies, available electronically at http://www.worldbank.org/ 1. Euromoney's semiannual country risk ratings range from 0 html/oed. (most risky) to 100 (least risky). Each rating is calculated as a 4. The evaluation form follows the style of OED's project weighted average of nine categories of indicators representing information form, and the methodology is that of OED's method- analytical, credit, and market indicators. The categories are ology for evaluating completed lending operations. See OED economic data (25 percent weighting), political risk (25 percent), 1997b. debt indicators (10 percent), rescheduled debt or debt in default 5. In addition to comparing the perspectives on growth and (10 percent), sovereign credit ratings (10 percent), access to bank poverty reduction in CAEs with those of empirical models, we finance (5 percent), access to short-term finance (5 percent), compared how well these judgments on country strategy perfor- access to international bond and syndicated loan markets (5 mance served as a predictor of subsequent Bank project perfor- percent), and access to and discount on forfeiting (5 percent). mance. While 25 observations limited the degrees of freedom, 2. The change in risk rating between the year of approval these judgments nevertheless serve as stronger predictors of and the year of exit has been used to classify countries. Improve- project performance than do macroeconomic policy measures. ment refers to betterment in risk rating of more than 5 percent; 6. The data on institutional quality refer to country ratings performance as expected refers to countries whose rating changed for bureaucratic quality, rule of law, and corruption as defined by by less than 5 percent; deteriorating economies are those with an the International Country Risk Guide. An average of the three increase in risk between the year of approval and the year of exit. ratings of greater than 4 on a 1-6 scale is considered satisfactory. 3. The estimated relationship between country risk and A rating of greater than 3.33 and less than 4.0 is considered portfolio performance during 1997-98 is used to forecast ratings marginally satisfactory. for projects exiting in 1999. The forecast of a 5 percentage point 7. This rating is based on a more limited sample of low- and deterioration in fiscal 1998 satisfactory projects is used as a middle-income countries. The policy index has been calculated on baseline scenario, assuming that country risk ratings in 1999 an annual basis for 1995, as in Burnside and Dollar (1997). would be the same as in 1998. In a more pessimistic scenario, 8. Using comparable data on income distribution for 45 country risk ratings would deteriorate by 6 percentage points. countries, Deininger and Squire (1996) show that over the 30- 4. World Bank 1990, box 3.3, p. 47. year period up to the early 1990s, there was no trend in within- 5. See Ravillion (1994a, 1994b, 1996) and Pollak (1991). country income inequality. In 29 countries the Gini coefficient-a 6. The estimated number of poor people in 2015, given the measure of income distribution-remained virtually constant, in 8 expected increase in population and assuming a 50 percent it increased, and in another 8 it decreased. In the 16 countries with reduction in poverty, is 900 million. This would call for lifting an increasing or decreasing trend, in 12 the change was small. almost 1 billion people out of poverty over the next 20 years, or Our shorter-term perspective, focusing on changes over five or about 50 million people a year. more years, suggests that in recent years a very different pattern has emerged. The trends in transition economies are an important 52 BIBLIOGRAPHY Bardhan, Pranab. 1997. "Corruption and Develop- Commander, Simon, and others. 1997. "The Causes ment: A Review of the Issues." Journal of Economic and Consequences of Government for Growth and Literature 35(3): 1320-46. Well-being." Background paper for World Develop- Barro, Robert J. 1995. "Technological Diffusion, ment Report 1997. World Bank, Washington, D.C. Convergence, and Growth." NBER Working Paper Deininger, Klaus, and Lyn Squire. 1996. "A New Data 5151. National Bureau for Economic Research, Set for Measuring Income Inequality." The World Cambridge, Mass. Bank Economic Review 10(3): 565-91. Barro, Robert J., and Xavier Sala-i-Martin. 1995. Demery, Lionel, and Michael Walton. 1998. "Are Economic Growth. New York: McGraw-Hill. Poverty and Social Goals for the 21st Century Bell, Clive, and Robert Rich. 1994. "Rural Poverty Attainable?" World Bank, Poverty Reduction and and Agricultural Performance in Post-Independence Economic Management Network, Washington, D.C. India." Oxford Bulletin of Economics and Statistics Easterlin, Richard A. 1998. "How Beneficent is the 56(2): 111-33. Market? A Look at the Modern History of Mortal- Berg, Elliot J. 1993. Rethinking Technical Coopera- ity." University of Southern California. Mimeo. tion: Reforms for Capacity Building in Africa. New Easterly, William, and Stanley Fischer. 1995. "The York: United Nations Development Programme. Soviet Economic Decline." The World Bank Eco- Bezanson, Keith. 1998. "Some Implications of the East nomic Review (9)3. Asian Crisis for Development Effectiveness-Per- EBRD (European Bank for Reconstruction and Devel- sonal Perspectives Prepared at the Request of the opment). 1997. Transition Report 1997: Enterprise Operations Evaluation Department of the World Performance and Growth. London. Bank." Institute of Development Studies, Sussex Edwards, Sebastian. 1997. "Trade Policy, Growth, and University, Brighton. Income Distribution." American Economic Review Birdsall, Nancy. 1998. "Life is Unfair: Inequality in Papers and Proceedings 87(2): 205-10. the World." Foreign Policy 111(summer): 76-93. Fischer, Bernhard, and Helmut Reisen. 1993. Burnside, Craig, and David Dollar. 1997. Aid, Poli- Liberalising Capital Flows in Developing Countries: cies and Growth. Policy Research Working Paper Pitfalls, Prerequisites and Perspectives. Washington, 1777. Washington, D.C.: World Bank. D.C.: Organization for Economic Cooperation and Caprio, Gerard Jr. 1996. "Banking on Financial Development. Reform? A Case of Sensitive Dependence on Initial Gallup, John Luke, and Jeffrey Sachs, with Andrew D. Conditions." In Gerard Caprio Jr., Izak Atiyas, and Mellinger. 1998. "Geography and Economic James A. Hanson, eds., Financial Reform: Theory Growth." Paper prepared for the Annual World Bank and Experience. Cambridge, Mass.: Cambridge Conference on Development Economics, April 20- University Press. 21, Washington, D.C. . 1998. Banking on Crises: Expensive Lessons Gwin, Catherine, and Joan M. Nelson. 1997. Perspec- from Recent Financial Crises. Washington, D.C.: tives on Aid and Development. Washington, D.C.: World Bank. Overseas Development Council. Caprio, Gerard Jr., and Daniela Klingebiel. 1996. Hadjimichael, Michael T., and others. 1995. Sub- Bank Insolvencies: Cross-Country Experience. Saharan Africa: Growth, Savings, and Investment, Policy Research Working Paper 1620. Washington, 1986-93. IMF Occasional Paper 118. Washington, D.C.: World Bank. D.C. Cassen, Robert, and associates. 1994. Does Aid Helleiner, Gerald, and others. 1995. Report of the Work? Oxford: Clarendon. Group of Independent Advisers on Development Claessens, Stijn, and Thomas Glaessner. 1997. Are Cooperation Issues between Tanzania and Its Aid Financial Sector Weaknesses Undermining the East Donors. Copenhagen: Royal Danish Ministry of Asian Miracle? Washington, D.C.: World Bank. Foreign Affairs. Clague, Christopher. 1997. Institutions and Economic IDS (Institute of Development Studies). 1998. Papers Development. Baltimore, M.D.: The Johns Hopkins prepared for the workshop on the East Asian Crisis, University Press. July 13-14, Sussex University, Brighton. 53 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS IMF (International Monetary Fund). 1995. "Transfers, Mathieu, Nicolas. 1998. Financial Sector Reform. Social Safety Nets, and Economic Growth." IMF Operations Evaluation Department Study Series. Staff Papers 44(1): 81-102. Washington, D.C.: World Bank. Ingram, Gregory. 1992. Social Indicators and Produc- Mauro, Paulo. 1995. "Corruption and Growth." Quar- tivity Convergence in Developing Countries. Policy terly Journal of Economics 110(3): 681-712. Research Working Paper 894. Washington, D.C.: Milanovic, Branko. 1998. Income, Inequality, and World Bank. Poverty during the Transition from Planned to Ito, Takatoshi. 1998. "What Can Developing Countries Market Economy. A Regional and Sectoral Study. Learn from East Asian Economic Growth?" In Boris Washington, D.C.: World Bank. Pleskovic and Joseph E. Stiglitz, eds., Annual World Mosley, Paul, Jane Harrigan, and John Toye. 1991. Bank Conference on Development Economics 1997. Aid and Power: The World Bank and Policy-based Washington, D.C.: World Bank. Lending. London: Routledge. Johnson, John H., and Sulaiman S. Wasty. 1993. OECD (Organisation for Economic Cooperation and Borrower Ownership of Adjustment Programs and Development). 1996. "Shaping the 21st Century: The the Political Economy of Reform. World Bank Contribution of Development Co-operation." Paris: Discussion Paper 199. Washington, D.C. Development Assistance Committee. Johnston, Timothy, and Nicolas Van de Walle. 1996. OED (Operations Evaluation Department). 1990. Improving Aid to Africa. Washington, D.C.: Over- Chile: Structural Adjustment Loans. Project Perfor- seas Development Council. mance Audit Report 08851. Washington, D.C.: Keefer, Philip, and Stephen Knack. 1997. "Why Don't World Bank. Poor Countries Catch Up? A Cross-National Test of _. 1995. The Social Impact of Adjustment Opera- an Institutional Explanation." Economic Inquiry tions: An Overview. Report 14776. Washington, 35(July): 590-602. D.C.: World Bank. Knack, Stephen, and Philip Keefer. 1995. "Institutions _. 1996a. Argentina Country Assistance Review. and Economic Performance: Cross-Country Tests Report 15844. Washington, D.C.: World Bank. Using Alternative Institutional Measures." Econom- _. 1996b. Ghana Country Assistance Review. ics and Politics 7(3): 207-27. Report 14547. Washington, D.C.: World Bank. Krugman, Paul. 1994. "The Fall and Rise of Develop- _. 1996c. "Technical Assistance." Lessons and ment Economics." In Lloyd Rodwin and Donald A. Practices 7. Washington, D.C.: World Bank. Sch6n, eds., Rethinking the Development Experi- . 1996d. The World Bank's Experience with ence. Harrisonburg, Va.: R.R. Donnelly and Sons. Large Dams: A Preliminary Review of Impacts. _. 1998. "The Confidence Game." New Republic Report 15815. Washington, D.C.: World Bank. 219: 23-25. -. 1996e. Social Dimensions of Adjustment. Levine, Ross. 1997. "Financial Development and Eco- Washington, D.C.: World Bank. nomic Growth: Views and Agenda." Journal of -. 1996f. Zambia Country Assistance Review. Economic Literature 35(2): 688-726. Report 15675. Washington, D.C.: World Bank. Levine, Ross, and David Renelt. 1991. "A Sensitivity . 1997a. C6te d'Ivoire Country Assistance Re- Analysis of Cross-Country Growth Regressions." view. Report 17015. Washington, D.C.: World Bank. American Economic Review 82(4): 942-63. _. 1997b. "Evaluating Development Operations: Levine, Ross, and Sara Zervos. 1993. "Looking at the Methods for Judging Outcomes and Impacts." Les- Facts: What We Know about Policy and Growth from sons and Practices 10. Washington, D.C.: World Cross-Country Analysis." Policy Research Working Bank. Paper 1115. Washington, D.C.: World Bank. _. 1997c. Kingdom of Morocco Country Assis- Lipton, Michael, and others. 1998. Successes in Anti- tance Review. Report 16326. Washington, D.C.: poverty. Geneva: International Labour Office. World Bank. Maddison, Angus. 1997. "The Nature and Functioning . 1997d. Mozambique Country Assistance Re- of European Capitalism: A Historical and Compara- view. Report 17209. Washington, D.C.: World Bank. tive Perspective." Banca Nazionale del Lavoro Quarterly Review 50(203). 54 Bibliography 1997e. Poland Country Assistance Review. Ravallion, Martin. 1996. "Issues in Measuring and Report 16495. Washington, D.C.: World Bank. Modeling Poverty." The Economic Journal (Septem- 1997f. "A Success and Challenge: AGETIP in ber). Blackwell. Senegal." Operations Evaluation Department Pr6cis Ravallion, Martin, and Shaohua Chen. 1997. "What 148. Washington, D.C.: World Bank. Can New Survey Data Tell Us about Recent Changes 1998a. Albania Country Assistance Review. in Distribution and Poverty?" The World Bank Report No. 18030. Washington, D.C.: World Bank. Economic Review 11(2): 357-81. . 1998b. 1997 Annual Review of Development Ravallion, Martin, and Gaurav Dart. 1994. Growth Effectiveness. Washington, D.C.: World Bank. and Poverty in India. Washington, D.C.: World . 1998c. Annual Report on Portfolio Perfor- Bank. mance Fiscal Year 1998. Washington, D.C.: World Reisen, Helmut. 1998. "Domestic Causes of Currency Bank. Crises: Policy Lessons for Crisis Avoidance." Paper . 1998d. Bangladesh Country Assistance Re- presented at the Institute of Development Studies East view. Report 17455. Washington, D.C.: World Bank. Asian Crisis Workshop, Sussex University, July 13- . 1998e. Bolivia Country Assistance Review. 14, Brighton. Report 17957. Washington, D.C.: World Bank. Rodrik, Dani. 1996. "Understanding Economic Policy . 1998f. Financial Sector Reform: A Review of Reform." Journal of Economic Literature 34 World Bank Assistance. Report 17454. Washington, (March): 9-41. D.C.: World Bank. Rose-Ackerman, Susan. 1998. "Corruption and Devel- . 1998g. The Impact of Public Expenditure opment." In Boris Pleskovic and Joseph E. Stiglitz, Reviews. Washington, D.C.: World Bank. eds., Annual World Bank Conference on Develop- . 1998h. NGOs in Bank-Supported Projects. An ment Economics 1997. Washington, D.C.: World OED Review. Report 18399. Washington, D.C.: Bank. World Bank. Sachs, Jeffrey D., and Andrew Warner. 1995. "Eco- - . 1998i. Philippines Country Assistance Review. nomic Reform and the Process of Global Integra- Report 17417. Washington, D.C.: World Bank. tion." Brookings Papers on Economic Activity. 1998j. Process Review of World Bank Grant Washington, D.C. Programs. Report 18317. Washington, D.C.: World Sala-i-Martin, Xavier. 1997. "I Just Ran Four Million Bank. Regressions." NBER Working Paper Series 6252. . 1998k. The World Bank's Experience with Cambridge, Mass.: National Bureau for Economic Post-Conflict Reconstruction. Report 17769. Wash- Research. ington, D.C.: World Bank. Sen, Amartya. 1976. "Poverty: An Ordinal Approach Picciotto, Robert, and Eduardo Wiesner, eds. 1998. to Measurement." Econometrica 46: 437-46. Evaluation and Development: The Institutional Di- Squire, Lyn. 1993. "Fighting Poverty." American Eco- mension. New Brunswick, N.J.: Transaction. nomic Review 83(2): 377-92. Pollak, R. 1991. "Welfare Comparisons and Situation UNDP (United Nations Development Programme). Comparisons." Journal of Econometrics 50: 31-48. 1998. Poverty in Transition? New York: Regional Psacharopoulos, George, and others. 1992. Poverty Bureau for Europe and the Commonwealth of Inde- and Income Distribution in Latin America: The pendent States. Story of the 1980s. Latin America and the Caribbean Working Group on International Financial Crises. Technical Department Report 27. Washington, D.C.: 1998. Report prepared by representatives of Argen- World Bank. tina, Australia, Belgium, Brazil, Canada, France, Ravallion, Martin. 1994a. "Measuring Social Welfare Germany, Hong Kong SAR, Italy, Japan, Korea, with and without Poverty Lines." American Eco- Mexico, the Netherlands, Singapore, South Africa, nomic Review, 84 (2): 359-64. the United Kingdom, and the United States. Meetings Ravallion, Martin. 1994b. "Poverty Comparisons." held in Washington, D.C., April 1998. Fundamentals of Pure and Applied Economics, vol. World Bank. 1990. World Development Report 1990: 56. Chur, Switzerland: Harwood. Poverty. New York: Oxford University Press. 55 1998 ANNUAL REVIEW OF DEVELOPMENT EFFECTIVENESS 1996a. "Partnership for Capacity Building in Africa." A report of the Working Party on the Impact of Bank Policies, Instruments, and Operational Prac- tices on Capacity Building in Africa. Washington, D.C. 1996b. World Development Report 1996: From Plan to Market. New York: Oxford University Press. 1997a. Annual Report 1997. Washington, D.C. 1997b. Helping Countries Combat Corruption: The Role of the World Bank. Washington, D.C. _. 1997c. "The Strategic Compact: Renewing the Bank's Effectiveness to Fight Poverty." Washington, D.C. _. 1997d. World Development Report 1997. Washington, D.C. _. 1998a. Assessing Aid. A Policy Research Report. Washington, D.C. _. 1998b. "Country Assistance Strategies: Retro- spective and Outlook." Managing Directors Opera- tions Policy and Strategy Group, Washington, D.C. _. 1998c. East Asia: The Road to Recovery. Washington, D.C. _. 1998d. World Development Indicators 1998. Washington, D.C. _. 1998e. World Development Report 1998/99: Knowledge for Development. New York: Oxford University Press. 1998f. "Supervision Quality in FY98: A QAG Assessment (RSA2)." Quality Assurance Group, Washington, D.C. 56 OPERATIONS EVALUATION DEPARTMENT PUBLICATIONS The Operations Evaluation Department (OED), an in- The World Bank InfoShop serves walk-in customers dependent evaluation unit reporting to the World Bank's only. The InfoShop is located at: Executive Directors, rates the development impact and performance of all the Bank's completed lending opera- 701 18th Street, NW tions. Results and recommendations are reported to the Washington, DC 20433, USA Executive Directors and fed back into the design and implementation of new policies and projects. In addition to All other customers must place their orders through the individual operations and country assistance programs, their local distributors. OED evaluates the Bank's policies and processes. Operations evaluation studies, World Bank discus- Ordering by E-Mail sion papers, and all other documents are available from If you have an established account with the World Bank, the World Bank InfoShop. you may transmit your order by electronic mail on the Summaries of studies and the full text of the Precis Internet to: books@worldbank.org Please include your and Lessons & Practices can be read on the Internet at account number, billing and shipping addresses, the title http://www.worldbank.org/html/oed/index.htm and order number, quantity, and unit price for each item. How To Order OED Publications Documents listed with a stock number and price code may be obtained through the World Bank's mail order service or from its InfoShop in downtown Washington, DC. For information on all other documents, contact the World Bank InfoShop. Ordering World Bank Publications Customers in the United States and in territories not served by any of the Bank's publication distributors may send publication orders to: The World Bank P.O. Box 960 Herndon, VA 20172-0960 Fax: (703) 661-1501 Telephone: (703) 661-1580 The address for the World Bank publication database on the internet is: http://www.worldbank.org (select publi- cations/project info). E-mail: pic@worldbank.org Fax number: (202) 522-1500 Telephone number: (202) 458-5454 Distributors of World Bank Chinese Corporation for Promotion of FINLAND IRAN Humanities Akateeminen Kirjakauppa Ketab Sara Co. Publishers Publications 52 You Fang Hu Tong, P.O. Box 128 Khaled Eslamboli Ave., 6th Street Prices and credit terms vary Xuan Nei Da Jie FIN-00101 Helsinki Delafrooz Alley No 8 from country to country. Beijing Tel: (358 0) 121 4418 P.O. Box 15745-733 C t y local distributor Tel: (86 10) 660 72 494 Fax: (358 0) 121-4435 Tehran 15117 onsu your Fax: (86 10) 660 72 494 E-mail. akatilaus@stockmannfi Tel: (98 21) 8717819: 8716104 before placing an order. ERL: http://www.akateeminen.com Fax. (98 21) 8712479 COLOMBIA E-mail: ketab-sara@neda.net.ir ARGENTINA Infoenlace Ltda. FRANCE World Publications SA Carrera 6 No. 51-21 Editions Eska; DBJ Kowkab Pubhlishers Av. Cordoba 1877 Apartado Aereo 34270 48, me Gay Lussac P.O. Box 19575-511 1120 Ciudad de Buenos Aires Santafe de Bogoti, D.C. 75005 Paris Tehran Tel: (54 11) 4815-8156 Tel: (57 1) 285-2798 Tel (33-1) 55-42-73-08 Tel: (98 21) 258-3723 Fax: (54 11) 4815-8156 Fax: (57 1) 285-2798 Fax (33-1) 43-29-91-67 Fax: (98 21) 258-3723 E-mail: wpbooks@infovia.com.ar COTE D'IVOIRE GERMANY IRELAND AUSTRALIA, FIJI, PAPUA NEW Center d'Edition et de Diffusion UNO-Verlag Government Supplies Agency GUINEA, SOLOMON ISLANDS, Africames (CEDA) Poppelsdorfer Allee 55 Ofig an tSolAthair VANUATU, AND SAMOA 04 B.P. 541 53115 Bonn 4-5 Harcourt Road D.A. Information Services Abidjan 04 Tel: (49 228) 949020 Dublin 2 648 Whitehorse Road Tel: (225) 24 6510; 24 6511 Fax: (49 228) 217492 Tel: (353 1) 661-3111 Mitcham 3132 Fax: (225) 25 0567 URL: http://www.uno-verlag.de Fax: (353 1) 475-2670 Victoria E-mail: unoverlag@aol.com Tel: (61) 3 9210 7777 CYPRUS ISRAEL Fax: (61) 3 9210 7788 Center for Applied Research GHANA Yozmot Literature Ltd. E-mail: service@dadirect.com.au Cyprus College Epp Books Services RO. Box 56055 URL: http://wsvw.dadirect.com.au 6, Diogenes Street, Engomi P.O Box 44 3 Yohanan Hasandlar Street P.O. Box 2006 TUC Tel Aviv 61560 AUSTRIA Nicosia Accra Tel: (972 3) 5285-397 Gerold and Co. Tel: (357 2) 59-0730 Tel: 223 21 778843 Fax: (972 3) 5285-397 Weihburggasse 26 Fax: (357 2) 66-2051 Fax: 223 21 779099 A-1011 Wien R.O.Y International Tel: (43 1) 512-47-31-0 CZECH REPUBLIC GREECE PO Box 13056 Fax: (43 1) 512-47-31-29 USIS, NIS Prodejna Papasotiriou S.A. Tel Aviv 61130 URL. http://www.gerold.co/at.online Havelkova 22 35, Stournara St. Tel: (972 3) 649 9469 130 00 Prague 3 106 82 Athens Fax: (972 3) 648 6039 BANGLADESH Tel: (420 2) 2423 1486 Tel: (30 1) 364-1826 E-mail: royil@netvision.net.il Micro Industries Development Fax: (420 2) 2423 1114 Fax: (30 1) 364-8254 Assistance Society (MIDAS) URL http://www.nis.cz/ Palestinian Authority/Middle East House 5, Road 16 HAITI Index Information Services Dhanmondi R/Area DENMARK Culture Diffusion P O.B. 19502 Jerusalem Dhaka 1209 SamfundsLitteratur 5, Rue Capois Tel: (972 2) 6271219 Tel: (880 2) 326427 Rosenoerns Alit 11 C.P. 257 Fax: (972 2) 6271634 Fax: (880 2) 811188 DK-1970 Frederiksberg C Port-au-Prince Tel (45 35) 351942 Tel: (509) 23 9260 ITALY, LIBERIA BELGIUM Fax. (45 35) 357822 Fax: (509) 23 4858 Licosa Commissionara Sansoni SPA Jean De Lannoy URL: http //www sl.chs.dk Via Duca Di Calabria, 1/1 Av. du Roi 202 HONG KONG, CHINA; MACAO Casella Postale 552 1060 Brussels ECUADOR Asia 2000 Ltd. 50125 Firenze Tel: (32 2) 538-5169 Libri Mundi Sales & Circulation Department Tel: (39 55) 645-415 Fax: (32 2) 538-0841 Libreria Internacional 302 Seabird House Fax: (39 55) 641-257 P.O. Box 17-01-3029 22-28 Wyndham Street, Central E-mail- licosa@ftbcc.it BRAZIL Juan Leon Mera 851 Hong Kong, China URL: http://wvw.fthcc it/licosa Publicac6es Tecnicas Quito Tel: (852) 2530-1409 Internacionais Ltda. Tel: (593 2) 521-606; Fax: (852) 2526-1107 JAMAICA Rua Peixoto Gomide, 209 (593 2) 544-185 E-mail. sales@asia2000.com.hk Ian Randle Publishers Ltd. 01409 Sao Paulo, SP. Fax (593 2) 504-209 ERL: http //www asia2000.com.hk 206 Old Hope Road, Kingston 6 Tel: (55 11) 259-6644 E-mail: librimul@librimundt.com.ec Tel: 876-927-2085 Fax: (55 11) 258-6990 E-mail: librimu2@libnmundi.com.ec HUNGARY Fax: 876-977-0243 E-mail: postmaster@pti.uol.br Euro Info Service E-mail: irpl@colis com URL: http://w-w.uol.br CODEU Margitszgeti Europa Haz Ruiz de Castilla 763, Edif. Expocolor H-1138 Budapest JAPAN CANADA Primer piso, Of. #2 1el: (36 1) 350 80 24, 350 80 25 Eastern Book Service Renouf Publishing Co. Ltd. Quito Fax: (36 1) 350 90 32 3-13 Hongo 3-chome, Bunkyo-ku 5369 Canotek Road Tel/Fax: (593 2) 507-383; 253-091 E-mail: euroinfo@mail matav.hu Tokyo 113 Ottawa, Ontario K1J 9J3 E-mail: codeu@impsat.net.ec Tel: (81 3) 3818-0861 Tel: (613) 745-2665 INDIA Fax: (81 3) 3818-0864 Fax: (613) 745-7660 EGYPT, ARAB REPUBLIC OF Allied Publishers Ltd. E-mail: orders@svt-ebs.co jp E-mail: order.dept@renoufbooks.com Al Ahram Distribution Agency 751 Mount Road URL:http://www.bekkoame or.jp/-svt-ebs URL: http.// www.renoufbooks.com Al Galaa Street Madras - 600 002 Cairo Tel: (91 44) 852-3938 KENYA CHINA Tel: (20 2) 578-6083 Fax: (91 44) 852-0649 Africa Book Service (E.A.) Ltd. China Financial & Economic Fax: (20 2) 578-6833 Quaran House, Mfangano Street Publishing House INDONESIA P.O. Box 45245 8, Da Fo Si Dong Jie The Middle East Observer Pt Indira Limited Nairobi Beijng 41, Sherif Street Jalan Borobudur 20 Tel- (254 2) 223 641 Tel (86 10) 6401-7365 Cairo P.O. Box 181 Fax: (254 2) 330 272 Fax: (86 10) 6401-7365 Tel. (20 2) 393-9732 Jakarta 10320 Fax (20 2) 393-9732 Tel: (62 21) 390-4290 China Book Import Centre Fax: (62 21) 390-4289 PD Box 2825 Beijing Legacy Books NEW ZEALAND ROMANIA SWITZERLAND Loita House EBSCO NZ Ltd. Compani De Librarii Bucuresti S.A. Librairie Payor Service Institutionnel Mezzanine 1 Private Mail Bag 99914 Str. Lipscani no. 26, sector 3 C(tm)tes-de-Montbenon 30 P.O. Box 68077 New Market Bucharest 1002 Lausanne Nairobi Auckland Tel: (40 1) 313 9645 Tel: (41 21) 341-3229 Tel: (254) 2-330853, 221426 Tel: (64 9) 524-8119 Fax: (40 1) 312 4000 Fax: (41 21) 341-3235 Fax: (254) 2-330854, 561654 Fax: (64 9) 524-8067 E-mail: Legacy@form-net.com RUSSIAN FEDERATION ADECO Van Diermen Oasis Official Isdatelstvo EditionsTechniques KOREA, REPUBLIC OF P.O. Box 3627 9a, Kolpachniy Pereulok Ch. de Lacuez 41 Dayang Books Trading Co. Wellington Moscow 101831 CH1807 Blonay International Division Tel: (64 4) 499 1551 Tel: (7 095) 917 87 49 Tel: (41 21) 943 2673 783-20, Pangba Bon-Dong, Socho-ku Fax: (64 4) 499 1972 Fax. (7 095) 917 92 59 Fax: (41 21) 943 3605 Seoul E-mail: oasis@actrix.gen.nz Tel (82 2) 536-9555 URL: http://www.oasisbooks.co.nz/ SINGAPORE; TAIWAN, CHINA THAILAND Fax (82 2) 536-0025 MYANMAR; BRUNEI Central Books Distribution E-mail. seamap@chollian.net NIGERIA Hemisphere Publication Services 306 Silom Road University Press Limited 41 Kallang Pudding Road z04-03 Bangkok 10500 Eulyoo Publishing Co., Ltd. Three Crowns Building Jericho Golden Wheel Building Tel: (66 2) 2336930-9 46-1, Susong-Dong, Jongro-Gu Private Mail Bag 5095 Singapore 349316 Fax: (66 2) 237-8321 Seoul Ibadan Tel: (65) 741-5166 Tel: (82 2) 734-3515 Tel (234 22) 41-1356 Fax: (65) 742-9356 TRINIDAD & TOBAGO Fax: (82 2) 732-9154 Fax: (234 22) 41-2056 E-mail: ashgate@asianconnect.com AND THE CARRIBBEAN Systematics Studies Ltd. LEBANON PAKISTAN SLOVENIA St. Augustine Shopping Center Librairie du Liban Mirza Book Agency Gospodarski vestnik Publishing Group Eastern Main Road, St. Augustine P.O. Box 11-9232 65, Shahrah-e-Quaid-e-Azam Dunajska cesta 5 Trinidad & Tobago, West Indies Beirut Lahore 54000 1000 Ljubljana Tel: (868) 645-8466 Tel: (961 9) 217 944 Tel (92 42) 735 3601 Tel: (386 61) 133 83 47: 132 12 30 Fax: (868) 645-8467 Fax: (961 9) 217 434 Fax (92 42) 576 3714 Fax: (386 61) 133 80 30 E-mail: tobe@trinidad.net E-mail: hsayegh@librairie-du- E-mail: repansekj@gvestnik.si liban.com.1b Oxford University Press UGANDA URL: http://www.librairie-du- 5 Bangalore Town SOUTH AFRICA, BOTSWANA Gustro Ltd. liban.com.1b Sharae Faisal For single titles: PO Box 9997, Madhvam Building PO Box 13033 Oxford University Press Southern Plot 16/4 Jinja Rd. MAIAYSIA Karachi-75350 Africa Kampala University of Malaya Cooperative Tel (92 21) 446307 Vasco Boulevard, Goodwood Tel: (256 41) 251 467 Bookshop, Limited Fax: (92 21) 4547640 P.O. Box 12119, N1 City 7463 Fax: (256 41) 251 468 P.O. Box 1127 E-mail: ouppak@TheOffice net Cape Town E-mail: gus@swiftuganda corm Jalan Pantai Baru Tel: (27 21) 595 4400 59700 Kuala Lumpur Pak Book Corporation Fax. (27 21) 595 4430 UNITED KINGDOM Tel: (60 3) 756-5000 Aziz Chambers 21, Queen's Road E-mail: oxford@oup.co.za Microinfo Ltd. Fax: (60 3) 755-4424 Lahore P.O. Box 3, Omega Park, Alton, E-mail: umkoop@tm.net.my Tel: (92 42) 636 3222; 636 0885 For subscrption orders- Hampshire GU34 2PG Fax: (92 42) 636 2328 International Subscription Service England MEXICO E-mail: pbc@brain.net.pk P.O. Box 41095 Tel: (44 1420) 86848 INFOTEC Craighall Fax: (44 1420) 89889 Av. San Fernando No. 37 PERU Johannesburg 2024 E-mail: wbank@microinfo.co uk Col. Toriello Guerra Editorial Desarrollo SA Tel: (27 11) 880-1448 URL: http //www.microinfo.co uk 14050 Mexico, D.F. Apartado 3824, Ica 242 OF. 106 Fax: (27 11) 880-6248 Tel: (52 5) 624-2800 Lima I E-mail: iss@is.co.za The Stationery Office Fax: (52 5) 624-2822 Tel (51 14) 285380 51 Nine Elms Lane E-mail: infotec@rtn.net.mx Fax (51 14) 286628 SPAIN London SW8 5DR URL: http //rtn.net.mx Mundi-Prensa Libros, S.A. Tel: (44 171) 873-8400 PHILIPPINES Castello 37 Fax: (44 171) 873-8242 Mundi-Prensa Mexico S.A. de C.V International Booksource Center Inc. 28001 Madrid URL: http://www.theso.co.ulk/ c/Rio Panuco, 141-Colonia 1127-A Antipolo St, Barangay, Tel- (34 91) 4 363700 Cuauhtemoc Venezuela Fax: (34 91) 5 753998 VENEZUELA 06500 Mexico, D.F. Makati City E-mail. libreria@mundiprensa.es Tecni-Ciencia Libros, S.A. Tel: (52 5) 533-5658 Tel: (63 2) 896 6501; 6505; 6507 URL: http.//wvw mundiprensa.com/ Centro Cuidad Comercial Tamanco Fax: (52 5) 514-6799 Fax: (63 2) 896 1741 Nivel C2, Caracas Mundi-Prensa Barcelona Tel: (58 2) 959 5547; 5035; 0016 NEPAL POLAND Consell de Cent, 391 Fax: (58 2) 959 5636 Everest Media International Services (R) International Publishing Service 08009 Barcelona Ltd. Ul. Piekna 31/37 Tel: (34 3) 488-3492 ZAMBIA GPO Box 5443 00-677 Warzawa Fax: (34 3) 487-7659 University Bookshop, University of Kathmandu Tel: (48 2) 628-6089 E-mail: barcelona@muidiprensa.es Zambia Tel: (977 1) 416 026 Fax: (48 2) 621-7255 Great East Road Campus Fax: (977 1) 224 431 E-mail: books%ips@ikp.atm.com.pl SRI LANKA, THE MALDIVES P.O. Box 32379 URL: http-//w w.ipscg.waw.pl/ips/ Lake House Bookshop Lusaka NETHERIANDS export/ 100, Sir Chittampalam Gardiner Tel: (260 1) 252 576 De Lindeboom/Internationale Mawatha, Colombo 2 Fax: (260 1) 253 952 Publicaties b.v.- PORTUGAL Tel: (94 1) 32105 P.O. Box 202, 7480 AE Haaksbergen Livraria Portugal Fax: (94 1) 432104 ZIMBABWE Tel: (31 53) 574-0004 Apartado 2681, Rua Do Carmo 70-74 E-mail: LHL@sri.tlanka.net Academic and Baobab Books (Pvt) Fax: (31 53) 572-9296 1200 Lisbon Ltd. E-mail: lindeboo@worldonline.nl Tel: (1) 347-4982 SWEDEN 4 Conald Road, Graniteside URL: http://www.worldonline.nl/ Fax: (1) 347-0264 Wennergren-Williams AB P.O. Box 567 -lindeboo P. 0. Box 1305 Harare S-171 25 Solna Tel: 263 4 755035 Tel: (46 8) 705-97-50 Fax: 263 4 781913 Fax: (46 8) 27-00-71 E-mail: mail@wwi.se OED SERIES 1999 Appr6ciation de 'efficacit6 du d6veloppement: L'valuation a la Banque mondiale et a la Socift6 financi6re internationale Determinar la eficacia de las actividades de desarrollo: La evaluaci6 en el Banco Mundial y la Corporaci6n Financiera Internacional Lessons of Fiscal Adjustment Monitoring & Evaluation Capacity Development in Africa Nongovernmental Organizations in World Bank-Supported Projects: A Review Philippines: From Crisis to Opportunity Public Sector Performance - Lessons from Urban Transport 1998 1997 Annual Review of Development Effectiveness Assessing Development Effectiveness: Evaluation in the World Bank and the International Finance Corporation Evaluation and Development: The Institutional Dimension (Transaction Publishers) Financial Sector Reform: A Review of World Bank Assistance India: The Dairy Revolution Rebuilding the Mozambique Economy: Assessment of a Development Partnership Public Sector Performance - The Critical Role of Evaluation The World Bank's Experience with Post-Conflict Reconstruction 1997 1995 Evaluation Results Agricultural Extension and Research: Achievements and Problems in National Systems Fiscal Management in Adjustment Lending Mainstreaming Gender in World Bank Lending: An Update Paddy Irrigation and Water Management in Southeast Asia Poland Country Assistance Review: Partnership in a Transition Economy Reforming Agriculture: The World Bank Goes to Market Zambia Country Assistance Review: Turning an Economy Around 1996 1994 Evaluation Results Industrial Restructuring: World Bank Experience, Future Challenges Lending for Electric Power in Sub-Saharan Africa Social Dimensions of Adjustment: World Bank Experience, 1980-93 The Aga Khan Support Program: A Third Evaluation THE WORLD BANK 1818 H Street, N.W. Washington, D.C. 20433, U.S.A. Telephone: (202) 477-1234 Facsimile: (202) 477-6391 Telex: MCI 64145 WORLDBANK MCI 248423 WORLDBANK World Wide Web: http://www.worldbank.org E-mail:books@worldbank.org 1 4474 5 780821 344743 ISBN 0-8213-4474-9