I~~~~~ o ©~~~~~~~~~~~~~ oo © NEBr,y<.l~~~~~~~~~~ THE WORLD BANK OPERATIONS EVALUATION DEPARTMENT a 25496 DEVELOPMENT COOPERATION AND PERFORMANCE EVALUATION: THE MONTERREY CHALLENGE Robert Picciotto Director-General, Operations Evaluation This working paper was prepared by the Director-General, Operations Evaluation, World Bank Group, as a contibutnon to a roundtable on June 5-6, 2002, in Washington, D.C., on "Better Measunng, Monitoring, and Managing for Development Results," an event sponsored by the Multilateral Development Banks in cooperation with the Development Assistance Committee of thc Organization for Economic Cooperation and Development. 2002 The World Bank Washington, D. C. OPERATIONS EVALUATION DEPARTMENT ENHIANCING DEVELOPMENTEFFECTIVENESS THROUGH EXCELLEENCE AND INDEPENDENCE INEVALUA TION The Operations Evaluation Department (OED) is an independent unit within the World Bank; it reports directyl to the Bank's Board of Executive Directors. OED assesses what works, and what does not; how a borrowerplans to run and maintain a project; and the lasting contribution of the Bank to a county's overall development. The goals of evaluation are to learn from experience, to provide an objective basisfor assessing the results of the Bank's work, and to provide accountabikiy in the achievement of its objectives. It also improves Bank work by identfying and disseminating the lessons learnedfrom experience and byframing recommendations drawn from evaluation findings. OED Working Papers are an informal series to disseminate the findings of work in progress to encourage the exchange of ideas about development effectiveness through evaluation. The findings, interpretations, and conclusions expressed here are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent. The World Bank cannot guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply on the part of the World Bank any judgment of the legal status of any territory or the endorsement or acceptance of such boundaries. Contact: Operations Evaluation Department Partnerships & Knowledge Programs (OEDPK) email: eline@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 http://www.worldbank.org/oed 2 Development Cooperation and Performance Evaluation: The Monterrey Challenge The United Nations Conference on Financingfor Development held in Monterrey, Mexico, in March 2002 was a watershed. It captured a new development paradigm and reflected a broad-based consensus about aid effectiveness. This note identifies the major challenges that develop- ment practitioners and evaluators must now face in order to adapt their structures, products, methods, and processes to the advent of a new authorizing environment. The Millennium Development Goals The Millennium Development Goals (MDGs) are grounded in the agreements and resolutions of world conferences organized by the United Nations. In September 2000, they were endorsed by all 189 Urnted Nations states. The means to achieve them were addressed at the Monterrey conference. The MDGs symbolize a focus on results. They enshrine poverty reduction as the overarching mission of development. Hunger eradication, empowerment of women, improvement of matemal and child health, prevention and cure of contagious diseases, and promotion of environmental sustainability represent complementary objectives that electorates m rich and poor countries alike can readily grasp (box 1). Unfortunately, they appear to be out of reach for many poor countries. Nevertheless, they should help make the efforts of the development community more coherent and effective and help enlist public opinion in the global fight against poverty. The Monterrey Challenge Broad agreement was reached at Monterrey regarding the basic elements of a newglobal partnership. It matches the adoption of improved policies and good governance1 in developing countries with the provision of increased aid and trading opportunities by developed countries. Given the diverse interests and contrasting visions of development that prevailed among conference partcipants, this was a substantial achievement I World Bank, Irorld Dewopment Report 1997. The State sn a Chang;ng WZorld (New York Oxford University Press, 1997). 3 Box 1. The Millennium Development Goals Tarects Ls Goal 1. Eradicate Target 1. Halve, between 1990 and 2015, the proportion of people whose extreme poverty and income is less than $1/day. hunger. Target 2. Halve, between 1990 and 2015, the proportion of people who suffer from hunger. Goal 2. Achieve Target 3. Ensure that, by 2015, children everywhere, boys and girls alike, universal primary will be able to complete a full course of primary schooling. education. Goal 3. Promote Target 4. Eliminate gender disparity in primary and secondary education, gender equality and preferably by 2005, and to all levels of education no later than 2015. empower women. Goal 4. Reduce child Target 5. Reduce by two-thirds, between 1990 and 2015, the under-five mortality. mortality rate. Goal 5. Improve Target 6. Reduce by three-quarters, between 1990 and 2015, the maternal maternal health. mortality ratio. Goal 6. Combat Target 7. Have halted by 2015 and begun to reverse the spread of HIV/AIDS, malaria, HIV/AIDS. and other diseases. Target 8. Have halted by 2015 and begun to reverse the incidence of malaria and other major diseases. Goal 7. Ensure Target 9. Integrate the principles of sustainable development into country environmental policies and programs and reverse the losses of environmental resources. sustainability. Target 10. Halve by 2015 the proporion of people without sustainable access to safe drinking water. Target 11. By 2020 to have achieved a significant improvement in the lives of at least 100 million slum dwellers. Goal 8. Develop a Target 12. Develop further an open, rule-based, predictable, non- Global Partnership discriminatory trading and financial system. for Development. Target 13. Address the special needs of the least-developed countries. Target 14. Address the special needs of landlocked countries and small island developing states. Target 15. Deal comprehensively with the debt problems of developing countries through national and international measures in order to make debt sustainable in the long term. Target 16. In cooperation with developing countries, develop and implement strategies for decent and productive work for youth. Target 17. In cooperation with pharmaceutical compariies, provide access to affordable essential drugs in developing countries. Target 18. In cooperation with the private sector, make available the benefits of new technologies, especially information and communications. Sourre: Milknnium Development Goa,& 2002, World Bank. 4 The U.N. Secretary General's report to the Preparatory Committee of the conference2 included 87 recommendations. However, no specific plans were offered to implement them. The final outcome of the conference, negotiated in advance, was couched in general terms. It did not include binding conmnitments from participants, e.g., quantitative targets for increased development assistance. But heads of state, leaders of private industry, and representatives of voluntary organizations from all over the world engaged in civil and substantive debate and found common ground. Specifically, the Monterrey consensus links aid effectiveness with developing country ownership of good policies and sound governance. Furthermore, it recognizes that developed countries have a responsibility to increase aid, ease debt burdens, and reduce barriers to trade. The new compact reflects hard-won lessons of development experience. First, evaluation and research have demonstrated that aid allocations yield better results if used to reward good performers, i.e., govemments that are responsive to their citizens and committed to equitable and sustainable development.3 In addition, a vast literature about development effectiveness calls for harmonized aid practices, improved aid coordination, and untying of aid4 as well as debt reduction for heavily indebted countries, more effective partnerships for the delivery of global public goods,5 and more equitable global public policies.6 Thus, the Monterrey challenge combines ambitious objectives, a focus on results, and an unprecedented partnership between developed and developing countries in pursuit of poverty reduction. For development practitioners, as will be shown below, the tasks ahead are daunting: a reconfiguration of the development agenda; a reform of aid practices; and a transformation of performance management, measurement, and evaluation systems. 2 United Nations, FinanangforDevekpment A Cniial Global Colboraion. Report of the Secretary-General to the Preparatory Comnmittee for the High-Level Intemational Intergovernmental Event on Financing for Development (New York: United Nations, 2002). 3 Nicholas Stem, Ian Goldin, Halsey Rogers, et al., The Role and Efediveness of DevelopmentAnirtance: Lerssonsfrom World Bank Experence (Washington, D.C.: World Bank, 2002). 4 Operations Evaluation Department, The Drive to Partnership: Aid Coordination and the World Bank (Washington, D.C.: World Bank, 2001). 5 Christopher D. Gerrard, Marco Ferroni, and Ashoka Mody, eds., Global Pmbh Poiies and Programs: IVRicationsforFinanang and Evalaon-Proeiengsfrom a WorldBank Workshop (Washington, D.C.: OED, World Bank, 2001). 6Joseph Stiglitz, "A Fair Deal for the World," New York Review of Books, May 23,2002. 5 Changing Conceptions of Development Cooperation From the very start, the development assistance enterprise was conceived as a transfer of financial resources and technical skills from rich countries to poor countries, and projects provided convenient vehicles for such transactions. They still do. Standard procurement, disbursement, and auditing controls help to ensure that the resources allocated to finance project inputs are u,sed for the purposes for which they are intended, while cost-benefit analysis verifies that projects "add value" to the economy of the recipient country. As long as physical investment was perceived as the primary engine of development, the "project approach" to performance measurement, monitoring, and management remained dominant. However, the concept of capital gradually evolved to incorporate human capital, natural capital, and social capital. The same evolution in development thinking identified policies and institutions as drivers of sustainable and equitable growth, confirming the "centrality of side effects" of development projects,7 undermining the relevance of simplistic cost-benefit calculations, and inducing a diversification of aid instruments (figure 1). Figure 1. Reconceptualization of the development agenda Phssical Capital Sociall Htim n- Natmral Dcvclopmnont | Major AMain Dominant acs Ojcive liistrtimcnts Discipline * Fifties * Reconstruction * Technical Assistance * Engineering * Sixties * Growth * Projects Finance * Seventies * Basic Needs * Sector Investment * Planning * Eighties Adjustment * Adjustment Loans * Macro-econormics * Nineties * Policy Reform and * Country Assistance * "Integrators" Institutional Development Strategy * Current * Poverty Reduction * PRSPs/Global Public * Multi-disciplinary Goods 7 Albert 0. Hirschman, Dvelopment Pro;xc4 Obsened (Washington, D.C.: Brookings Institution, 1967). 6 Evaluation methods have had to adapt to these new emphases. At the World Bank, the economic return threshold has remained at 10 percent for economic implementation of investment projects. However, development interventions today are assessed through a multiplicity of techniques, drawing on many disciplines. Development effectiveness is defined as the efficient contribution to the equitable and sustainable development agenda of a development action. Thus, all development operations are now rated in terms of their outcomes (that is, the relevance of their objectives and the extent to which these objectives are achieved efficiently), their sustainability, and their institutional development impact (box 2) by the Operations Evaluation Department. Box 2. Rating development performance The World Bank's Operations Evaluation Department (OED) uses an objectives-based evaluation approach based on five key ratings: outcome, sustainability, institutional development impact, and Bank and borrower performance. OED evaluates outcome by considering three factors: the relevance of the intervention's objectives in relation to country needs and institutional priorities; efficacy, i.e., the extent to which the developmental objectives have been (or are expected to be) achieved; and efficiency, i.e., the extent to which the objectives have been (or are expected to be) achieved without using more resources than necessary. OED's sustainabiity measure assesses the resilience to risk of net benefits flows over time by answering these questions: At the time of evaluation, what is the resilience to risks of future net benefits flows? How sensitive is the project to changes in the operaing environment? WllU the project coninue to produce net benefits, as long as intended, or even longer? How well will the project weather shocks and changing circumstances? Sustainability reflects the resiliency to risks of a project as measured by the likelihood that its estimated net benefits will be maintained or exceeded over the project's intended useful life and beyond. The institutional development impact measure evaluates the extent to which a project improves the ability of a country or region to make more efficient, equitable, and sustainable use of its human, financial, and natural resources and evaluates each project's success in fostering such changes. OED's assessments of Bank and borrower performance focus on how good a job each partner has done during the different stages of the project cycle, i.e., project identification, preparation, appraisal and implementation. Bank performance is judged based on the extent to which services provided by the Bank ensured quality at entry and supported implementation through appropriate supervision (including ensuring adequate transition arrangements for regular operation of the project). Borrowerperfornmance evaluates the extent to which the borrower assumed ownership and responsibility to ensure quality of preparation and implementation, and complied with covenants and agreements, toward the achievement of development objectives and sustainability. 7 Beyond investment projects, the toolkit available to development assistance practitioners has expanded. First, policy-based instruments relating aid disbursements to compliance with policy prescriptions have become common and now absorb a substantial share of World Bank lending resources.8 Second, the operating environment has become more complex, volatile, and pluralistic, leading to the introduction of adaptable instruments.9 Third, all operations are increasingly designed as vehicles for capacity development and policy learning.1" Fourth, programmatic lending operations in support of country-based poverty reduction programs are being piloted. As a result, evaluation methods have evolved (box 3). Box 3. Evaluation methodologies are being adapted to reflect dominant paradigms of development policy Dominant concepts C. - - Characteristics of .of4ev4op~ient 7 develop~ment evlutioJni * Project focus * Project evaluation BEFORE * Investment-driven growth ' Cost-benefit analysis * Import substitution . Shadow pricing * Central planning * Self-evaluation * Country focus * Portfolio evaluation * Structural adjustment Policy evaluation NOW * Outward-oriented policies Risk analysis * Decentralized decisionmaking Participatory evaluation 8 Externally imposed conditions over reluctant govermments proved meffective so that adjustment lending now favors ex- port Ccarrots") rather than ex ante ('sticks') conditions. Adaptable Program Loans (APLs) and Poverty Reduction Support Credits (PRSCs) now include performance triggers (as do some individual PRSCs, e.g., Uganda). 9 Madhur Gautam, 2000-2001 Annual Report on Operationx Eubaaaion (Washington, D.C.: OED, World Bank, 2002). 10 Research economists have long argued that fungibility of finncial resources undercuts the "additionality" feature that projecr evaluations used to take for granted. 8 The New Development Paradigm A paradigm arises when a professional community adopts new beliefs about reality and subscribes to common "symbolic generalizations" about its expert discipline." The development paradigm displayed in Monterrey combines a results-orientation; domestic ownershbp of improved policies;partnersbh between governments, the private sector, and the civil society; and a long-term, bolistic approach that recognizes the interaction between development sectors and themes. The principles of effective aid issued by the DevelopmentAssistance Committee (DAC) of the OrganizationforEconomic Cooperation and Development (OECD) embody these tenets.2 The same principles animate the Comprehensive Development Framework (CDF) and the Povert Reduction Strategy Program (PRSP) endorsed by the Development Committee of World Bank governors.U They also underlie the United Nations DevelopmentAssistance Framework (UNDAF) used to improve coherence among the activities of United Nations specialized agencies at the country level. The new paradigm poses significant challenges for development cooperation and evaluation.'4 In terms of results, growth of per capita incomes in developing countries would have to be twice the levels achieved in the 1990s for the next fifteen years in order to reach the MDGs' income and poverty reduction objective. Only 33 developing countries are on track to meet the goal. Another 65 are unlikely to meet the goal without major policy changes and expanded assistance. Similarly, the prospects for cutting the number of malnourished people by half by 2015 and achieving many of the other goals are not bright given the uncertain prospects for reform of policies and institutions in developing countries, increased aid, and more equitable global public policies (figure 2). With respect to ownershbp of improved policies, an OED study of 43 adjusting countries over the period 1975-96 showed that only 12 percent of countries demonstrated the capacity and the commitment to achieve durable and major improvements in their policy regimes. They achieved GNP per capita growth three times as high as the rate of countries that had not yet achieved durable adjustment and six times higher than countries that have oscillated between weak and strong policy environments." II Thomas S. Kuhn, The Stncre of Saentft RevouAdons (Chcago and London: The Uriversity of Chicago Press, 1996). 12 Development Assistance Committee (DAC), Pnnialks ofEefective Aid (Paris: Organization for Economic Cooperation and Development, 1992). 13 Nagy Hanna et al., 1999AnnualResiew ofDevelometnEftctvneveess (Washington, D.C.: OED, World Bank, 2000) and Operations Evaluation Departmrent (OED), 'Toward a Comprehensive Development Strategy," Precis No. 197 (Washington, D.C.: OED, 1999). 14The MDGs comprise 18 targets and 48 indicators (refer to box 1). 5 Robert Buckley, 1998AnnuaiReiew ofDeseIpment EffecMaeness (Washington, D.C.: OED, World Bank, 1999). 9 Figure 2. Are the MDGs being reached? 16 ErastA and Pncfic pAnd Cwn AIa 23 coutl 28U uniis .lOV% .100% 0% 0r% ___ D#_ - - _ - - ~~~~~~~~~~~~~~~~50%_ 0-1OOX~ -- 0* *t -Ai'* Labtnwlca Sdea thb Eat wi Nh Aca 32 oLnbtee6oLMe 10*%- - ** S~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~5 - .5- ltr˘ 513% 10 0* PI*s5 0~~~~~~ 0* k00~~~~ 1W1MA20 0* P,=900 ~ HJ _~ * 1 5 SIY fl0li . %5 r a*1 a i oQyk n50Sv n5 r.1y 4 -~~~~~~~~~~~~~~~~~~~~-0 dd .0w~~ Soutors ia theirhaningrtica C cPircomlo a Cortaibev mantbnaon hlrn ehlrno ppont g oy noldinpiayad o1y e 0000lv ilh, HVADSann on 150% 10 - U m m - - 5~~~~~~~~~~~~~0%- -50% 510% nuodU, age . y opltngt3t oft-MY "5t501 scoo .10 Tr t ety thret- P.(k1 The indicators and their targets Child malnutrition Pmary school completion Gender equality in school Child Mortaity Matenal mortlity HIVIAIDS prevalence Indicator. Prevalence of lndicator Percentage of Indicator Ratio ofgirts to Indicator. Underfive chld :ndicaor. Matenal deaths Indicator. Prevalence of malnutriton among children children of appropriate age boys enmolled in prirary and mortality. per 100,000 live birts, HIV/AIDS among young under age five, measred by completing last grade of secondary school. Target: Target Reduce by three- women (ages 15-24). weight for age. official pnmary school. Target- Achieve equally in Reduce by two-thirds quanters between 1990 and Target Have halted by 2015 Target: Reduce by hall Target Achieve 100% enrollment ratios by 2005. between 1990 and 2015. 2015. and begun to reverse the between 1990 and 2015 -completion by 2015. spread of HIlVIAIDS. 16 ol Bank, Development Economics Group, Wlorld Develpment indicatorsr 2002:- Mil6l,nium Development Goals (Washington, D.C: World Bank, 2002). 10 With respect to partnership, developed countries are not living up to the doubling of aid volumes that will be required to meet the MDGs. In addition, protectionist pressures are rising and aid reforms have been partial and slow. Finally, tensions have arisen between the holistic, long-term, comprehensive approach to development and the selectivity required for achieving results efficiently.'7 Does Aid Work? The development consensus favors growth-oriented, market-friendly development and equitable access to social services and social safety nets. But initial conditions, factor endowments, and political economy considerations vary so widely across regions and countries that no standard policy prescription commands universal acquiescence. Nor does unanimity prevail as to the aid strategies suitable for countries plagued by conflict, corrupt governance, or lack of commitment to sound policies. Development pessimists view the global quest for poverty reduction as elusive, the history of aid as riddled with costly failures, and the pleas to meet predetermined aid volume goals as ill-informed. On the other hand, development optimists contend that the overall record of development assistance is a historic achievement and that the determinants of growth are by now well known. They advocate larger volumes of aid allocated to good performers to achieve poverty reduction. There is little doubt that economic growth and improvements in social indicators in the developing world have been unprecedented over the past 50 years. But the impressive growth record of China and the respectable performance of India (two large and populous countries that have had modest access to aid) weigh heavily in the aggregate statistics. In the rest of the world, only East Asia has achieved substantial poverty reduction. No gain in poverty reduction was achieved in Latin America, while the incidence of poverty increased in Africa and the former'Soviet Union.'8 Thus, a case-by-case approach guided by professional judgment and inspired by best practice (rather than a dogmatic compliance to standard prescriptions) is key to development effectiveness. Hence, the need for better performance management, measurement, and evaluation. 17Nagy Hanna and Robert Picdotto, eds., Making Development Work. Development Learn in a WorldofPoveryand Wcalh, World Bank Series on Evaluation and Development, Volume 4 (Washington, D.C.: World Bank, 2002). I J ehan Arulpragasam and Giovanna Prennushi, Povny Rednction and The World Bank: Prvgrs in Operaeionalitjng the WDR 2000/01 (Washington, D.C.: World Bank, 2000). Implications for Performance Management Greater accountability for results has become a public imperative. Rating agencies have been developing indicators to help channel capital toward environmentally and socially responsible uses. Private companies are adopting "triple bottom line" concepts to assess investment performance. Greater transparency in reporting is being adopted voluntarily and/or legislated. Pension funds are increasingly moving their investments out of companies with poor social and environmental records. Nongovernmental agencies are scrutinizing the social development consequences of aid and foreign direct investment. The time has come to adopt similar principles in the development assistance business. The new development paradigm emphasizes results, partnership, coordination, and accountability. This implies excellence in performance management. Basic transformations in the structures, products, and processes of the aid industry are needed in order to enhance development effectiveness. Projects still dominate the aid landscape. They provide a credible fiduciary environment for channeling aid monies in operating environments often characterized by weak public administrations and corruption. A recent World Bank and International Monetary Fund [M) study has found that most HIPC countries cannot yet track poverty expenditures adequately."9 Control considerations focus on procurement of inputs, disbursement and auditing of funds, and compliance with social and environmental safeguards. These requirements account for substantial aid transaction costs. More often than not, aid is poorly coordinated and does not conform to the comparative advantage of partners. The efficiency of aid delivery has dropped as donors have multiplied and aid volumes have declined. Harmonization of aid delivery mechanisms, fiduciary processes, and reporting requirements have lagged behind. Pool funding for sector-wide approaches is the exception rather than the rule. The resulting administrative burdens on government officials have mortgaged the skills needed to improve the effectiveness of the overall government administration.20 Participation of the civil society and the private sector is essential for effective public service delivery and accountability for results.2' But increased citizens' participation in government processes involves substantial costs if handled project by project. Thus, in order to achieve results, the new development paradigm calls for scaling up of aid beyond projects to the higher plane of policy and institutions. 19 World Bank/IMF, Actions to Strngnyben the Tracking of Povei-Reduing Pubk Spendng in Heamv Indebted Poor Cowntries (fMPC) (Washington, D.C.: World Bank, 2002). 2 Operations Evaluation Departraent, The Drive to Pa1norshabx Aid Coordintion and the World Bank (Washington, D.C.: World Bank, 2001). 21 Operaions Evaluation Departrent, Pubh Sector Peiformance-Tbe Cri.icalRok of Evaluation (Washington, D.C.: World Bank, 1998). 12 Implications for Performance Measurement First and foremost, development indicators should go beyond the measurement of inputs (number of projects, volume of conunitments, disbursements, and the like) in order to capture program results, i.e., outputs, outcomes, and impacts. Second, the primary unit of account for monitoring and evaluation should reach out from the individual project to the country program. Third, monitoring indicators should allow tracking of progress toward the MDGs-as well as the intermediate objectives embedded in country programs. Fourth, the performance of individual partners should be assessed in terms of their distinctive accountabilities and reciprocal obligations. For development assistance agencies these principles should be reflected in corporate scorecards consistent with results-based management principles (figure 3). Figure 3. Corporate scorecard model Results Chain I. Contribution to Millennium Development IMPACTS Goals t \- Attnibution * EXOGENOUS FACTORS II. Country Program Results OUTCOMES a GLOBAL POLICIES III. Partnership Processes PAERFRMANCE (CDF, PRSP, UNDAF) REACH ˘ . / -* AGENCY PERFbORMANCE IV Financial & Knowledge OUTPUTS Products V. Budget Resources & Skdlls INPUTS 13 This is a demanding agenda considering that the record of monitoring and evaluation has been dismal even at the project level. This is partly explained by a lack of domestic evaluation capacity. But it is also due to distorted organizational incentives and to the low priority given to inputs versus results (i.e., outputs, outcomes, and impacts) by aid donors and recipients alike. Meager resort to independent verification by qualified academic institutions and voluntary agencies compounds the problem. Furthermore, the fragmentation of aid among hundreds of projects translates into high costs for expert data collection and interpretation. Unless donors change their aid procedures to deliver aid on a programmatic, common pool basis, it is doubtful that the situation will improve rapidly. At the county level, donor efforts have focused on generating household surveys and improving national statistics. Public expenditure program evaluation through logical frameworks, tracking surveys, and participatory methods has been neglected. Country assistance strategies leave much to be desired in terms of their "evaluability."22 Results chains are rarely used to make transparent the linkages between program and project actions and development outcomes, including the MDGs. Evaluation capacity development has lagged and relevant data are often not collected, interpreted, or used for decisionmaking.23 At the,globallevel, monitoring and evaluation is largely absent.24 Collaborative programs designed to deliver global public goods are not subjected to independent appraisal and, as a result, often lack dear objectives and verifiable performance indicators. In addition, the impact of developed country policies on poor countries is not assessed systematically even though aid, debt, foreign investment, pollution, migration pattems, and intellectual property regimes are shaped by the decisions of developed country governments. Implications for Performance Evaluation The new development paradigm has raised the bar for evaluation. The conceptual foundations for objective-based performance evaluation exist. However, the credibility of evaluation hinges in large part on its governance, i.e., on the set up of independent evaluation units that report to country legislatures or to goveming bodies-or other independent verification mechanisms.25 This prerequisite of credibility is missing in the evaluation systems used by most governments, companies, and development agencies. The frequent option of resorting to consultants does not guarantee independence. 2 Joseph S. Wholey, EvaluabityAsuesmen: ImpIbroinng Evalration, Management, and Permanac (Washington, D.C.: U.S. General Accounting Office and University of Southern C alifornia, 2002). 23 Operations Evaluation Departnent, Annual Report on Evalation Capaay Development 2002 (Washington, D.C.: OED, World Bank, forthcoming.). 24 Um a Ldle, Global Publ Po&ies and Prqgrams (Washington, D. C.: OsED, World Bank, forthcoming.). 25 Comptroller General of the United States, GovovnmentAudiiing Standards, 1994 Rision (Washington, D.C.: GPO, 1994). 14 Independence does not mean isolation. Evaluation processes should combine independent and self-evaluation. They should be participatory and become an integral part of business processes, public administration, and aid procedures in order to combine accountability with learning and adaptability. This means that self-evaluation should be built into all major corporate processes and transactions at project, country, and global levels and that independent evaluation should attest to their ngor and quality. In effect, monitoring and evaluation should be treated as fiduciary requirements. Independent and self-evaluation are to the public sector what accounting and auditing are to the private sector. Excellence in evaluation requires the adoption of appropriate survey mstruments and analytical tools wielded by competent and experienced evaluators. Timeliness in evaluation implies the set up of "just in time" qua1i-y tracking mechanisms for advisory and capacity building operations to complement retrospective evaluation of investment projects and adjustment operations (figure 4). Figure 4. Evaluation must connect to real-time quality assurance and tracking EVALUATION Impact evaluation Quality at exit IndependentEvalua don IDENTIFICATION Self-Evaluation MONITORING PREPARATION QuaityAssurance _SUPERVISION APPRAISAL . Quality at entry 15 The Country Program as Unit of Account The shift from projects to county programs as the unit of account for performance management and evaluation requires the application of triangulation methods designed to overcome three major methodological challenges: (i) aid allocations; (ii) aggregation; and (iii) attribution.26 Given that development effectiveness hinges in large part on the quality of the enabling policy and institutional envirornent,performance-based allocations are critical. Such allocations must be informed by the results of evaluation. They require the regular compilation of policy and institutional indicators at the country level. Their selection raises complex conceptual challenges and their interpretation and use call for objective and transparent quality assurance mechanisms.27 Selectivity of instruments and choice of products in line with the comparative advantage of individual donors through aid coordination is another important determinant of development effectiveness.28 The use of performance indicators is closely related to the allocation problem. Optimally, such indicators should allow tracking of the MDGs and connect them (through an explicit or implicit program theory)29 with the policy and institutional actions promoted by the country assistance program. This approach yields the results-based evaluative dimension against which progress of country programs should be rated. The problem of aggregation arises because the quality of a country assistance program must also be judged through quality assessments of the individual building blocks that make up the program. This meta-evaluation technique relies on independently validated ratings of individual operations through regular business processes-a golden rule that is rarely practiced. Such aggregation of evaluative judgments regarding individual operations does not necessarily equate to an evaluative judgment of the overall program. The relevance of individual operations is not accurately assessed in isolation. Depending on the quality of country programming and dialogue, there can be synergy among financial services, advisory services, and other knowledge-based operations. The whole can be more-or less-than the sum of the parts.30 26JohnJohnson and Ruben Lamdany, OED Methodologvfor CounhyAsirstane Evauations (Washington, D.C.: OED, World Bank, forthcoming.). 27 Stephen Eccles, "IDA Review Review of the Performance-Based AUocation System, IDA 10-12," OED Working Paper Series (Washington, D.C.: OED, 2001). 28 Wllliam Battaile, 2001 Annual Review of DevpmentEffetiveness: Making Choices (Washington, D.C.: OED, Wordd Bank, 2002). 29 Operations Evaluation Department, Monitonng & Evaluaion: Some Tools, Methods, &Appnpaches (Washington, D.C.: OED, World Bank, 2002). 30 William Battaile, 2001 Annual Review of Development Effecveness: Making Choices (Washington, D.C.: OED, World Bank, 2002). 16 Finally, the problem of attribution has to do with assessing the contribution of various partners that (together with initial conditions and exogenous factors) help to determine the development outcomes and impacts of country assistance programs. Such assessments focus on the compliance of development actors with the agreed policies and procedures imposed by their own governance. Use of focus groups, client surveys, and advisory committees is needed for such assessments. Toward a New Evaluation Architecture Just as a lack of coherence raises the cost of the aid business, the fragmentation of evaluation products and the diversity of evaluation methods among donors contribute to "evaluation bombardment." For aid evaluation as for aid delivery, the solution lies in harmonization, coordination, and the forging of partnerships. More resources need to be devoted to nurturing of convergence in evaluation methods through networks of evaluators, development of evaluation capacities, professional associations (such as the International Development Evaluation Association, IDEAS),31 joint or parallel evaluations, and country- based evaluations connected to enhanced processes of public expenditure management. Harmonization of evaluation methods among multilateral development banks has made good progress at the level of the individual project. Joint or parallel evaluation activities among MDBs with respect to country assistance evaluations have been undertaken. By contrast, progress has been slow in the bilateral aid business and the United Nations. Harmonization of country assistance evaluation methods has lagged and may remain elusive until operational policies are brought into line across all multilateral and bilateral donors and aid coordination is strengthened so as to enlist comparative advantage and achieve selectivity. The logic of the country-based poverty reduction strategy process is to implement the new development paradigm reflected in the Monterrey consensus. The development architecture will eventually be reshaped. It will inevitably rely on "pool funding" allocated according to performance and results. To facilitate this transformation process, high-quality monitoring and evaluation will be required in order to ensure accountability and learning at the country and global levels. Hence, it is not too early to lay the foundations for a country-based evaluation architecture that embraces the U.N. system, the multilateral development banks, and the bilateral aid system as well as governments, civil society, and the private sector so as to better meet the global poverty reduction challenge (figure 5). 31 The International Development Evaluation Association, IDEAS, seeks to legitimize and strengthen evaluation soceties and associations by promoting the systematic use of evaluation in civil society. It will build evaluati on capacity, develop principles and procedures in evaluation, encourage the development of new societies and associations, procure resources for cooperative activity, and be a forum for the exchange of good practice and theory in evaluation. 1 7 Figure 5. Toward a new evaluation architecture CO L -B , I OI t UN X\- I:xl11iC)1: NIDB I,-IlK16l:lri -~~~ DAC'\ W'lrking Party 18 References Arulpragasam, Jehan, and Giovanna Prennushi. 2000. Povertj Reduction and the World Bank.- Progress in Operational:Ung the WDR 2000/01. Washington, D.C.: World Bank. Battaile, William. 2002. 2001 Annual Review of Development Effectiveness: Making Choices. Washington, D.C.: OED, World Bank. Comptroller General of the United States. 1994. Government Auditing Standards, 1994 Revision. Washington, D.C.: GPO. Development Assistance Committee (DAC). 1992. Pnnciples of Effective Aid. Paris: Organization for Economic Cooperation and Development. Ecdes, Stephen. 2001. "IDA Review: Review of the Performance-Based Allocation System, IDA 10-12." OED Working Paper Series. Washington, D.C.: OED. Gautam, Madhur. 2002. 2000-2001 Annual Report on Operations Evaluation. Washington, D.C.: OED. Gerrard, Christopher D., Marco Ferroni, and Ashoka Mody, eds. 2001. Global Public Policies and Programs: Implications for Finanang and Evaluation-Proceedings from a World Bank Workshop. Washington, D.C.: OED, World Bank. Hanna, Nagy et al. 2000. 1999 Annual Reidew of Development Effectiveness. Washington, D.C.: OED, World Bank. Hirschman, Albert 0. 1967. Development Projects Observed. Washington, D.C.: Brookings Institution. Johnson, John, and Ruben Lamdany. Forthcoming. OED Methodology for Country Assistance Evaluations. Washington, D.C.: OED, World Bank. Johnston, Timothy, and William Battaile. 2001. 2000 Annual Review of Development Effectiveness: From Strategy to Results. Washington, D.C.: OED, World Bank. Kuhn, Thomas S. 1996. The Structure of Scientific Revolutions. Chicago and London: The University of Chicago Press. Lele, Uma. Forthcoming. Global Public Po/icies and Programs. Washington, D.C.: OED, World Bank. 19 OED (Operations Evaluation Department). Forthcoming. Annual Report on Evaluation Capaciay Development 2002. Washington, D.C.: OED, World Bank. . 2002. Monitoring & Evaluation: Some Took, Methods, & Approaches. Washington, D.C.: World Bank. . 2001. The Dnve to Partnersbhp: Aid Coordination and the World Bank. Washington, D.C.: World Bank. . Autumn 1999. "Toward a Comprehensive Development Strategy." Precis No. 197. Washington, D.C.: OED. 1998. Public Sector Performance-The Crtizcal Role of Evaluation. Washington, D.C.: World Bank. Stern, Nicholas, Ian Goldin, Halsey Rogers, et al. 2002. The Role and Effetiveness of Development Assistance: Lessonsfrom World Bank Experience. Washington, D.C.: World Bank. Stiglitz, Joseph. 2002. "A Fair Deal for the World." New York Reniew of Books, May 23. United Nations. 2002. Financng for Development: A Critical Global Collaboration. Report of the Secretary-General to the Preparatory ComnIiittee for the High-Level International Intergovermnental Event on Financing for Development. New York. Wholey, Joseph S. 2002. Evalmabikdy Assessment: Improting Evaluaton, Management and Petfo7nance. Washington, D.C.: U.S. General Accounting Office and University of Southem California. World Bank/IMF. 2002. Actions to Strengthen the Tracking of Povert-Reducing Pub& Spending in Heavi/ Indebted Poor Countries (HIPC). Washington, D.C.: World Bank. World Bank. 2002. World Development Indicators 2002: Millennium Development Goals. Washington, D.C.: World Bank. 1997. World Development Report 1997: The State in a Changing World. New York, NY: Oxford University Press for the World Bank. 20