United Nations Development Programme Evaluation Office (UNDP/EO) Chinese National Center for Science and Technology Evaluation (NCSTE) The World Bank Operations Evaluation Department/World Bank Institute (OED/WBI) 27978 EVALUATION CAPACITY DEVELOPMENT IN ASIA EVALUATION CAPACITY DEVELOPMENT IN ASIA Selected Proceedings from the International Conference Beijing, October 1999 Edited by Khalid Malik and Christine Roth United Nations Development Programme Evaluation Office (UNDP/EO) Chinese National Center for Science and Technology Evaluation (NCSTE) The World Bank Operations Evaluation Department and World Bank Institute (WBI) The views expressed in this report are those of the authors and do not necessarily represent those of the United Nations or the United Nations Developement Programme. © Evaluation Office 2000 Evaluation Office United Nations Developement Programme One United Nations Plaza New York, NY 10017, USA Design: Julia Ptasznik, Colonial Communications Corp. Cover Photograph: David Kinley/UNDP CONTENTS Foreword .......................................................................................................... v Acronyms ....................................................................................................... vii Executive Summary ........................................................................................ ix P a rt I: The Evolving Context for ECD 1. Opening Speech .............................................................................................. 3 Ms. Den Nan, Vice Minister of Science and Technology of PRC 2. Opening Statement: The New Dimensions of ECD in International Development .................... 5 Mr. Khalid Malik, Director, UNDP Evaluation Office 3. The Asia Perspective in Evaluation Capacity De velopment: The Challenges of the New Millennium ........................................................ 9 Mr. Adil Khan 4. Aid Evaluation:A Donor Perspective .......................................................... 19 Mr. Niels Dabelstein, Head of Evaluation Secretariat, DANIDA, OECD/DAC WP P a rt II: Evaluation of Performance Management in the Public Sector 5. Results Based Management in the De velopment Cooperation Agencies: A review of Experience ........................................... 23 Ms. Annette Binnendijk, Consultant UNDP 6. Evaluation Capacity Development in the People’s Republic of China: Trends and Prospects ...................................... 39 Mr. Ray Rist, Evaluation Advisor, World Bank 7. Evolution of Evaluation in the People’s Republic of China ........................ 45 Mr. Peter C. Darjes, Operations Manager, Asian Development Bank 8. Linking Evaluation to Policy Formulation and Budget Processes – Lessons Learned from Aust ralia ................................... 49 Mr. Leonard J. Early, Depatment of Finance – Australian Public Sector 9. Public Performance Evaluation and Improvement ..................................... 55 Mr. Marc Holzer, Executive Director, National Center for Public Productivity, NJ - USA ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the Ne w Millennium iii P a rt III: A) Monitoring Evaluation in a Results-Based Management Approach B) Evaluation Initiatives in Science and Te c h n o l o gy 10.Evaluation Parternships ............................................................................... 65 Mr. Niels Dabelstein , Head of Evaluation Secretariat DANIDA, OECD/DAC WP 11.Evaluation Tools and Methodologies:Case Studies ................................... 69 Mr. Jacques Toulemonde, Director, Centre for European Evaluation Expertise 12.Evaluation Initiatives in Science and Technology Programs ..................... 77 Mr. Rustam Lalkaka, President, Business & Technology Development Strategies 13.Review of Draft of Science and Technology Evaluation Standard of China ................................................. 91 Ms. Chen Zhaoying, Vice President NCSTE P a rt IV: ECD Opportunities for National Initiatives and International Cooperation 14. Guide to Diagnosing ECD Needs: Video Speech ...................................... 97 Mr. Keith Mackay, Senior Evaluation Officer, World Bank Institute P a rt V: Meeting W r a p - U p 15.Meeting Wrap-Up ...................................................................................... 103 Ms. Christine Roth, Evaluation Advisor, UNDP P a rt VI: A n n e x e s Annex 1.List of Participants ...................................................................... 107 Annex 2.Conference Agenda ..................................................................... 111 iv E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing FOREWORD There is growing awareness in both developed and developing countries of the potential role that evaluation can play in improving the quality of public sector decision-making. Evaluation is becoming recognized as particularly important in promoting transparency and accountability; in some instances, it is even viewed as an integral part of responsible civic action and civil society development. Concurrent with the increasing interest in evaluation as a component of public sector development and civil service reform, evaluation capacity development (ECD) is being placed on the agenda of cooperation between developing countries and development assistance agencies. The development of national evaluation capacities can be a critical component of efforts to strengthen the responsiveness of public management systems to both internal and external demands for transparency, effectiveness and a results orientation. Therefore, the World Bank, for example, organized a high-level seminar in Abidjan, Côte d’Ivoire, in 1998, during which senior officials from various African countries and international development assistance agencies looked at the status – and helped to plan the future – of monitoring and evaluation (M&E) capacity development in Africa. In the past few years, UNDP has also been particularly active in the field of ECD. A survey of existing national M&E systems was undertaken in more than fifteen programme countries. In collaboration with national entities, recommendations were made to strengthen the effectiveness of these systems. In this context, the United Nations Development Programme and the Chinese National Center for Science and Technology Evaluation co-sponsored the Beijing Conference on Evaluation Capacity Development (ECD) with strong support from the World Bank. The conference, which was held 27-28 October 1999, focused particularly on issues of performance management and public sector reform. Participants debated intensely on such issues as transparency and the independence of the evaluation function and, as a corollary, the challenges of introducing a culture of evaluation at the national level. Some of these challenges stem from the political environment and the need to develop an appreciation at decision-making level of the importance of and the need for evaluation. Other challenges are linked to the systemic problems of institutionalizing and integrating evaluation into the government machinery and the weakness of internal capacities, human and financial, necessary to support a professional cadre of evaluators. The development of national capacities in evaluation requires strong alliances and partnership-building. Conference participants, aware of the benefits of such relationships, showed strong interest in launching an international development evaluation association to help forging links with evaluators engaged in development related activities in both donor and programme countries and support individual efforts of countries interested in creating such an association at the national level. UNDP and the World Bank emerged as the main promoters of this international association. Another topic of considerable interest for the Conference was the work carried out by aid agencies on the interface between results-based management (RBM) and monitoring and evaluation in connection with public sector reform and good FOREWORD V governance. Despite the enormous methodological dividends. In particular, the Confernce highligted the and technical challenges linked to this management strong potential for maximizing synergies between approach, results orientation in the public sector is the work of development agencies, development banks clearly the way of the future. and national entities to promote the role of evaluation as part of good governance and public sector reform. Given the high level of participation and interest in the topics debated during the Conference, it is clear It is hoped that the insight contained in the formal that the relatively modest investment made to finance presentations of this volume will stimulate further the Conference has the potential for yielding big discussions and developments in the field of ECD. ACKNOWLEDGMENTS The sponsors of this conference were: s The United Nations Development Programme Evaluation Office (UNDP/EO) s The National Center for Science and Technology Evaluation (NCSTE) s The World Bank Operations Evaluation Department and World Bank Institute (WBI) Special thanks go to Professor Fang Yan and Christine Roth for their dedicated work in organizing the Conference. Thanks also go to others who provided substantive inputs in preparing the Agenda: Ray Rist, Keith McKay and Niels Dabelstein. This event would have not been possible without the logistical support provided by Feng Bo, Bousso N’Diaye, Victoria Vieitez,John James and the whole administrative team of the NCSTE. The report was produced by the UNDP’s Evaluatin Office staff including Christine Roth (task manager) Habila Maiga, Barbara Brewka (editor) and Anish Pradhan. vi E V A L UATION C A PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing ACRONYMS ADB: Asian Development Bank CBC: Construction Bank of China CIECC: China International Engineering Consulting Company CMED: Central Monitoring and Evaluation Division C N AO: China National Audit Office DAC: Development Assistance Committee DANIDA: Danish International Development Agency ECB: Evaluation Capacity Building ECD: Evaluation Capacity Development EPU: Economic Planning Unit EXSL: Exemplary State and Local Programs GASB: Governmental Accounting Standard Board IFI: International Finance Institutions IMF: International Monetary Fund ITD: Industrial Technology Development KPIO: Key Project Inspectorate Office MEANS: Methods for Evaluating Actions of a Structural Nature MOF: Ministry of Finance ODA: Overseas Development Administration OECD: Organization for Economic Co-operation Development PEO: Programme Evaluation Organizations PPMS: Project Performance Management System PRC: People’s Republic of China R&D: Research and Development RBM: Results-Based Management RTO: Research and Technology Organizations S & T: Science and Technology SAA: State Audit Administration SDB: State Development Bank SDPC: State Development Planning Commission SERU: Socio-Economic Research Unit SLEA: Sri Lanka Evaluation Association SME: Small and Medium Enterprise SPC: State Planning Commission SPO: State Planning Office SRF: Strategic Result Framework SURF: Sub-regional resource facilities TBI: Technology Business Incubators TEC: Training and Enterprise Council UNDP: United Nations Development Programme UNFSTD: United Nations Fund for Science and Technology Development UNICEF: United Nations Children’s Fund USAID: United States Agency for International Development U S PAP: Uniform Standards for Professional appraisal Practice WAITRO: World Association of Industrial and Technology Research Organization ACRONYMS vii EXECUTIVE SUMMARY In both developed and developing countries, there is a growing awareness of the potential role that the evaluation function can play in improving the quality of public sector decision-making. Evaluation is being recognized as a mechanism for providing accountability for the use of public resources as well as an instrument for organizational learning. Concurrent with the increasing interest in evaluation as a component of public sector development and civil service reform, evaluation capacity development (ECD) is being placed on the agenda of cooperation between developing countries and development assistance agencies. Against this backdrop, the United Nations Development Programme (UNDP) and the Chinese National Center for Science and Technology Evaluation (NCSTE) co-sponsored the Beijing Conference on Evaluation and Capacity Development, which took place from 27-28 October 1999. This initiative also received strong support from the Asian Development Bank (ADB), the Development Assistance Committee of the Organisation for Economic Co- operation and Development (OECD/DAC) and the World Bank. The objectives of the Conference were fourfold: (a) to stimulate reflection on the role of evaluation in good governance and public sector reform, including sharing of experiences; (b) to explore the interface between results-based management (RBM) and monitoring and evaluation in connection with good governance; (c) to identify strategies and resources for building monitoring and evaluation (M&E) supply and demand in Asian countries;and (d) to encourage and support the creation of country and regional networks to facilitate follow-up actions. To achieve these objectives, the Conference agenda was organized around two broad categories of evaluation issues: (a) The policy and institutional context of evaluation: s Policy frameworks for public sector results-orientation s Organizational learning vs. management accountability s Linkage between evaluation and policy formulation, resource planning, programme monitoring and management ; (b) The practice of evaluation: s Evaluation criteria and conceptual approaches s Options for organization of the evaluation function s Skills, resources and methodologies s Challenges of evaluation in specific sectors, e.g., science and technology. The Conference brought together high-level officials from 13 Asia and Pacific countries (China, Democratic Republic of Korea, Fiji,India,Indonesia, Kazakhstan, Kyrgyzstan,Malaysia,Mongolia, Nepal, Pakistan, Sri Lanka, Viet Nam), staff of the UNDP country offices in the region, members of EVALNET (the UNDP evaluation network), a large number of representatives of the international public sector and development agencies such as the World Bank,The National Center of Public Productivity (USA), The Centre for European Evaluation Expertise, EXECUTIVE SUMMARY ix OECD/DAC, the Asian Development Bank In view of the short period allocated for group and resource persons from the community of work, the participants were not requested to develop international experts. action plans for ECD in their countries but rather to identify a few concrete, realistic actions/strategies The agenda comprised formal presentations by to promote and strengthen ECD in their respective experts in M&E and RBM that explored general countries in the short term. themes as well as specific experiences of donor agencies and national institutions. Participants The resulting action points isolated some important shared their experiences and discussed issues elements that were seen as critical to any plan to surrounding the building of a national M&E strengthen ECD: system at the country level. Part of the agenda was also designed to start the process of reflection by s Institutional support. Institutional support participants on the design of their own ECD for ECD is crucial. There should be an awareness action plans. and acceptance of M&E activities in key decision-making centres. Evaluation should Discussions during the Conference highlighted be promoted as an independent function and some important issues faced by national entities. standards and criteria should be developed. First and foremost,there was unanimous agreement that in a world of globalization and shrinking s Training. Training still represents an important resources, governments and public organizations are element of any ECD programme: training of faced with growing scrutiny from their constituency officials and technicians involved in M&E; to ensure increased responsiveness, transparency participation of officials and civil society in and accountability in the delivery of services. In joint evaluations with external aid agencies. this context, the role of both evaluation as part of good governance and ECD programmes that could s Networks. The concept of networks emerged help to support planned public sector reforms become as a powerful instrument for facilitating the extremely important. Second, the independence exchange of experiences, access to best practices of the evaluation function was hotly debated as and sharing of databases. were the potential constraints to the establishment of a culture of evaluation. These constraints include The establishment of evaluation networks at both the political environment and the weak demand the national and regional levels is the first concrete for evaluation in many countries, the systemic follow-up action to which participating countries problems that still exist in various countries that made a commitment, with the support of the hinder the application of sound evaluation systems, international community. China in particular and and limited internal capacities in terms of financial the National Center for Science and Technology and qualified human resources. Third, with the spread Evaluation volunteered to establish and maintain of results-based management and the realization such a network for the region. that development outcomes require the forging of strategic alliances, discussions centered on the At the international level, both UNDP and the importance of partnerships in joint evaluations and World Bank, in agreement with several other the need to involve stakeholders and civil society. development agencies, are now working together to promote the establishment of an international Based on the experiences shared by the various development evaluation association. The idea of participants, it became quite obvious that despite the association was widely endorsed at the Beijing the commonality of issues,there is no standardized Conference. Such an association will help in approach to strengthening evaluation capacity forming links and support the individual efforts of development and developing effective monitoring the several countries that expressed interest in and evaluation systems. creating such an association at the national level. x E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing PART I The Evolving Context for ECD E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing OPENING SPEECH 1 Deng Nan, Vice Minister of Ministry of Science and Technology of PRC Distinguish guests, Ladies and Gentlemen: Good morning! First of all,please allow me on behalf of the Ministry of Science and Technology, as well as myself, to warmly welcome all the officials from UNDP and the World Bank, as well as the distinguish guests and experts,scholars on the evaluation from all the other countries participating at the International Conference on Evaluation Capacity Development. We Chinese people have just celebrated the fiftieth anniversary of the founding of the great People’s Republic of China, and together with other people in the world we are looking for the coming of the new century. Today, initiated jointly by UNDP and National Center for Science & Technology Evaluation, delegates from all over the world gathered here in Beijing, with the topic of improvement of capacity of evaluation to discuss and try to make improvement on the scientific decision making as well as the improvement of responsibility mechanism of government, and the utilization of public resources. This meeting has not only been a very important strategic impact on the course of the Chinese Open-door policy, but also a great impact on the development of all the other countries, so as to the progress of human being. As everybody knows, humanity is facing a brand new century, with marks of information technology, and a focus on the competition of hi-tech. Technology innovation has greatly motivated the technological progress of the world. In order to grasp the future of our development, to manage the society more efficiently, for the best utilization of the resources, which are common to us, for steadily making a better living environment, the improvement of the quality of decision making and efficiency of management are vital. Along with the development of Chinese reform and its open-door policy, the Chinese government, as early as 1986, has started to focus on the importance of scientific decision making according to the public opinions. In 1993, the Ministry of Science and Technology of China initated technological evaluation methods into the process of macro science and technological management, which is also regarded as an important breaking through of the reform of management on the Chinese National Key Science and Technological Plan. It has resulted in the foundation of the Center of the National Center of Science and Technological Evaluation. Studying international experience, coping with the Chinese realities, the National Center for Science ant Technology Evaluation (NCSTE) has made relevant and proper evaluation rules and indicators systems. The national Center for Science and Technology Evaluation has made a series of effective evaluations on the effectiveness of national key science and technological projects; the capacity of innovation, as well as the operational mechanism of the R&D insititutions and industrial bases. Although, those activities are still explorative, people have seen the importance of the adoption of evaluation for the improvement of macro level management, the reinforcement of the responsibility of the government and public resource utilization, which are shown in the following aspects: 1. Improve the process of decision making more scientifically: The utilization of evaluation has optimized the process of decision making. The professional agencies on the evaluation, the utilization of standard evaluation programs OPENING SPEECH 3 and technical ways have provided operational characteristics, the professional evaluation, plays foundation for the scientific decision making. more and more important roles. In the process of evaluation, the collection of information from multi-channels,different angles Recently, the central committee of CPC and the have provided more objective and comprehensive State Council clearly appointed out in the document references for the scientific decision making. of « Decisions on the Enforcement of Technological The technique based on the objective facts to Innovations, Promotion of hi-tech, and industrial- get rid of personal bias, has provided warrantee ization of hi-tech, the evaluation agency system on the justice of scientific decision making. should be promoted in the near future. Since the capacity of evaluation is still far short from the 2. Improved the capacity of adjustment of science requirement of development of the socialist economy. and technical macro-management: The intro- Thus, we are now making efforts on the building of the duction of technical evaluation methodology Chinese science and technological evaluation system. has improved the scope and depth of the viewing of management department. The third For the promotion of the healthier development of the party role and standard evaluation methodology, course of Chinese science and technology evaluation, make it possible for the management department a management rules and relevant counter measures to understand the real situation of the evaluated, have been under research and making, as well as the in order to find the problem and key for the technique criterion and standards, thus to make improvement. The evaluation analysis based on effort on the perfection of the evaluation system, the facts, have provided references and reasons and improve the quality of the institutions professional for the management department for the and quality of their works. improvement of planning management, adjustement and allocation of resources. We believe, experts and scholars from all over the world gathered together to exchange experience, 3. Promoting the innovation on the science which will greatly promote the building of the and technological management system: The capacity of evaluation of China and delegates’s introduction of evaluation methodology has countries. I appreciate very much the active spirit separated the judgement from decision making, suggested in the handbook of UNDP Evaluation which has restrained various unhealthy trends, Office, that is « Study ». Evaluation is a process of such as curtness, personal favor, the lack of study, a process of fighting for the realization and information and poor data etc. It is also helpful re-study on objective matter. Not only the manage- to prevent corruption of the government ment, but also the people undertaking evaluation, departments. To understand the objective reality and the evaluated, all have the opportunity to via the professinal agencies, the government increase their knowledge over the objectives in the departement may focus on the decision making process of evaluation, from which to acquire new process. This has changed the old system from knowledge, thus to improve the capacity to master point to point management to a new triangle the objective and reform the objective. The evaluation balance system,which has provided guarantee of science and technology has just started in for the correctness, effectiveness and safety of China, thus we should even emphasized more on the investment from the government. the spirit of study. One is to actively study the lessons internationally; the second is to cope with the 4. Reinforcement of the authority of the making characteristics and reality of the China. Thus to and enforcement of the National science plan: develop scientific and democratic principle. I believe The establishment of the process of science and it is the same way for the other countries. technological evaluation, has helped to improve and attach importance to the plan contents for Ladies,gentlemen, and friends, the autumn of Beijing all levels and aspects, since it is from the third is with a good fame of « Golden Time », the sky is angle, based on the content of the plan,and blue and high. It is a season full of harvest and verified with the reality. This practice helps to prospective future. For a long time, the Chinese prevent people from making false for the government, the Ministry of Science and Technology approval from the government, thus to raise of PRC enjoyed a broad,friendly and good cooperation the quality of project approval. It is also helpful with UNDP, the World Bank and the other countries. for promoting the management staff keeping The opening of this meeting is not only a fruit of on study the latest development, and undertaking our past cooperation, but also a new page for the investigations. And it is helpful for monitoring cooperation in the future in the field of evaluation. the enforcement, and for the self-conscienceness Now,please allow me to wish success to the symposium, for the enforcement of the plan accordingly. and wish every delegates to enjoy a harvesting time in the Beijing « Golden Time» together. Generally speaking, in the process of setting up our socialist market economy with the Chinese Thank you. 4 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing The New Dimensions in Evaluation and Evaluation Capacity Development 2 Khalid Malik,Director EO, UNDP When evaluators get together at a meeting, there is an inevitable interest in sharing information on concepts and good practices. And this meeting in no exception. However, this meeting is also an opportunity for us to collectively “push up” in the national and international agenda the relevance and value of evaluation concerns as an integral part of decision making and strategy setting. It should allow us to reinforce the need for evaluation capacity development at the country level, by building professional alliances and by broadening our understanding of the range of institutional capacities, strategies, resources and directions. The dialogue started here should also help identify where genuine demand exists for evaluation to assess effectiveness and relevance of public action. It will also set the basis for exploring various modalities for the donor community to support ECD programmes in Asia. THE CASE FOR EVALUA T I O N The 1999 UNDP Human Development Report is on globalization. While documenting the different forces promoting globalization and its likely consequences, it also makes a point that a lot of the momentum we see today in the world is the result of conscious decisions taken to promote an open trade and competitive environment. In a rapidly globalization and increasingly uncertain world,taking correct decisions and learning from experience become essential for survival of the firm or organization or indeed of the nation. Evaluation and the role of evaluators become more, not less important. But to grasp these new challenges, the field of evaluation itself has to develop and become more relevant to the need of the day. As many speakers are going to talk about at this conference, the field of evaluation is changing rapidly, partly as a consequence of dealing with the new demands being placed on it. Increasingly, evaluation is being recognized as an essential part of good governance and public sector reform,and in particular in promoting transparency and accountability. Some go further and see it as part of responsible civic action and civil society development. Let me in this paper focus on probably one of the most promising areas which has recently emerged, and, that is the interface between results-based management and evaluation. This is the future and the areas which represent enormous tech- nical and methodological challenges for all of us. Generally, it is fair statement to make that taxpayers, whether they finance domestic or international agencies, in donor or recipient countries, are ultimately The Ne w Dimensions in Evaluation and Evaluation Capacity Dev e l o p m e n t 5 interested in results. Public sector performance also try to connect individual efforts with the has been an issue among citizens of industrialized macro picture, i.e.outcomes, the larger purpose countries for some time - more during the 1980’s of why we are doing things, whether it is to and 1990’s, as taxpayers have challenged govern- reduce poverty or promoting science and ments to demonstrate value for money in the technology. This approach raised many provision of public services. And, increasingly, methodological issues such as those concerning taxpayers are questioning the very relevance of attribution, when assessing the contribution institutions themselves and their mandates in a of different groups to the realization of a world for rapid change. Are police forces actually specific outcome. leading to better safety and reduce crime, how do related socio-economic forces affect the pictures; But the basic idea is to move away from a pre- are R&D institutions actually leading to products occupation with individual operations towards and services in the market place, etc. assessing the outcome and eventually the impact of development interventions. This At the organizational level, similar questions can approach puts the notion of results in the also be raised: what is the value of a project or centre of planning and management efforts programme or indeed the entire organization; and and therefore “managing for results” emerges are the expected results being attained. as a consequence of this approach, with This enormous pressure on demonstrating results performance systems geared to delivering on has led to the introduction of results-based man- results rather than an introverted risk-averse agement in many domestic agencies,mostly in the culture that stresses the control and management DAC countries, and increasingly international aid of inputs and activities. agencies. UNDP is no exception. This in turn has produced a sea of change in the methodologies 2. Strategic Partnerships: Producing outcomes within UNDP by which we evaluate and assess require the concerted effort of many actors in performance. Benchmarking, indicators,measure- society and institutions - the government, the ment are becoming familiar buzzwords in the civil society and relevant aid agencies. If our evaluation community. focus is now on outcomes, then clearly we have of necessity to invest in partnerships The challenge for international aid agencies has which in turn requires us to have a much been to revisit their M & E arrangements, looking clearer idea of our own strength and the con- for ways to develop effective learning systems and tribution we can make to the partnership. to support their partner countries. Efforts to reshape This approach fits in well with the idea that the development assistance business are also putting development is a broad-based multi-faceted pressure on evaluators to re-think and upgrade their exercise. Importantly, it provides a specific work in capacity building in programme countries. purpose to the need for partnerships. In some ways, we are really moving towards the notion of “shared accountability” which T O WARDS A COMMON FRAMEWORK recognizes that valuable roles are played by many actors in society, from NGOs to local Evaluation is changing and is likely to change even associations and so on. There are methodological more. A shared framework is necessary so that challenges here, for instance how do we access evaluation efforts can build on synergy and promote “shared accountability” to the issue of how broad knowledge sharing. Drawing upon ongoing best we promote national capacity building in work, some thoughts for such a framework could evaluation that includes the representatives of be highlighted. The key building blocks may cover: civil society, indeed as part of our aims of strengthening transparency and accountability 1. Moving towards the notion of outcomes so in society. that the larger developmental purpose could be adequately addressed. In some ways, at least 3. Treating evaluation as a key part of good internationally, the global conferences sup- governance. I will not elaborate on this since ported by the UN provide a strong basis for several other speakers are talking about it. follow-up and assessment at the country level. But, a brief comment may still be in order, evaluation is a natural analogue to strategic More broadly, the focus on outcomes implies planning and management and the basis for that we not only look at the micro picture but sound decision-making. 6 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing 4. And this brings me to my fourth point, we separate the two. In the UN, we feel M have to provide a better fit between evaluation goes with E, since the focus is very much on and decision-making through what is being learning. Objective, discrete evaluations, referred to in the literature as real time or just are part of this search for learning benefiting in time assessments. For too long, we have from a structured approach to interacting not adequately responded to the criticisms with different stakeholders and trying to that evaluations are done late and are as such form a considered response to the famous not of much relevance to decision making and evaluator’s ‘so what’ question, you may strategy setting. A large part of this is due to have carried out a project well, but ‘so the fact that the traditional focus of evaluation what’, what does it amount to. has been on projects. We have to move beyond this and perhaps venture into areas c) Link micro assessment with the macro that hitherto have been seen as off limits to questions. Traditional evaluation tries to evaluators. To become relevant, evaluators form an image of effectiveness in terms of have to take on additional, more difficult efficiency, sustainability and the achievement responsibilities, of tracking management of objectives,essentially at the micro-level. responses and contributing to the conditions Moving to next level may not be easy, which may lead to the implementation of but it is increasingly becoming necessary. evaluation recommendations. We have to determine the best ways of collecting and assessing empirical evidence 5. Refining evaluation methodologies in light of so that future policies and programmes the emerging new role of evaluation in a draw upon the right lessons and that “results-based environment”: organizations are more focussed on doing the right things even as mandates change a) Asking the right questions:The questions and we all try to adapt to a fast changing that evaluators ask may well determine environment. the answers- this may be obvious at the end, but not always at the beginning. So Since we are in China, let me close with a quotation how do we construct an approach, which from the sage Confucius: “If concepts are not clear, ensures that the important questions do words do not fit. If words do not fit, the day’s work cannot get asked? be accomplished. If the day’s work cannot be accomplished, morals and arts do not flourish, punishments are not b) Trying to find the right balance between just. If punishments are not just, the people do not M&E. There has been a tendency to know where to put hand or foot” (Analects XIII). The New Dimensions in Evaluation and Evaluation Capacity De v e l o p m e n t 7 ASIAN PERSPECTIVE IN EVALUATION CAPACITY DEVELOPMENT: The Challenges of the New Millennium 3 Adil Khan, Senior Advisor, M&E,UNOPS/UNDP, Sri Lanka I. INTRODUCTION The nexus between Evaluation Capacity Development (ECD) and good development management cannot be ignored.However one defines it, the emerging globalization process presents a new context for all activities including ECD. The Human Development Report 1999 stressed that as an outcome of globalization;“Shrinking space, shrinking time and disappearing boarders are linking people’s lives more deeply, more intensely, more immediately than ever before”.Therefore, the dictum “when a butterfly flutters its wings in Shanghai, it creates a typhoon in San Francisco” is probably more valid than ever before - meaning that the current situation imposes on each country the complex responsibility of tracking and monitoring changes, both global and local with utmost alertness and developing capacities that can help responding to these changes with equal promptness. The effects of globalization have been both good and bad for every country including that of the Asian economies. However, the recent upheavals in some of the Asian economies drive one home truth - that every country’s ability to tackle the emerging challenges will very much depend on how well they can organize within themselves a good learning mechanism - a mechanism that is capable of tracking both progress as well as crises and the effective way in which the lessons learnt are feedback to the planning, policy and reform processes of a government. The world today has more opportunities for people than ever before.Child death rates have fallen by half since 1965 and a child born today can expect to live a decade longer than a child born then. During the last three decades,school enrolment in developing countries has doubled. Adult literacy has increase from 48% in 1970 to 72% in 1997. The world is definitely more prosperous, and a better place to live now than ever before. In the past 50 years, average per capita income tripled. The share of people enjoying medium development has risen more from 55% in 1995 to 66% in 1997 and the share of low human development fell,during the same period,from 20% to 10%. THE CONTEXT The current globalization process is indeed a very important context of evaluation. As has been explained above, with the promise of gains, globalization has also increased uncertainties,making it imperative the need for continuous monitoring and assessment of emerging situations. Likewise, the nagging problem of poverty which consistently challenges the gains of development presents the development context of evaluation. In this context, evaluation is seen as an important tool to learn lessons of experience and help ensure growth with equity. Similarly, post ASIAN PERSPECTIVE IN EVALUATION C A PACITY DEVELOPMENT: The Challenges of the New Millennium 9 implementation sustainability of development The story is not very different with the other projects and weak sustainability management of major regional development bank either- the invested resources present another important Asian De velopment Bank’s (ADB) recent studies context of evaluation. It has been argued that confirm that despite experiencing more success with post implementation monitoring and evaluation disbursements, successes with project sustainability of completed projects has the potential to are less satisfactory (Asian Development Bank, greatly enhance their sustainability. Problems 1993).In fact, recent trend shows that less and less of accountability, corruption etc. provide the projects are being sustained. The implication of governance context of evaluation. the latter is that while the debt burden of the recipient countries has been rising their capacity to It is evident that despite the phenomenal progress earn income on these borrowings and service debts made during the last two or three decades, these on borrowed resources have been falling, leaving a benefits of globalization did not spread evenly. The mounting debt burden on the future generation. 1999 HDR reports (UNDP, 1999): s Nearly 1.3 billion people do not have access to These and the lingering problems of poverty, clean water. falling governance standards and degradation of s One in seven children of primary school age environment make it imperative that more attention are out of school. About 840 million are be given to the aspects of: (i) improved planning; malnourished. (ii) prudent and transparent use of existing resources; s An estimated 1.3 billion people live on incomes (iii) creation of a social infrastructure, allowing of less than $ 1 (1987 PPP $) a day. freer and wider participation of people in the decision making; and (iv) to complement the In addition to the above, it is also becoming apparent above, establishment of an institutional framework, that many developing countries in their quest for an evaluation system, that enables achie vement of rapid economic development have also been bringing good governance and sustainable development and about, perhaps inadvertently, the most wanton allows continuous research, lessons learning and management feedback. In a recent paper, Mr. destruction to their environment. These disap- R. Picciotto (1999) of the World Bank echoes pointing trends which are concurrent to some of similar sentiments and presents the following as the more positive outcomes of globalization are the contemporary context of evaluation - “Accountability, not only compromising the gains made so far, but transparency, participation and the rule of law have even threatening their sustainability, which indeed become generally the standards of economic gov- indicate that serious efforts are needed to review ernance” and according to him, these elements constantly its dynamics.The concomitant changes, form the basic framework for evaluation. Late both positive as well as negative need to be ana- Mahbub Ul Haq, the architect of the concept of lyzed constantly and that timely and appropriate “human governance” extends implicitly the context feedback given.It is expected that such continuous of evaluation to equity issues and states, “Every monitoring of progress will help ensuring on the governing institution, every policy action should one hand the sustainability of the gains made and be judged by one critical test. How does it meet on the other, mitigate the losses incurred therein. the genuine aspiration of the people” (Human Development Centre, 1999). It is only through Against this backdrop one has also to recognize evaluation that such a “critical test” can be installed that while worldwide the investible resources are and eventuated. declining, demand for services are increasing implying, continuous need for prudent use of available resources and maximization of return on MONITORING TO EVALUATION: the invested ones. However, several recent studies THE CHANGING PARADIGM OF ECD suggest that large amount of resources invested in development projects did not always meet with Until recently, however, most donors as well as the expected results. Some facts of life (World government attention on ECD seemed to have Bank 1993): focussed more on monitoring than on evaluation. s a recent World Bank study indicates that, This has been due (i) firstly, to unsatisfactory during 1981-1991, the proportion of successful experience with implementation;and (ii) secondly, projects has fallen from 85% to 63%, globally. the belief that early implementation of projects s the same study reports that the proportion of will automatically contribute to generation of major problem projects had increased from expected results. However, all available evidences 11% in 1982 to 20% in 1991. suggest that while monitoring did help speeding 10 E V A L UATION C A PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing up implementation, these did not automatically In summar y, concerns with efficient use of public produce desired results. On the contrary, in resources,transparency and accountability, changing some cases these actually acted predatorily - some conditions with globalization, lingering problems development projects made things worse for the with poverty and concerns with good governance poor and in some cases, the environment got e t c ,a re making it increasingly obvious that efficient degraded.1 In these countries, absence of an institution and effective evaluation of social and economic of evaluation deprived the decision-makers from dynamics and of public sector programmes and learning the truth and correcting the situation policies are a sin qua non to good development before it was too late. Further with globalization management and an important tool to help facing and with integration of the national economies to the new challenges of the coming millennium. the global system, many assumptions based on which projects and programmes are formulated tend also to become increasingly invalid - making E V A L UATION CA PACITY DEVELOPMENT (ECD): THE CONCEPT it imperative the need for continuous information gathering and lessons learning and making adjustments ECD can be defined as an activity or a set of activities to project conditions. that contribute to the establishment of evaluation There are thus, a variety of reasons why more than capacities within the development administratio n ever before evaluation should be taken seriously structures of developing countries. Development and be treated as an important tool in development of such capacities involves evolution of systems management. Such as: and methodologies that assist lessons learning from on-going or past projects and programmes I. With increasing democratisation of the societies, and, through these lessons, adjust projects and individuals are becoming more conscious programmes in such a manner that they achieve about their rights and demanding better services their planned objectives or improve the quality of and more value-for-money from public expen- design of similar projects in future. diture. There is only one way a government can know whether what they are doing, are in The World Bank argues that ECD enables its fact, measuring up to the expectations of the member countries to develop suitable tools to people - through evaluation. ensure “quality of its loan portfolio” (World Bank, 1994: iv).The Bank thus sees ECD as a part of an II. Increase in population is creating increasing overall exercise in public sector reform. It justifies demand for more services, but resources are the need for strong ECD by stating that: not increasing at the same rate. Evaluation helps to save money and ensure best use of “Effective evaluation - both “concurrent” (i.e.while existing resources. a programme or project is in progress) and “ex-post” (i.e. after the programme or project is completed) - III. With globalisation and liberalisation of national enhances the quality of public investments and of economies, institutions, relationships, pattern the Bank’s portfolio. Well focused and properly of production, pattern of economy and social timed evaluation can provide the information behaviour etc. are constantly changing, making needed to bring about mid-course corrections in it extremely difficult for the governments to programmes and projects, analyze and resolve control and manage events to its own advantage. systematic policy issues, the design of future Constant vigilance through monitoring and operations, and increase country ownership” evaluation and research of the changing events (World Bank, 1994: iv). are seen as some of the important imperatives of good management in the new millennium. Most other donors generally agree with the Bank’s IV. Evaluation ensures transparency and account- ECD strategy, although UNDP as well as some ability in public finance, which are also key bilateral donors wish to focus more emphatically elements of good governance. on the equity aspect of evaluation, and not limit the tool to assess financial and economic sustainability V. Evaluation can also be politically rewarding to of projects and programmes only. In their view, it a government which is benevolent - for, an is important that each evaluation looks more candidly evaluating government is also likely to be at the distribution aspects of project benefits viewed by its citizens as a listening and a and through lessons learnt consistently supports caring government. programmes and policies that help achieving 1 Alauddin and Tisdell (1989) explain how investment in shrimp culture in Bangladesh pushed small and marginal farmers out oftheir land, made rich richer and poor poorer and degraded the environment. ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the New Millennium 11 human development objectives. To ensure that of good governance – “the right to know” by people equity is incorporated as an important agenda in might also be guaranteed. These arrangements evaluation research, these donors also stress the functioned well for a while, but with the changing need for incorporating the beneficiaries into the political and moral environment of the country evaluation process.Thus, according to this school, such noble innovation of the bygone leadership participatory evaluation (mainly as a sequel to could not be sustained. In recent years, PEO though participatory development) should form the basis continues to undertake evaluation work, weak of all future ECD work. However, as corruption political commitment to its works seemed to have and inefficient use of public funds are also becoming made some of its initiatives somewhat less tenable. concerns of many donor agencies, the aspects of Nevertheless, the PEO model leaves one important transparency and accountability are being regarded guidance which is worth paying attention to, the as important subjects of evaluation as well. provision of wider dissemination of evaluation information has indeed the potential to assure maximum feedback, an aspect which continues to remain a problem in many evaluation systems built in Asia. However, around this time another important I I . THE ASIAN EXPERIENCE trend in India to which a parallel similar to that of AND THE EMERGING ISSUES growth of evaluation could also be drawn. As in many things in development, Asia also played While with the eroding idealism and falling standards a pioneering role in setting up formal evaluation in governance many institutions of checks and balance practices in many of its countries. Interestingly started to become weak in India, the institution enough, some of these initiatives came from within of Judiciary emerged as the only institution that and not from outside. Presented below are some of continued to serve the interest of the people, the key lessons of ECD in Asia. independently and fairly. But what made it possible? Many believe that two things worked in favor of Judiciary - one, the democratic environment within THE PIONEERS: THE INDIAN AND THE MALAYSIAN EXPERIENCE which the institution operated allowed transparent expression of its views, and second ly, the political Perhaps the most radical of all evaluation capacity and administrative immunity that the institution building initiatives came from India (Khan,1989). received from or been endowed with, by the Inspired by the post-colonial idealism,the political constitution also helped the institution to operate leadership of the day, led by Pandit Nehru set freely and openly. By drawing lessons from this, about to form a benevolent governance structure in many tend to argue that evaluation being a vital the body politic of the country. Search for truth tool of check and balance and indeed, of develop- and anxiety to learn from facts to help improving mental justice, as well as that of public sector the conditions of its citizens, especially that of the accountability, its institutional arrangement cannot poor and the disadvantaged came as a natural simply be left to the monopoly of governmental choice. As early as 1950, a Programme Evaluation whims. Rather, evaluation should be seen as an Organization (PEO) was created within the integral part of good governance if not, of human Planning Commission of India. Its mandate was development and thus, its institutional arrangement to undertake post-evaluation of completed projects be hoisted in such a manner that it enjoys legislative and give feedback to the planning process. Among immunity.2 However, prevalence of a democratic other things, the most impressive aspect of this environment in the country is equally important to initiative has been the way the arrangements were ensure free and fair dissemination of information. made to ensure feedback. To guarantee maximum attention, transparency and response to evaluation Malaysia’s monitoring and evaluation capacity findings, provisions were made to circulate the building, which also came from within, was started evaluation reports to the widest possible audience, in 1960, under the direct patronage of the then including the Public Accounts Committee, the Prime Minister Tun Abdur Razak (Khan, 1999). Parliament Library and the Press.It was envisaged Established mainly to speed up delivery of services that wider circulation of evaluation findings of in the rural areas, the Malaysian M&E became government programmes would not only ensure attractive to many Asian governments, who like maximum response, but by making provision to their Malaysian counterpart were also grappling circulate these reports to the Press, a key element with the problems of implementation. In course of 2 Reportedly, the evaluation organisation of Brazil has been established with constitutional backing and statutory funding which is helping it to operate independently and consistently, regardless of changes to governments and their attitudes. 12 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing time, government also developed a separate agency However, results of these ECD initiatives have for evaluation (the Socio-Economic Research Unit - been somewhat mixed. In many Asian countries SERU), but this has since been disbanded and evaluation continues to receive low priority. In evaluation as an activity has now been re-established some, evaluation has a stand-alone position with as a small unit within the Economic Planning Unit little effect on the planning process. Lessons learnt (EPU), the central planning agency of the government. are not fed into formulation of new policies and programmes. Nor, the evaluation and the planning Presently, not much is known about Malaysia’s units are institutionally linked to enable formalization M&E. As far as monitoring is concerned most times of an evaluation feedback. are spent, almost on a daily basis, on monitoring those projects deemed important by the Prime Minister. There is, however, a Computerized reporting CIVIL SOCIETY INVO LVEMENT IN EVALUATION: system to which all projects regularly report physical THE SRI LANKAN INITIATIVE and financial progress. But, it is not known, Despite these disappointments, in some Asian whether the current system goes beyond physical countries, for example the civil society in Sri Lanka, and financial monitoring and assesses the results. seemed to have taken initiatives to broaden the institutional framework of evaluation beyond the government and took initiatives to establish the DONOR SUPPORT TO EVALUATION INSTITUTION BUILDING Sri Lanka Evaluation Association (SLEA) in September this year. In addition to these two pioneering initiatives, other Asian countries that established fairly With support from UNICEF and UNDP, several permanent institutions of monitoring and evaluation University teachers, government researchers and are -Indonesia, the Philippines, Sri Lanka, Nepal, private sector consultants took initiatives to establish Thailand, China, Bangladesh and South Korea. the Association. Dedicated to de veloping more of Some of these countries also received a variety of a culture than a mere system of evaluation, the ECD inputs from a range of donor agencies. For SLEVA among others, gave itself the following example UNDP intervention in 1991 saw the key objectives: (i) to function as an advocacy group creation of the Central Monitoring and Evaluation for growth of evaluation culture in the country; Division (CMED) in Nepal in the National (ii) to help establishing ethics and standards in Planning Commission. The ADB intervention in evaluation; and (iii) to establish national and 1992 in Sri Lanka saw the establishment of the international networking for information exchange Performance Evaluation Unit within the Ministry and capacity development. of Plan Implementation and Parliamentary Affairs. In 1993 World Bank supported the Establishment and nurturing of groups such as Ministry of Finance in China to establish a broad SLEA has to be the most exciting innovation in promotion of an evaluation culture in a society. based institutional framework for undertaking Appearing as an advocacy group such an association evaluation in the country. During early to mid has the potential to provide vital link between the 1990s both UNDP and the Asian Development civil society and the state and pursue evaluation on Bank continued to provide ECD support to demand basis. Such groupings which enlist various Asian countries.3 While the UNDP memberships of individuals and institutions of support focused mainly on results based monitoring both the government as well as the non-government and evaluation and pursued introduction of and academic institutions may equally help in participatory methodology in evaluation, ADB establishing a collegial link between the government introduced what it calls the Project Performance and the civil society and in many cases, help removing Management System (PPMS).4 In those cases the misgivings associated with evaluation. where ADB felt that the countries were making significant progress in ECD, it followed up in those countries with second and third generation TRAINING: BROADENING THE OPTION assistance. The Philippines, Sri Lanka, Thailand, Nepal and China seemed to have fallen under Another useful lesson learnt in ECD in Asia is these categories of countries. that while these initiatives helped training a large 3 UNDP initiatives in preparation and publication of Country Monographs on National Monitoring and Evaluation Systems equally helped in sensitising and informing the key stakeholders about the situation ofM&E in their countries.This analytical e xercise provided useful foun- dation for future ECD. 4 PPMS basically means management of the Project Cycle, with inputs from evaluation. ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the New Millennium 13 number of officials in evaluation methods and are also contentious arguments with regard to techniques,these gains could not be sustained fully “indicators” of success. It has been stated in a over time. Three main reasons are given for this recent study that in the Philippines many evaluation lapse – (i) frequent turnover of staff from M&E studies - mostly undertaken by academics - are institutions to other agencies; and (ii) secondly, rejected by program managers due to disagreements poor staff quality; and (iii) thirdly, absence of facilities with one or more of the above factors (Callanta in conducive to application of evaluation skills learnt Khan,1989). Callanta suggests that the absence of through ECD. Moreover, training in evaluation a wider pre-evaluation stakeholder analysis and continues to be an ad hoc, oneoff activity. Further, lack of adequate consultation with regard to the most government training institutes that received need, objectives and methodologies of evaluation ECD inputs, tend generally to operate in isolation risk both the credibility as well as the ownership from the other non-government academic institu- and the use of evaluation.5 tions and the Universities, particularly from those (overseas institutes) that are engaged in innovative In recent years the introduction of rapid and research in evaluation (e.g. participatory evaluation participatory methods in evaluation has caught the etc.). Absence of this institutional linkage tends to attention of many and indicates an increasing make it difficult for the government M&E train- willingness to apply rapid methods to conduct ing institutes to maintain required standards. It is, evaluation. Evaluators of developing countries therefore important that future ECD work in require extensive training in these methods. But, training consider on the one hand, creating better at the same time, they also need to bear in mind linkage between the government and the non-gov- that the rapid method is not a substitute for ernment training institutes and, on the other, the rigorous method , rather, one complements the involve the national level universities and non- other and the choice of one method against government research institutions to incorporate the other would very much depend upon the courses within their curricular, and build research evaluation questions that one would need to answer.6 capacities in evaluation.Many believe that incorpo- ration of Universities in ECD is likely to yield fol- In the matter of methodology training in ECD, lowing important gains:(i) Firstly, the Universities, some donors tend to introduce their own in-house unlike the government training institutes will have jargons which seem to create some confusions built-in incentives to run and improve courses as among the developing country practitioners. well as undertake research and publications in evalu- ation; and thus ensure better the aspect of sustain- Recently, in the arena of evaluation methodology ability of ECD inputs; and (ii) secondly, continu- there are discussions about involving the beneficiaries ous training through its graduate and post-gradu- into the evaluation process. This undoubtedly is a ate courses shall guarantee stream of graduates positive initiative and has the potential to enhance trained in evaluation, thus consistently filling the ownership of evaluation at all levels. Such emphasis stock of qualified evaluators in a country. will also enable gathering of more qualitative information. However, in societies where benefi- ciaries are divided by unequal power structure and METHODOLOGIES AND TECHNIQUES: where socio-economic relationships are unevenly M OVING TO NEW DIRECTIONS structured, the ability of the evaluators to access the targeted beneficiaries may prove somewhat Suitable methodologies to measure substantive difficult. In such situations some non-threatening aspects of a project - that is, project effects and interactive methods of data gathering may become impacts, remain a matter of highest concern in all the obvious choice. Training in interactive and evaluation work. First of all,there are complaints that non-threatening methods of data gathering can be evaluation research is too costly and that it takes another aspect of future ECD activities. too long to be of any benefit to management. Secondly, debates also occur with regard to sampling With regard to participatory evaluation, Chambers and research designs that are capable of making (1997) talks about “empowerment evaluation” which “before-after” and “with-without” comparisons.There involves incorporation of the beneficiaries right 5 Yee (1999) talks of similar difficulties when his team wanted to evaluate the impact of Asian Development Bank assistance in China (The study was called, Country Assistance Programme Evaluation -CARE) and concludes that wider participation ofprogramme stakeholders and adoption of qualitative assessment principles help assessing impacts in a more “plausible”manner. The study also introduced a agreed-upon simple development effectiveness index to assess success. 6 A recent concept paper, Impact assessment ofUNDP Country Interventions,argues that qualitative studies (quite often the, rapid and partic- ipatory studies) are useful pre-requisite to quantitative studies. However, the paper cautions that, “by relying upon qualitative analysis alone, one risks taking perception for reality” . 14 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing WHAT IT TAKES TO MAKE IT W O R K Monitoring and evaluation (M&E) systems - especially evaluation systems are neither easy to introduce nor are they easy to sustain, especially a good quality system. A recent United Nations Development Programme (UNDP) study - Generic Issues in Monitoring and Evaluation. What Works and Does Not - reflect that success and sustainability of an M&E system depends on a variety of factors, including (Khan, 1993): s political support and its continuity. s suitable institutional framework with adequate feedback linkage. s staff training and logistics support. s demonstrated benefits of evaluation. s democratic governance environment. s develop a “culture” rather than a system - Link civil societies. The study argues that all of these factors are equally important and that weakening of any one of these factors may has threaten the success of the entire system. For example,the study reports that while political support at the introduction may assist in introduction and institutionalization of a system, if that support is not maintained (by way of use and backing), its operational efficiency and effectiveness will also ultimately weaken. Similarly, a well laid out institutional arrangement will not yield the desired result if it is not backed up by good quality staff and a good feedback arrangement. The study concludes that for M&E to be successful,all the factors that have been identified as key to the success of the system must be equally emphasized and provided with. A lapse of any one element, the report repeatedly argues, will lead to the collapse of the entire system. at the planning stage where they also become indicate that more emphasis should be given to the principal evaluators and through evaluative rapid methods and innovative research designs. participation, empower themselves to drive the Involvement of Universities/research institutes implementing process to targeted results. 7 in evaluation research may also be seen as an important part of future ECD activities. The past ECD works in Asia have also identified a variety of operational problems that affect administration of efficient evaluation. Principal FEEDBACK: DEMOCRATISING THE PROCESS among these is the lack of basic project data. In many countri e s ,p roject offices are either dismantled Feedback relates to methods of linking evaluation or reduced in size after a project has been completed findings to the decision-making process of the and most of the staff get shifted to another project. government, particularly its planning process. In These changes affect undertaking of quality evaluation. Asia different types of methods and channels are Moving staff away not only means project knowledge currently being used to disseminate evaluation is lost but in many cases, even the project documents information. The most commonly used methods get lost. Further, the lack of a Project Implementation are: (i) reports/returns; (ii) review meetings; (iii) or Completion Report (PCR) equally constrains the Workshops/ seminars; and (iv) newsletters and opportunity to capture updated data at the time of computer networking. As a result of ECD inter- completion of implementation of a project. Efforts ventions,some countries have tried to institutionalise to build new evaluation capacities in developing feedback arrangements, without much success. countries must consider ways of collecting and Further, many of the evaluation organisations collating project information in one place. seem to continue to function in isolation from the Establishment of a documentation centre with briefs planning institution. 8 This institutional gap seems of completed projects will greatly assist in alleviating to be defeating the very purpose of evaluation. It this problem. Further, active involvement of bene- was stated earlier how India had a system of sharing ficiaries into the evaluation process,is also likely to evaluation information with a wide audience help retention of data at the grass-root level. including the Parliamentarians and the general public. However, falling governance standards, In summary, it appears that as far as methodology weak government commitment and politically is concerned, there appears to be a lot of room for vulnerable institutional arrangement seemed to have improvement and problems with time and money weakened that resolve. Increased democratisation 7 In Bangladesh,BRAC’s (an NGO) Rural Development Programme seemed to have incorporated this methodology quite successfully. 8 For example, in Sri Lanka the Performance Evaluation Unit of the Ministry of Plan Implementation is not institutionally linked to the planning process. New policies and programmes continue to be de veloped without much consultation with the Performance Evaluation Unit. ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the New Millennium 15 of societies, legislative enshrinment of evaluation is so vital to good governance. institutions,involvement of civil societies and most s institutional development in evaluation importantly, development of democratic culture in a capacity has been viewed by many as purely an country seem to represent the important elements, aid management ritual and not as a tool for if not the key challenges of evaluation capacity good development management. To ensure development and feedback in the new millennium. sustainability of evaluation, its institutional Improvement to evaluation feedback must go hand- development (both as an organization as well in-hand with other democratic reforms in a country. as a function) may require legislative backing. Prevalence of a democratic environment in a All in all, while recent ECD works made important society may also be seen as another important contributions in introducing and initiating evaluation element of evaluation capacity development. practices in many Asian countries,further works are s limiting evaluation training purely within the necessary to make these interventions institutionally government training institutes encountered better structured and operationally more efficient. usual difficulties associated with many There is indeed a need for holistic thinking on ECD government institutions of this nature - limited and further work on this aspect of development research, limited exposure to outside world, management should weigh carefully the reform low incentives for high standards etc. risked process within which each society is evolving. Work quality and sustainability. on ECD should contextualise the work on democratic s poor staff quality often contributed to poor reforms that are currently being undertaken in many products coming out of evaluation, affecting countries and support institutional development in the credibility of the organisation and even evaluation in such a manner that the activity itself the merit of the practice. becomes both an agent of change as well as the s inappropriate and costly methodologies beneficiary of such a charge. contribute to low reliability and poor acceptance of evaluation. s inadequate or poor feedback remains a nagging problem. I I I . CHALLENGES OF ECD IN THE NEW MILLENNIUM THE OPTIONS Overall, it is reasonable to state that the most The new ECD measures must consider these impressive aspect of ECD interventions has been difficulties seriously and come up with solutions, the recognition to and establishment of separate which are consistent with the changing political evaluation units in most developing countries, and economic environment of the world. It is in especially in Asia. There are also evidences that this context following suggestions are put forward: due to improved evaluation,project performance in some of these countries has also improved.However, s More emphasis should be given to the devel- despite these gains, practice of evaluation as an opment of an evaluation culture rather than obvious and untempered element of good governance just building systems within government is yet to be fully realized. The new challenges institutions.The former would require linking of evaluation stem from a variety of factors. the civil societies - the Universities, research Broadly, these are institutional related, but there institutions, evaluation societies etc. - to the are several organizational matters which also need evaluation process of the government. to be looked at. In this context it is strongly suggested that future ECD measures should include in its THE CHALLENGES various elements the aspect of providing support to the growth of evaluation societies in each As has been observed, a variety of challenges still remain: and every country and through this foster s tentativeness with evaluation structures make a collegial link between the civil society, and these units vulnerable to changing govern- the government. ment moods. s confining institutional development of evaluation s Not just for making evaluation effective, but to purely within government structures,leaves out ensure growth of good governance, continuous a gap and tend to exclude the most important efforts should also be made to systematically actor, the civil society - from an activity that democratise all societies, create an environments 16 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing of free and fair exchange of information and Governments should be made aware of this. give the ordinary people the opportunity to exercise their “right to know and right to tell’. The governments need to remove from themselves It is only in an environment of openness the fear of evaluation. They need to realise that that evaluation and for that matter, all other evaluation is one tool that will help them doing the institutions of checks and balance and lessons right thing at the right time. They need to be learning can grow. shown that it is an inseparable element in the journey to good governance and sustainable s In community development or in poverty human development. alleviation activities, Patton’s “Development Evaluation” (where evaluation is used to continuously monitor change and with lessons REFERENCES learnt make adjustments to achieve goals) and Alauddin, M. and Tisdell,C. (1989), “Poverty Resource Distribution Chamber’s “Empowerment Evaluation” (where and Security. The Impact of New Agricultural Technology in Rural Bangladesh”, Journal of Development Studies, Volume 24 beneficiaries are included in the evaluation Number 4, Pages 510 -570. process to drive project process to intended The Asian Development Bank (1993) Country Synthesis Report on results) methodologies should be given high Post Evaluation findings. The Asian Development Bank. Manila, 1993. priority in ECD agenda.9 Chambers,R. (1997) Whose Reality Counts? Putting the Last First. London Intermediate Technology Publications,1997. s Future ECD inputs in training should seriously Khan, M. Adil (1989) A South Asian Region Study on Current consider (in addition to capacity building of Thoughts and Practices in Monitoring and Evaluation. The government training institutions) providing Economic Development Institute. The World Bank Washington D.C. 1989. support to the Universities/research institutions in building capacities in evaluation training Khan, M. Adil (1991) Initiatives in Efficiency: Experience of Malaysia in Monitoring and Evaluation. Central Evaluation Office. Unites and research. Nations Development Programme. New York. 1991. Khan,M. Adil (1993) Generic Issues in Monitoring and Evaluation: Evaluation can be a highly political exercise. It is a What Works and What Does Not. Central Evaluation Office. game of truth. It can be both pleasant as well as United Nations Development Programme,New York,1993. unpleasant.However, truth must be told truthfully, Patton,M. S. (1999) “Development Evaluation” Evaluation Practice, but constructively. Evaluators need to be sensitive 1994,15 (3),311-319. to the difficulties of implementers and be conscious Picciotto, R. (1999)” The New Challenges of Evaluation Capability of the environment within which they operate. It Development Programme” Evaluation Insights. Evaluation Co- operation Group Newsletter. June 1999 Volume 1,Number 1. must be remembered that role of evaluator is not United Nations Development Programme. Human development that of a development police but that of a develop- report,1999. UNDP. ment facilitator. More and more, evaluation should The World Bank (1993). Evaluation results for 1991. Operations be seen as a resource, a reform agent and as an Evaluation Department. The World Bank. Washington D.C. 1993 . arbitrator of development justice. Constructively Yee, Y.L.(1999). “The ADB’s first Country Assistance Programme used, evaluation can be very effective in delivering Evaluation”. Evaluation Insights. Evaluation Co-operation both equity and justice as well as efficiency. Group Newsletter. June 1999 Volume 1,Number 1. 7 These approaches also provide the general methodological framework for results-based M&E – which basically imply: (i) that the process must be seen as dynamic and flexible and not rigid;and (ii) that evaluation should be treated as a continuous planning tool and not me rely a post- facto activity and institutionally linked as such. ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the Ne w Millennium 17 AID EVALUATION: A Donor Perspective 4 Niels Dabelstein, Head of Evaluation Secretariat DANIDA,OECD/DA CWP In this intervention I would like to address four issues: The DAC Evaluation Group; Aid evaluation from a donor’s perspective; the move towards partner country responsibility and, what donors can do. Bilateral and multilateral aid agencies have a long tradition of evaluation beginning in the fifties and becoming institutionalised and systematic in the mid-sixties.In most donor countries, development aid agencies/ministries established formalised evaluation procedures long before domestically oriented public institutions. In order to enhance the quality and utilisation of evaluations the Development Assistance Committee (DAC) of the Organisation for Economic Co-operation and Development (OECD), established a Working Party on Aid Evaluation already in 1982. The purpose of the Working Party is to: i. Strengthening exchange of information and experience in order to: improve evaluation activities; encourage standardisation of methodological and conceptual frameworks; and improve co-ordination in planning major evaluations; ii. Contributing to improved aid effectiveness through the synthesis of evaluation lessons; iii. Examining the possibility of launching joint studies of aid effectiveness; iv. Seeking ways to promote developing countries’ own evaluation capabilities. Today, it has developed into the leading forum for exchange of information and experience between donor agencies; for development of methodologies; standardising of terminology; and co-ordination of joint evaluation efforts.The WP-EV today consists of representatives of evaluation units in bilateral donor agencies of OECD member countries and the European Commission. In addition, UNDP, the World Bank, IMF, and the regional development banks (AsDB, AfDB, IADB) hold status as observers and participate as such in WP-EV meetings twice a year. The Working Par ty has established an evaluation definition widely recognised and adopted by its members. Evaluation is: “An examination as systematic and objective as possible of an on-going or completed project or programme, its design, implementation and results. The aim is to determine the relevance and fulfilment of objectives, developmental efficiency, effectiveness, impact and sustainability. An evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of both recipients and donors.” Generally, aid evaluation performs several functions: it provides lessons for managers and staff of ongoing aid programmes thereby leading to improvements in imple- mentation; it establishes information and documents experience that can be used in planning and designing future aid programmes; it is used by managers, strategy- and policy makers for revising existing — and for devising new — aid strategies and policies; and finally aid evaluation delivers information on efficiency and AID EVALUA T I O N : A Donor Perspective 19 effectiveness of aid programmes, and thus ensures oriented. The need for accountability, participation, accountability towards politicians and the public. transparency, rule of law has only recently been considered necessary for improving governance Government financed development aid is expenditure practices. Moreover, the demand for evaluation comes of taxpayers money outside national borders. In more often from the donors community (which addition, aid is often spent through partners or co- has limited influence), than from the partner countries’ operating governments over which the donor Parliament or taxpayers. government has no formal control. This is a feature distinguishing aid evaluation from evaluation of It appears that credible evaluation is a function of most other publicly funded programmes. For that good governance i.e. demand for accountability reason alone, the accountability function may very more than of evaluation institution and professional well be the most important for aid evaluation. Not capacity. Recent years’ focus on public sector reform, the least in periods of aid fatigue and pressure on on public expenditures management, on democratic government budgets, continued public support for aid reforms, and on development effectiveness has, I hope, expenditure is contingent upon aid agencies’ account- increased the demand for evaluation as a necessary ability and their ability to document aid results. and useful element of good governance. There are many ways in which donors can support evaluation capacity: Development aid evaluation has continuously developed from an early focus on individual activities s Promote an agency ECB support policy or and projects to broader evaluations of sectors, strategy, particularly in view of new aid programmes, and cross cutting issues or themes such forms being introduced,including programme as gender, environment, institutional development assistance for institution and capacity building and sustainability. Aid evaluations are no longer as part of good governance initiatives at simply about evaluating efficiency, effectiveness national and sector levels. and impact of aid in terms of technical, economic, socio-cultural, institutional and environmental s Advocate and stimulate the evaluation issue in factors, but about evaluating the combined effects country dialogues and sector programme assistance. on developing societies of the complex of political, economic and technical inter ventions by donors. s Provide technical advice and advise on training facilities and materials on evaluation issues. The emergence of new types of aid interventions and the inclusion of novel themes in de velopment aid pose new challenges to aid evaluators. Some of s Support the establishment of twinning those challenges are methodologica l :h ow to evaluate arrangements between other domestic evaluation support to good governance, human rights, civil institutions and host country institutions. service reform, and privatisation. Others are wider ranging: the current trend to move away from s Develop consistent evaluation methodologies project aid and to more varied and flexible modes and terminology. of assistance within a sector framework frequently addressing key policy and institutional problems. s Co-ordinate their evaluation programmes with host countries and other donors in order to These new trends and challenges need to be addressed. optimise the use of resources and the constrained Donors alone should not bear the responsibility for capacity of recipient countries’ evaluation systems. evaluation alone.More emphasis on joint evaluations would be one way. Another is increased emphasis s Arrange joint-evaluations with a genuine par- on evaluation capacity building. Through such ticipatory approach, where the needs of both emphases,many of the future evaluation challenges parties are incorporated from the start and can be addressed. where the capacity building element is taken into account specifically. The developing countries, in their own interest, should take on these challenges and further de vel- In my opinion joint evaluation is one of the most op evaluation capacities to assess the performance concrete tools to enhance evaluation capacity. To of the public sector to which development aid is, ensure that evaluations become efficient learning after all, a small input. tools,promote good governance, enable the partners to be ful ly accountable, and are cost effective they Evaluation institutions exist in many developing must be planned and executed jointly with the countries, but most have little impact on policy recipients. Ultimately, both the donors and the making and management decisions,partly because recipient authorities are jointly accountable both there is little demand for independent and transparent to the taxpayers in developing countries and to the evaluation. Governments are not yet fully results people of the developing countries. 20 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing PART II Evaluation of Performance Management in the Public Sector E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing RESULTS BASED MANAGEMENT IN THE DEVELOPMENT COOPERATION AGENCIES: A Review of Experience 5 Annette Binnendijk,Consultant,UNDP OV E RV I E W This paper provides a broad overview of some key concepts and elements of results based management (also called performance management), which has been a central feature of recent public sector reforms in the OECD countries. While most of the remarks made and issues raised apply generally to government agencies’ efforts to implement results based management, the particular focus taken here is from the perspective of donor agencies. Experience and examples are drawn primarily from USAID, although other donors’ experiences shared at a Workshop on Performance Management and Evaluation held in New York in October 1998 are also taken into account. PUBLIC SECTOR REFORMS During the 1990s, many of the OECD countries have undertaken extensive public sector reforms in response to economic, social and political pressures. Budget deficits, structural problems, growing competitiveness and globalization, lack of public confidence in government, and growing demands for better and more responsive services and for more accountability have all been contributing factors. Popular catch phrases such as “Reinventing government”, “Doing more with less”, “Demonstrating value for money”,etc.describe the movement towards pub- lic sector reforms that have become prevalent in the OECD countries. Often, government-wide legislation or executive orders have driven and guided the public sector reforms. While there have been variations in the reform packages implemented in the OECD countries, there have been many common aspects. For example: s Focus on performance issues and on achieving results s Devolution of management authority and responsibility s Orientation to customer needs and preferences s Participation by stakeholders s Reform of budget processes and financial management systems s Application of modern management practices RESULTS BASED MANAGEMENT Perhaps the most central feature of the reforms has been the emphasis on improving performance – that is, on ensuring that government activities achieve desired results. A recent study of the experiences of ten OECD Member countries RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Re v i ew of Experience 23 with introducing performance management Most OECD governments now see evaluation as showed that it was a key feature in the reform part of the overall performance management efforts of all ten. (See In Search of Results: Public framework, although perspectives concerning the Management Practices, OECD, 1997). degree of integration and independence var y. The view of evaluation as a separate or specialized function, Performance management, also referred to as but integrated into performance management has results based management, can be defined as a been gaining momentum. This is reflected in PUMA’s broad management strategy aimed at achieving Best Practice Guidelines for Evaluation (OECD, 1998) important changes in the way government agencies which was endorsed by the Public Management operate, with improving performance (achieving Committee.The Guidelines state that “evaluations better results) as the central orientation. must be part of a wider performance management framework”. Less emphasis is placed on independence, A key component of results based management is and evaluation is seen as one of many instruments performance measurement, which is the process of used in the overall performance management objectively measuring how well an agency is framework. However, some degree of independent meeting its stated goals or objectives. It typically evaluation capacity is being preserved; such as the involves several phases: e.g., articulating and evaluations conducted by central evaluation offices agreeing on objectives, selecting indicators and or performance audits carried out by audit offices. setting targets, monitoring performance (collecting data on results), and analyzing those results vis-a-vis Evaluation is viewed as complementary to — and the targets. While performance measurement is in some respects superior to — routine performance concerned more narrowly with the production or measurement techniques. For example, evaluation supply of performance information, performance allows for more in-depth study of performance, management is broader. It is equally concerned can cover a longer time period, may incorporate with generating management demand for performance factors such as outcome or impact too difficult or information — that is, with its uses in policy, expensive to assess through on-going monitoring, program, and budget decision-making processes and can analyze causes and effects in detail. and with establishing organizational procedures, Nevertheless, some concerns over the future of mechanisms and incentives that actively encourage the evaluation function remains. Establishing its use. In an effective performance management performance measurement systems are taking system,achieving results and continuous improvement considerable effort and resources, often without based on performance information is central to the additional funds being allocated for this purpose. management process. In situations of diminishing overall budgets, this may well lead to a competition for resources between performance measurement and evaluation activities within OECD government agencies. THE ROLE OF EVALUATION IN PERFORMANCE MANAGEMENT The role of evaluation vis-a-vis performance USES OF PERFORMANCE INFORMATION management systems in many cases was not initially clarified, and still remains somewhat Performance should not be measured for measure- vague. In part, this is because evaluation was well ment’s sake. Performance information, both from established in many OECD governments before performance monitoring and evaluation sources, the introduction of performance management and should be used. One key use is for transparent the new approaches did not necessarily incorporate reporting on performance and results achieved to evaluation. New performance management techniques external stakeholder audiences. In many cases, were developed partly in response to perceived government-wide legislation or executive orders failures of evaluation; for example, that the uses of have recently mandated such reporting. Moreover, evaluation findings were limited relative to their such reporting can be useful in competition for costs. Moreover, evaluation was often viewed as a funds by convincing a skeptical public or legislature specialized function carried out by external experts that the agency programs produce significant or independent units, whereas performance results and provide “value for money”. Annual management, which involves reforming core performance reports are often directed to ministers, management processes,was essentially the respon- parliament, stakeholders, customers, and the sibility of managers within the organization. general public. 24 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing Performance information should also be used for 1. Identifying in clear, measurable terms the internal purposes,such as for management decision- objectives or results being sought and developing making and identifying areas for improvement. a conceptual framework for how the results This requires that performance information be will be achieved. integrated into key management systems and processes of the organization; such as in policy 2. Selecting indicators that will be used to measure formulation, in project/program planning and progress towards each objective. management, and in budget allocation processes. 3. Setting explicit targets (i.e., planned results to be Performance budgeting is the term generally used achieved by specific dates) for each indicator to refer to the infusion of performance information that will be used to judge performance. into resource allocation processes. The concept of performance budgeting is essentially the process of 4. Developing performance monitoring systems to linking budget levels to expected (or actual) regularly collect data on actual results achieved. results, rather than to inputs and activities. Many OECD countries are experimenting with a variety 5. Analyzing and reporting actual results vis-a- of approaches to more closely associate expected vis the targets (or other criteria for making performance with requested funding levels, as part judgements about performance). of their broader reforms efforts to become more results-oriented.(See OECD, Budgeting for Results: 6. Integrating evaluations to provide complemen- Perspectives on Public Expenditure Management, 1995). tary information on performance not readily A recent report by the U.S. General Accounting available from performance monitoring systems. Office reviews some of the many challenges facing U.S. government agencies attempting to implement 7. Using performance information (from both performance budgeting, including a variety of performance monitoring and evaluation performance measurement issues, cost accounting sources) for internal management learning and shortcomings, and the essentially political nature of decision-making and for external reporting to budget allocation processes.(See GAO, Performance stakeholders on results achieved. Budgeting: Initial Agency Experiences Provide a Foundation to Assess Future Directions, July 1999) The first three phases or steps general ly relate to a results-oriented planning approach, sometimes When performance information is used in internal referred to as strategic planning. The first five steps, management processes with the aim of improving together, are usually included in the concept of performance and achieving better results, this is performance measurement.All seven phases combined often referred to as managing-for-results. Such are essential to an effective results based management actual use of performance information has often system. That is, integrating complementary infor- been a weakness of performance management in mation from both evaluation and performance the OECD countries.Too often, government agencies monitoring systems and ensuring management’s have emphasized performance measurement for use of this information are viewed in this paper as external reporting only, with little attention given critical aspects of results based management. to putting the performance information to use in internal management decision-making processes. These two major uses of performance informatio n may not be completely compatible with one OTHER COMPONENTS OF another, and may require different types of results RESULTS BASED MANAGEMENT data and collection methods. Satisfying both needs or uses without over-burdening the performance Other reforms are often associated with results management system remains a challenge. based management systems. Often, these other components act to stimulate or facilitate the use of performance information. Some of these organizational changes include: KEY PHASES OF s Accountability — instituting new mechanisms RESULTS BASED MANAGEMENT for holding agency managers and staff account- able for achieving results within their sphere Some key elements or phases of results based of control or influence (e.g., results-oriented management include: management contracts and personnel appraisals). RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Rev i ew of Experience 25 s Decentralization — delegation of authority effective and results-oriented. “Aid fatigue”, the outs to the field and down to the management public’s perception that aid programs are failing to level that’s being held accountable for results. produce significant development results, declining In other words, empowering managers at aid budgets, and the government-wide reforms have appropriate levels with flexibility to shift resources all contributed to the donor agencies’ recent efforts from poorer to better performing activities. to establish results-oriented management systems. s Client focus — consulting with beneficiary Thus far, the donor agencies have gained most groups concerning their preferences and satis- experience with establishing performance meas- faction with goods and services provided, and urement systems — that is, with the provision of being responsive to their needs and desires. performance information — and some experience with external reporting on results. There is less s Participation — including partners and documented experience with the actual use of stakeholders in all aspects of performance performance information for internal management measurement and management processes (e.g., decision-making. jointly setting objectives, defining indicators, collecting, analyzing and reviewing data, Results based management and measurement conducting evaluations, and using the infor- processes may take place at various organizational mation for learning and decision-making). levels within the donor agencies. The first level, which has been established the longest and for s Reformed policies and procedures — new which there is most experience, is at the project policy and procedure directives for changing level. However, some agencies such as USAID the way the agency conducts its business (e.g., have also recently established more strategic country new requirements, roles and responsibilities level systems within their country operating units. for strategic planning, for performance Moreover, establishing performance measurement measurement and evaluation, and for use of and management systems at the third level — the performance information in external reporting corporate or agency-wide level — is now taking on and internal decision-making processes). urgent importance due to increasing public pressures and government-wide mandates requiring annual s Supportive mechanisms — various ways of reporting on agency-wide performance and results. assisting managers to effectively implement Effectively linking and aggregating performance performance management, such as providing measures across these various levels remains an issue. reengineering training, technical assistance services, new performance information data- The donor agencies face special challenges and issues bases, guidebooks, tips and best practices in establishing their performance management series, and other management tools. and measurement systems, that are either unique or more pronounced than those confronting the s Cultural change — equally important for domestic government agencies. This can make successful implementation of results based establishing performance measurement systems management is transforming the organizational more complex and costly than normal. For example, culture and attitudes (e.g., reorientation these agencies: towards achieving results rather than imple- s Work in many different countries and contexts. menting inputs and processes, an openness to s Have a wide diversity of projects in multiple learning from failures as well as successes, and sectors. a commitment to objective and transparent s Often focus on capacity building activities, performance reporting). which are harder to measure than direct service delivery activities. s Are moving into new areas such as the envi- ronment, where there’s little performance measurement experience. SPECIAL CHALLENGES s Often lack standard indicators on results that FACING THE DONOR A G E N C I E S can be easily aggregated across projects. s Are usually only minor actors affecting As has been the case more broadly for the public impacts, with consequent problems in sector of the OECD countries, the donor agencies attributing them to their agency’s activities. have faced considerable external pressures to s Typically rely on outcome and impact data reform their management systems to become more collected by partner countries, who have limited 26 E V A L UATION C A PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing technical capacity and resources, with conse- used to attain them. (See quent quality, coverage and timeliness problems. Figure 1). The logframe FIGURE 1: is based on a five-level PROJECT LOGFRAME In particular, these factors can complicate donor hierarchy model with HIERARCHY LEVELS A N D agencies’efforts to aggregate results across projects assumed cause-effect rela- TYPES OF INDICA T O R S and programs to higher organizational and agency- tionships among them, wide levels. with those at the lower Coal level of the hierarchy contributing to the attain- Impact Indicators ment of those above. Thus, inputs are used to PERFORMANCE MEASUREMENT undertake project activi- AT THE PROJECT LEVEL ties that lead to the Purpose delivery of outputs (goods/ Outcome Indicators Performance measurement at the project level is services), that lead to the concerned with measuring both a project’s imple- attainment of the project mentation progress and results achieved. Two purpose that contributes broad t ypes of project performance measurement to project goal. Outputs might be distinguished. (1) Implementation measurement Output Indicators is concerned with whether or not project inputs and activities are in compliance with design budgets, 2 . SELECTING INDICA T O R S workplans, and schedules, and (2) results measurement is concerned with whether or not actual results are Next, performance indi- Activities achieved as planned. Results are usually measured cators are selected for at three levels — immediate outputs, intermediate measuring progress in Process Indicators outcomes and long-term impacts. implementing activities and in achieving results. Whereas traditionally the emphasis has been The logframe provides Inputs mostly on implementation concerns, with the rise a hierarchical structure of results based management the focus is increasingly around which the indi- Input Indicators on measurement of results. Moreover, emphasis is cators are typically con- shifting from more immediate outputs to longer- structed.Indicators specify term outcomes and impacts. what to measure along a scale or dimension in order to gauge progress (e.g.,number of workshops held, Measuring performance at the project level can be percentage of farmers attending demonstration sessions, changes in crop yields, etc.). The relative divided into five elements or phases, as briefly importance of indicator types is likely to change outlined below. The importance of participatory or over the project’s life cycle, with more emphasis on collaborative approaches in all phases of performance input and process indicators at first, then shifting measurement is stressed — that is, involving not to output, outcome (purpose-level), and impact only donor agency project managers but also (goal-level) indicators later on as the project representatives from the implementing agency, the matures.Also, different management levels tend to partner government, the intended beneficiary place emphasis on different indicator types. For groups, and other stakeholders. This helps build example, project field staff will find input and agreement around the project’s objectives and com- process indicators of most use, whereas project mitment to the performance measurement process. managers will be more interested in achievement of project outputs and purpose /outcomes. Senior agency officials will be interested in the longer- 1 . FORMULATING OBJECTIVES term and broader social and economic impacts of the project, which may not be evident until after As part of project planning, the project’s objectives the project is completed.These different intended should be clarified by defining precise and measurable uses and users need to be kept in mind when statements concerning the results to be achieved selecting indicators. (outputs. purpose, and goal) and then identifying the strategies or means (inputs and activities) for Often checklists of criteria are devised, against meeting those objectives.The project logframe is a which proposed indicators can be judged and favorite tool used for conceptualizing or modeling selected. For example, some commonly used a project’s objectives and the strategies that will be indicator selection criteria include: RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Rev i ew of Experience 27 FIGURE 2: PROJECT PERFORMANCE MONITORING PLAN (Format for Recording Key Aspects of Data Collection) Data Frequency Responsibility Type of Indicators and Data Sources Collection and Schedule for Data Indicator Definitions Methods of Collection Acquisition Impact Indicators Outcome Indicators Output Indicators Process Indicators Input Indicators Risk Indicators s Valid — Does the indicator directly represent 3 . SETTING T A R G E T S the result it is intended to measure? Once indicators have been identified, actual baseline s Objective — Is the definition precise and values should be collected for each, ideally just unambiguous about what is to be measured? before the project gets underway. This will be important for gauging whether progress is being s Reliable — Is the data consistent or compara- made later. Often agencies also set explicit targets ble over time? for their indicators. A target specifies a particular s Practical — Can data be collected easily, on a value for an indicator to be accomplished within a timely basis and at reasonable cost? given timeframe. (For example, 200 workshops to be held before September 2001, contraceptive s Useful — Will the data have utility for deci- prevalence rate increased to 65% by 2003.) Targets sion-making and learning? help clarify exactly what needs to be accomplished by when.It represents a commitment and can help s Owned — Do stakeholders agree that this orient and motivate project staff and mangers to indicator makes sense to use? the tasks at hand. Tradeoffs among these indicator selection criteria A natural tension exists between setting targets may exist. Probably the most important, overarching that are high enough to make project managers consideration is that the indicators provide managers and staff stretch to achieve them, and yet low with the information they need to do their job. enough to be realistic and achievable. While on the one hand, indicator data should be of sufficient quality to be credible and ensure the If they are set unrealistically high and unattainable, right decisions are made, on the other hand it confidence and credibility will suffer and may even should be practical — timely and affordable. Care set in motion perverse incentives to hide or distort should be taken to limit the number of indicators the figures. Any information that helps to ground selected to the minimum needed to adequately a target setting exercise and ensure its realism is capture the key dimensions of a result. Keep the useful. For example, it is useful to establish a baseline, performance measurement system simple to avoid identify historical trends, seek implementing overburdening managers and staff with unnecessary agency views, survey expert opinion about what is data collection responsibilities. possible, review research findings, or identify 28 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing benchmarks (i.e., compare what results have been patterns (e.g., typical variations in data collection achieved by similar projects with a reputation for sources/methods, frequency of collection, and high performance). assignment of responsibility) are summarized in Figure 3. As one moves to higher and higher levels of the logframe hierarchy, there is the 4 . MONITORING PERFORMANCE (COLLECTING DATA) tendency for data collection efforts to become more expensiv e, time-consuming, and technically Once indicators are selected and targets are set, complex.Also, there is a tendency for data collection actual data for each indicator is collected at regular efforts to be conducted less frequently. The placement intervals. Implementation monitoring involves the of responsibility for data collection also tends to frequent, on-going recording of data on project shift from the implementing agency at the lower operations — e.g.,tracking funds and other inputs, levels to the donor agency and/or to the partner and processes. It involves keeping good financial government at the higher levels. accounts and field activity records, and frequent checks to assess compliance with workplans and Data on project inputs, processes, and outputs are budget. Results monitoring involves the periodic generated mostly by project staff and are based on collection of data on the project’s actual achievement simple reporting systems updated frequently. Data of results — e.g., its short-term outputs, medium- on outcomes are generally collected periodically term outcomes, and long-term impacts.This type (e.g.,annually) from low-cost rapid appraisal methods, of monitoring demonstrates whether a project is mini-surveys or consultations with project clients. moving towards its objectives. Measuring impacts usually require conducting expensive sample surveys or relying on already Project managers have found it useful to prepare existing data sources such as national surveys, performance monitoring plans to record key censuses, registration systems, etc. Impact data are aspects of data collection,such as providing defini- usually only collected every few years or at the tions for each indicator, source and methods of projects beginning and end (or ex post). data collection, frequency/schedule for collection, and assignment of responsibility for collection. Data collection at the higher levels — especially Figure 2 illustrates a matrix format that USAID has at the impact level — is often considered beyond found useful for recording summary information the scope of the implementing agency’s normal about data collection plans. responsibility. Donor agencies will need to make special arrangements with partner country organi- Data collection approaches vary according to levels zations with statistical/data collection expertise for of the project logframe hierarchy. These common conducting or adding-on to planned surveys. Since FIGURE 3: CHARACTERISTICS OF PROJECT DATA COLLECTION EFFORTS (By Logframe Hierarchy Levels) Type of Data Collection Frequency of Organizational Indicator Method Data Collection Responsibility Impact Indicators censuses and surveys, multi-year partner governments, national statistics donor agencies Outcome Indicators customer surveys, annually donor agency and rapid appraisals, implementing agency consultations Output Indicators project records quarterly, biannually implementing agency Process Indicators project records weekly, monthly implementing agency Input Indicators project records, weekly, monthly implementing agency financial accounts RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Re v i ew of Experience 29 several donor agencies working in the same sector results (example: percentage of students may share needs for similar impact-level data, it attending project schools who are female). would be useful to consider coordinating or jointly supporting these data collection efforts, to avoid s Customer satisfaction — how well project duplication of effort and to share costs. Moreover, outputs correspond to client preferences to ensure valid and reliable data, supporting capacity- (example: percent of clients satisfied with building efforts may be called for as well. health services delivered). At what level should the focus of performance s Effectiveness — the extent to which results — monitoring be placed? Concentrating on just one outputs, outcomes, or impacts — are being level of the logframe hierarchy may have unintended, achieved as planned (targeted). even dysfunctional, consequences. For example, concentrating only on the output level may result s Attribution — the extent to which outcomes in “doing the wrong things well”. Concentrating and impacts can be attributed to outputs from only on higher outcome and impact levels may a particular project. lead to lack of basic monitoring information about project activities and services, and result in poor s Cost-effectiveness — the relationship between implementation. The answer appears to lie in taking as comprehensive and balanced an approach as is project costs and results attributable to the project. possible, within reason/practicality. s Sustainability — the capacity for results to Developing a more comprehensive performance extend beyond the formal life of the project. monitoring system that recognizes the need for performance information at various levels is least s Relevance — the continued appropriateness likely to lead to distortions. Moreover, as already of a project ’s results to the needs of the target discussed, different stakeholder groups and man- population, the partner country’s national agement levels will have varying interests in these development priorities and to the development levels of results, so satisfying everyone means having agency’s corporate-level goals. a comprehensive system. Periodic reviews of performance data by project management will help alert them to problems or 5 .A N A LYZING AND REPORTING PERFORMANCE DATA shortcomings vis-a-vis plans and targets, which may lead directly to taking actions or signal Periodic management reviews,analysis and reporting the need for more in-depth studies focused on of project performance monitoring data most typically specific performance issues. Routine performance emphasizes effectiveness in achieving targets, by monitoring alone may not be adequate for comparing actual results with planned results. addressing some of the performance issues listed However, analysis of performance monitoring data above(e.g., cost-effectiveness, attribution), which may address a broad variety of issues. For example: because of their long-term nature and/or complexity may require special in-depth assessments or s Economy — the relationship between costs evaluation studies. and physical inputs (i.e., an organization is economical if it is purchasing inputs as cheaply A number of donor agencies have established as possible). performance rating systems whereby managers, s Efficiency — the relationship between costs drawing on data from performance monitoring and outputs (example: cost per kilometer of systems, judge their project’s performance by road built). assigning a rating along a scale (e.g., highly satisfactory, satisfactory, unsatisfactory, or highly s Productivity — relationships between inputs unsatisfactory), against a number of criteria and outputs (example: number of demonstrations (e.g., effectiveness, efficiency, relevance, sustain- handled per extension worker). ability, etc.). These performance ratings or self- assessments are typically reported to agency s Excellence/quality — producing high quality headquarters in standard reporting formats at outputs (example: percent of units produced specific times, such as at project completion or in that meet technical standards). annual reports. A particular ly useful characteristic of project performance rating systems is that they s Equity — the extent to which needy/disadvan- enable consistent comparisons and aggregation taged sub-populations have equitable access to across the project portfolio. 30 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from an International Contference in Beijing PERFORMANCE MEASUREMENT A conceptual tool that is being used by USAID for strategic planning and performance measurement AT THE COUNTRY LEVEL at the country program level is the results framework. A results framework is a graphic display of the A few donor agencies have developed performance strategies necessary and sufficient for achieving a measurement systems for broader country pro- significant or strategic development objective in a grams — defined as sets of related projects or developing country. The results framework relies activities sharing the same development objective on objective tree concepts, and diagrams the logical within a partner country, usually at a national sector cause-effect relationships between activity outputs level. USAID pioneered this approach during the at the bottom, intermediate results or outcomes in mid-1990s,abandoning its previous focus on projects the middle, and the strategic development objective and moving towards more strategic and results- at the top. Thus, it embodies the development oriented country programming approaches as part hypotheses underlying multiple partners harmonized of its broader reengineering reforms. strategies for achieving a shared development objective. The country program approach is a much more See Figure 4 for a hypothetical example of a results comprehensive and strategic approach to performance framework. Results frameworks should be developed management and measurement than the project via collaborative processes involving all donor agencies approach. It focuses on a significant development and other development partners working towards objective within a country, usually a sector, sub- a shared development objective, ideally under the sector, or a crosscutting objective.Thus, the unit of leadership of the partner country government. analysis is not a single project but a country program that typically includes many projects or activities Results frameworks are useful as strategic planning implemented by different donor agencies and partner and management tools as well as structures for organizations over a relatively long time period. performance measurement. They help identify what program strategies are necessary and sufficient Performance measurement frameworks and systems to achieve a significant development objective, and developed at the country program level are thus then enable collaborating partners,working in harmony, comprehensive, long-term, multi-activity and multi- to sort out their individual responsibilities or site endeavors that usually include many projects contributions to the overall strategy. This can help and different organizational actors/contributors donor agency operating units to better align (focus within a given country sector or sub-sector. and concentrate) their assistance activities into FIGURE 4: HYPOTHETICAL ILLUSTRATION OF A COUNTRY PROGRAM RESULTS FRAMEWORK (Development Objective and IntermediateOutcomes) Increased availability of food in domestic market Markets constructed More efficient transportation Increased food production (Ministry of Agriculture) of goods to market Improved Better More credit Improved Upgraded production tech- Irrigation available to farm to transport nologies available (Ministry of farmers market roads vehicles to farmers Agriculture) (DANIDA) (World Bank) (USAID) (CIDA) RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Rev i ew of Experience 31 those program strategies for which they have taken many development partner actors — e.g., various responsibility, rather than just have a diverse portfolio donor agencies, the NGO community, and of of seemingly unrelated projects. The country course the partner country government. Some of development objectives and intervention strategies the tools developed for country program level selected by a unit usually have to be in line with strategic planning and performance measurement the donor agency’s overall corporate goals and should be particularly well suited to new modes of areas of comparative advantage. development assistance based on joint multi-donor/ partner sector programs in which investments and The framework is also a performance measurement activities are harmonized to achieve shared country tool — providing a structure for measuring and development objectives. monitoring progress towards the achievement of those results for which the unit is responsible. While this approach holds considerable promise, Performance data from the monitoring system is its actual use thus far may be falling short of its used to alert managers when actual results are not potential. For example in USAID, country operating meeting targets as planned,indicating the need for units have too often tended to develop results adjustments to be made in relevant projects and frameworks in relative isolation and from their activities. It may be useful to occasionally supplement own agency perspectives. more routine reviews of the performance monitoring data with complementary strategic, program-wide While there is typically some limited participation evaluations that assess the relative effectiveness by their implementing agency partners and immediate and cost-effectiveness of alternative strategies and stakeholders,the focus is usually on USAID’s own activities for achieving the development objective. programs and strategies, rather than placing equal focus on all relevant partners’ programs. Whereas the project approach puts equal weight on monitoring all elements of the logframe hierarchy, and may even traditionally have favored implemen- tation monitoring, the country program results framework puts the higher-order development PERFORMANCE MEASUREMENT objective and intermediate outcomes at center-stage. AT THE AGENCY LEVEL It is less concerned with defining the individual project means (inputs/processes) and outputs, and Largely driven by domestic public pressures and much more concerned with measuring and achieving government-wide legislation for annual reporting the higher-level results. The shift from individual on agency performance, the donor agencies — like projects to programs also implies a different time- government agencies more generally — are clarifying frame dimension ,f reed from the confines of a single their overall goals and seeking ways to summarize project’s life cycle. By focusing on country level their achievements vis-a-vis those goals.Measuring development objectives and intermediate outcomes, and reporting on results at the agency-wide level the timeframe now becomes longer-term,outliving poses a significant challenge for the development the comings and goings of individual project activities. agencies.They face a number of obstacles in attempt- ing to aggregate results, some of which are either Individual project activities tend to be less well defined unique to or complicated by the nature of development in this approach,allowing for more flexible designs cooperation work (See previous discussion). and implementation, rather than rigid “blueprint” approaches. Moreover, in USAID, headquarters Agencies such as USAID have recently developed no longer approves projects. Instead, authority is and issued policy papers or strategic plans that delegated to operating units in the field so they can clearly articulate the agency’s overall mission and shift course mid-stream if results monitoring the key development goals or priority areas on information indicates certain activities are not which they will concentrate. Usually the agency working well. Nevertheless, it is important to be goals are sector-oriented (e.g., better education, able to link individual project activities, outputs improved health, etc.) although some may be and their associated costs within the broader crosscutting special concerns (e.g.,gender equality, results frameworks. partnerships) or internal management efficiency goals. These statements about agency goals serve This country program level approach puts a premium to articulate to external audiences what the overall on partnerships and more collaborative approaches, aims of the development assistance program are, and since achieving a strategic,long-term development provide a framework or structure for gathering and objective is clearly dependent on the activities of reporting data on overall agency results achieved. 32 E V A L UATION C A PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing This is viewed as important in an era of declining contribute towards achieving and how it intends aid budgets, increasing competition for funding, to contribute. The hierarchies serve as detailed and growing public skepticism about the effective- structures for reporting on agency results at several ness of development aid. Clarifying agency-level levels. For example, a typical three-level hierarchy goals has also been useful as an internal management structure might include agency goals, sub-goals, tool for strategic planning — that is, for focusing and supporting program approaches. USAID has and concentrating the agency’s assistance portfolio also found it useful to present their strategic and resources within priority goal areas.In the case framework as graphic diagrams or visual displays, of USAID, country operating units have been using objective tree concepts. Figure 5 provides an asked to align their country strategic objectives example for the environmental goal from USAID’s and program activities within the new USAID strategic framework. goal structure. Collecting data for agency-wide performance USAID has further elaborated its agency level assessments and reporting may take place from goals into several sub-categories, forming a several basic sources;(1) from existing internation- multi-level framework or hierarchy of objectives. al sources/databases that maintain country level Such a multi-level strategic framework can serve statistics on sector development trends, (2) from to clarify even further what the agency aims to the project/program performance measurement systems FIGURE 6: HYPOTHETICAL ILLUSTRATION OF A COUNTRY PROGRAM RESULTS FRAMEWORK (Development Objective and Intermediate Outcomes) Agency Goal 4 Environment managed for long –term sustainability Agency Agency Agency Agency Agency Objective 4.1 Objective 4.2 Objective 4.3 Objective 4.4 Objective 4.5 Biological Global climate Sustainable Increased Sustainable diversity change threat urbanization provision of natural resource conserved reduced promoted environmentally management and pollution sound energy reduced services Agency Program Agency Program Agency Program Agency Program Agency Program Approaches Approaches Approaches Approaches Approaches 1) Improve 1) Reducing 1) Increasing 1) Increasing 1) Managing management greenhouse access to energy forests of protested gas emissions water and efficiency sustainability areas from energy sanitation 2) Increasing 2) Managing 2) Promoting use services use of water sustainable 2) Reducing 2) Promoting renewable resources use of net green- improved energy 3) Practicing biological house gas urban 3) Introducing agricultural resources emissions management innovative sustainability 3) Supporting from land use 3) Promoting clean 4) Managing ex-situ 3) Assisting pollution technologies coastal conservation adaptation prevention zones of genetic to climate and control sustainability diversity change RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Rev i ew of Experience 33 FIGURE 5: USAID’S STRATEGIC FRAMEWORK FOR THE ENVIRONMENTAL GOA L Core Set of Indicators Core Set of Indicators Economic well-being Reducing extreme poverty 1. Incidence of extreme poverty The proportion of people living in extreme poverty in 2. Poverty gap ratio developing countries should be reduced by at least 3. Poorest fifth’s share of national consumption one-half by 2015 4. Child malnutrition Social development Universal primary education 5. Enrolment in primary education There should be universal primary education in all 6. Completion of primary education countries by 2015 7. Adult literacy rate Gender equality Progress towards gender equality and the empowerment of women should be demonstrated by eliminating gender disparity in primary and 8. Gender equality in education secondary education by 2005 9. Gender equality in adult literacy Infant and child mortality The death rates for infants and children under the age of five years should be reduced in each developing 10. Infant mortality rate country by two-thirds the 1990 level by 2015 11. Child mortality rate Maternal mortality The rate of maternal mortality should be reduced 12.Maternal mortality ratio by three-fourths between 1990 and 2015 13.Births attended by skilled health personnel Reproductive health and population Access should be available through the primary health care system to reproductive health services for all individuals of appropriate ages, no later 14.Contraceptive prevalence rate than the year 2015 15.HIV prevalence rate Environmental sustainability and regeneration Environment There should be a current national strategy for 16.Countries with national environmental plans sustainable development,in the process of 17.Access to safe water implementation, in every country by 2005, so 18.Intensity of fresh water use as to ensure that current trends in the loss of 19.Biodiversity: Land area protected environmental resources effectively reversed at 20.Energy efficiency both global and national levels by 2015 21.Carbon dioxide emissions maintained by the agency’s country operating units, results data across project/program portfolios. or (3) from the agency’s evaluation reports. In Results of these agency-wide analyses of aggregate most cases, these data are entered and stored in project/program performance and results are usually automated, central agency databases to facilitate reported in annual performance reports. agency-wide analysis and reporting. Donor agencies have a number of basic options to Computer databases and software programs facilitate consider for aggregating or summing up performance data sorting, aggregation, statistical analysis and and results achieved at the agency-wide or corporate graphic presentation of results.They can greatly aid level. At the two extremes — project outputs and the work of analyzing large amounts of performance/ country-level sector statistics — aggregation of 34 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing indicator data may be relatively easy. But in the agencies. On the other hand, there are some serious case of outputs, the question “so what?” may be issues with using this approach, especially in the raised. With country level statistics, it is rarely context of reporting on agency performance. possible to link changes credibly to a single Attempting to link and attribute these country- agency’s interventions, especially on a year-to-year level and global-level socio-economic improvements basis. In the middle are project outcomes, which to the activities of a single donor agency is a wide should be both significant yet have clearer linkages stretch of the imagination that many will question. to agency activities than national statistical trends. Donor agencies using this approach would be The problem here is that often there is great diversity advised to adopt the goals and indicators from among in projects’ objectives and in their performance those agreed to by the international community (as measures, so aggregating across standard indicators articulated in the DAC report, Shaping the 21st is often not possible. Some agencies have overcome Century: The Contribution of Development Co-oper- this by developing rating systems that score a project’s ation, 1996). Moreover, their reports might clarify success in meeting its objectives and then summing that these results are the consequence of many across projects the numbers and percentages that partners’ contributions, and cannot be attributed were successful or unsuccessful in achieving outcomes. to individual agencies. (See Figure 6). Another complication is that agency performance reporting These three basic options for aggregating results is usually required annually, whereas data on coun- are discussed below. try development trends is often only available at intervals of several years apart. Moreover, even if it was available annually, the long-term nature of 1 . SELECTING THE PROJECT OUTPUT LEVEL development improvements at this level means FOR AGENCY-WIDE REPORTING ON RESULTS year-to-year changes may not be significant. Outputs of projects (such as number of units of goods and services delivered or numbers of 3 . SELECTING THE PROJECT/PROGRAM OUTCOME LEVEL beneficiaries/clients reached) are relatively easily FOR AGENCY-WIDE REPORTING OF RESULTS. summed up across similar types of projects,assuming they are comparable. For development agencies In between project outputs and macro-statistics, with fairly centralized structures and a standard set there’s the level of intermediate outcomes. of project approaches with comparable outputs (goods/services) across country settings, this A major advantage is that performance monitoring approach may be feasible. However, reporting at systems at the project or program level are generally the output level will only be valuable to the extent already established and thus data on project/ that the intended external audiences/stakeholders program outcome achievement should be available. will be impressed with this level of results. If the A key problem with aggregating project outcomes response is “Is that all we’re achieving?”, summing for agency-wide reporting is the typically great and reporting on outputs may be counterproductive diversity of outcomes and their indicators,especially in terms of defending the aid program before in decentralized agencies such as USAID. Without parliament or the taxpaying public. standard indicators of project/program outcomes, direct aggregation is not possible. Agencies may deal with this problem in different ways. For example, 2 . SELECTING LONG-TERM SECTOR DEVELOPMENT T R E N D S by developing standard outcome indicators for FOR AGENCY-WIDE REPORTING OF RESULTS common “program approaches” (groupings of like projects). However, this approach is only advisable Another option for reporting on results achieved for more centralized agencies with fairly structured at the corporate level is to report on long-term program approaches. Another approach for getting social and economic improvements at the country around this incomparability problem is to devise sector and global levels using international statistical rating systems that score a project’s success in datasets,which have some measure of comparability meeting its objectives. The agency can then aggregate across countries. Advantages of this approach across projects within an objective or program area include its appeal in terms of reporting on significant with statements like “85% of projects or programs impacts that matter to stakeholders, (e.g., alleviate aimed at improving child survival have successfully poverty, reduce infant mortality, achieve universal met or exceeded their outcome targets”. Issues primary education) and the ready availability of with this approach may include the extent to international indicator datasets covering many which standard criteria for making judgements (not all) of the sector concerns of the development about scores are applied across projects, and the RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Re v i ew of Experience 35 reliability of “self-assessment” ratings especially when managers’ project/program performance self- managers may fear the consequences of poor scores. ratings, etc.) Typically performance data are presented in standard, comparable formats None of these options for aggregating results to that can be easily entered into databases and the agency-wide le vel appears to be ideal, and the analyzed across the portfolio. They are meant donor agencies face considerable challenges in to provide consistent types of information their current efforts to summarize and report covering a broad range of performance issues performance at the corporate level. Given this and results, but without great depth of analysis. state-of-the-art, agencies may benefit from mixing Evaluations, on the other hand, usually are or balancing the various approaches available. less standardized and follow individual scopes How to best co-ordinate, synthesize and integrate of work.Moreover they tend to focus on fewer findings from evaluation reports into annual performance issues but analyze them in agency performance reports is another issue greater depth. needing attention. s Performance monitoring reports focus mostly on whether or not results were achieved as planned, whereas evaluations can better explain why and how they were achieved or THE ROLE OF EVALUATION VIS-A-VIS not. In other words, evaluations seek to analyze PERFORMANCE MEASUREMENT and understand the projects or program’s context and factors influencing performance, Performance measurement and evaluation are generally both internal (within managers’ control) and viewed as two distinct but complementary sources external (beyond managers’ control). of performance information, both of which are necessary for effective results based management. s Routine performance monitoring can ser ve as an early warning system to alert managers when In USAID, performance monitoring is defined as there are performance shortfalls. However, a process of collecting and analyzing data to measure they do not assess the causes of the shortfalls the performance of a program, process or activity nor make recommendations for appropriate against expected results (targets), whereas evaluation management actions, as do evaluations. is defined as a relatively structured analytical effort undertaken selectively to answer specific management s Because of timing as well as the need to use questions regarding programs or activities. Some more rigorous methods and in-depth analysis, further distinctions often made by agencies follow: some performance issues, such as long-term impact, attribution, cost-effectiveness, and s Performance monitoring reports are self- sustainability, can probably be better addressed assessments by project or program managers, by evaluation than by routine performance whereas evaluations are typically conducted by monitoring reports. larger evaluations teams, often comprised of external evaluators that can provide an inde- Thus, evaluations and performance measurement/ pendent judgement about project/program monitoring can be viewed as distinct but comple- performance. However, trends towards more mentary functions. Both are management tools. Both participatory forms of evaluation in some are important sources of performance information agencies may make this less of a distinction. that together can contribute to management learning and decision-making processes and to external s Performance monitoring reports are typically performance reporting requirements. mandatory for larger projects or programs and thus provide a reasonably complete coverage However, there is some concern that performance of the overall portfolio, whereas evaluations measurement and reporting tasks, often required are often conducted on a much more selective by government-wide law or executive orders, may (i.e., occasional, optional) basis for projects/ be “crowding out” evaluations. That is, they may be programs of particular interest or concern. competing for the same, increasingly scare staff and other resources. For example, a survey of evaluation s Performance reports involve relatively straight- offices within the U.S. federal government agencies forward presentations of performance data found this to be a concern (See U.S. General (e.g.,actual results achieved vis-a-vis expected Accounting Office, Program Evaluation: Agencies results, actual expenditures data vis-a-vis budgets, Challenged by New Demand for Information on 36 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing Program Results, April 1998). Typically, agencies A special type of decision-making that performance have not been given additional funding to establish information is increasingly being called upon to their performance management and measurement influence is resource allocations. In USAID, for systems, while overall agency budgets have often example, country operating units must submit been on the decline. annual reports (called Results Reports and Resource Requests) that ties their request for funds to the These overall trends and concerns are evident in results they expect or plan to achieve with those USAID. For example, since the mid-1990s when resources. The reports also contain self-assessments reengineering reforms mandated the establishment by operating units of their actual program performance of performance management and measurement (extent to which results targets were achieved) over systems, the size of USAID’s central evaluation office the previous year. Their performance ratings are staff and resources has declined rapidly. Moreover, then ranked across countries and programs, and the number of evaluations conducted by USAID’s this information further influences the budget country operating units also declined sharply, from allocation process across countries and programs. 489 reports in FY1994 to 70 reports in FY1998. On the margin, better performing programs tend to get more funds, and poorer performing programs get less. However, performance remains a compar- atively minor factor in these allocation decisions, KEY USES OF PERFORMANCE INFORMATION outweighed by predominant foreign policy criteria and constrained by Congressional earmarks. In results based management systems,performance information (drawn from both performance meas- urement and evaluation sources) serve two primary EXTERNAL REPORTING (ACCOUNTABILITY-FOR-RESULTS) uses or purposes. One use is as an internal man- agement tool for making program improvements; The second key use of performance information is the second is for external accountability reporting. to report agency performance to various stakeholder audiences. Donor agencies, like other domestic government agencies, are accountable for achieving INTERNAL MANAGEMENT IMPROVEMENT and reporting results to the taxpaying public and (MANAGING-FOR-RESULTS) their elected representatives, and to designated central oversight agencies. Often there are now This first intended use of performance information government-wide legal requirements for reporting is for continuous feedback to managers about the results, at certain times and in specific formats, results they are achieving, so they can then use the which are being audited by oversight agencies. information to improve their performance even Moreover, overall agency accountability for results more. Sometimes discussions of this internal is increasingly being devolved and translated into management use are further sub-divided into two accountability at lower organizational levels (e.g., related aspects or processes — promoting learning operating units, teams, or even individual managers). and facilitating decision-making. USAID for example is experimenting with manage- ment contracts and personnel appraisal systems (a) Promote Learning. Performance information that specify what results are to be achieved, when, promotes continuous management learning and by whom. In addition to being responsible to about what results are being achie ved by their domestic stakeholders, the donor agencies are also projects/programs and why — i.e., what factors accountable to their partner country governments are influencing good or poor performance. and ultimately to their intended beneficiary groups. Improved knowledge is a prerequisite for better decisions. POTENTIAL CONFLICTS AMONG USES (b) Facilitate Decision-making. Management’s learning in turn facilitates their making appropriate As experience with using performance information decisions.Continuous performance information grows, the potential for conflict between its two about progress towards results and about the key intended uses is emerging. Managing-for- factors influencing performance will facilitate results implies a shift in focus in from inputs and good decision-making and timely action. Lessons processes to outputs, and from outputs to even from experience can help agency managers higher outcomes and impacts. Not only is it to continually improve their development important to know what results are being achieved assistance projects and programs. at these different levels,but also to understand the RESULTS BASED MANAGEMENT IN THE DEVELOPMENT CO-OPERATION AGENCIES: A Re v i ew of Experience 37 cause-effect linkages between them, e.g., why an science research. Moreover, it would favor a self- activity is successful or not, which approaches assessment approach to ensure management’s work better, and under what conditions or contexts. ownership and a first-hand learning experience, Emphasis on accountability-for-results, however, and also would encourage more participatory may ironically shift focus back down to outputs, methods and stakeholder involvement. In contrast, which can be more easily attained and attributed an audit/accountability approach might either call to agency activities, and for which data can be easily for more independent assessments or for a system collected on an annual basis. Managers have relatively of spot-checks, reviews and verification of greater control over outputs and thus are under- management self-assessments. standably more comfortable with being held accountable for this lower level of results than for These conflicting aims present a dilemma for outcomes or impacts over which they have less donor agencies, as it does for other government influence. Moreover, outcomes and especially agencies implementing results based management. impacts are longer-term changes that may not Both uses should be kept in mind when establishing show improvements quickly or annually. Since performance measurement and evaluation systems. performance reporting is generally conducted To the extent possible, the systems will need annually, this further encourages managers to to address both uses and mix or balance data focus and report on lower-level results that will collection and analysis approaches to satisfy both show changes faster. Furthermore, there is a growing interests. For example, an independent series of concern among auditors and oversight agencies central impact evaluations might be undertaken with attributing results to agency interventions. to address auditor’s concerns about attribution, Since demonstrating attribution becomes increasingly while managers in the field might be encouraged difficult for higher-order outcomes and impacts, to conduct self-assessments employing more rapid this also acts to encourage managers to focus and appraisal and participatory techniques. report at lower results levels. Fu rt h e rm o re, accountability reporting tends to emphasize measuring Another potential conflict among performance what is being achieved (and comparing it to pre-set information uses is more unique to the donor targets), rather than analyzing why or how it is being agencies. Donor agencies are accountable not only achieved.In contrast,a management improvement to domestic stakeholder audiences but also to the approach is equally concerned with analyzing the partner country stakeholders. context and factors influencing performance, and with drawing lessons for improving performance. To the extent that donor agencies’ performance measurement and reporting systems may vary Accountability reporting versus management considerably for one to the next, partner country improvement uses also implies different data governments will have to deal with trying to collection and analysis approaches. For example, coordinate, compare and make sense of widely attributing outcomes and impacts to specific different donor agency approaches, frameworks, agency interventions requires rigorous designs and indicators, data, etc. Harmonization among donor data collection methods. It also implies extensive agencies of their performance measurement and attention to data quality, validity and reliability, reporting systems,particularly at the country level, and to independent verification. On the other would lessen the burden on partner country organ- hand, a management improvement approach izations. However, the extent to which this can be would tend to emphasize more rapid and low-cost accomplished may be limited given the variation in data collection/appraisal techniques, with data of government-wide performance reporting systems sufficient quality for decision-making needs but that have evolved in different OECD countries not necessarily up to standards required for social that may dictate the donor agencies’ approaches. 38 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing EVALUATION CAPACITY DEVELOPMENT IN THE PEOPLE’S REPUBLIC OF CHINA: Trends and Prospects 6 Ray Rist,Evaluation Advisor, The World Bank Institute INTRODUCTION Evaluation is relatively new in the People’s Republic of China (PRC). Indeed, before the early 1980s, it was unknown there. This unfamiliarity with evaluation reflected the orientation of the social science at that time, the virtual absence of any evaluation literature published in Chinese, and the lack of systematic contacts by chinese with those practicing evaluation in other parts of the world. Some activities under way, however, whithin the PRC did come to resemble evaluation, including some policy analysis,economic and management studies, survey research, project completion reviews, and what was broadly termed “experience summarization”. But these efforts were not called evaluation, nor were they systematic or focused on the issues now encompassed inour understanding of evaluation. In the 1950s, the PRC established policy and economic research institutes in national and ministerial, provincial and even some county governments. Over the past nearlyt 50 years, these institutes have undertaken a wide variety of studies using an array of analystic methods. Early work was largely economic, but the later work has branched out into studies of financial systems, social affairs, environmental protection, and sustainable development, to name four. Although there are few formal studies of the use of material from these institutes in national planning, a general consensus is that the institutes have been helpful. The policy community is aware of their work and sees them increasingly as sources of pertinent data of analysis. The material is largely anecdotal, but some important findings from the institutes have directly affected government decision making, especially after the economic reforms and openings to the world in 1978. As the reform movement in the PRC has grown over the past 20 years, a number of studies in agriculture, fiscal and tax policy, financial policy, foreign trade, and enterprise management have contributed signicantly to the formulation of reform policies. As the impetus for development grew from the 1950s to now, the PRC built methodological experience in how to study development projects and programs. It focused on the technology, engineering, and cost-effectiveness of development initiatives such as dams, highways, bridges, power plants, construction projects, irrigation, heavy industry, and railroads. Studies of these projects examined cost and quality control, financial benefit, and compliance with development objectives. As the 1978 reforms took hold, more capital and development assistance came into the PRC. As researchers built contacts elsewhere, comprehensive and regular evaluations were more often undertaken. The past 18 years have seen a growing capability in and understanding of technological and engineering analysis, E V A L UATION C A PACITY DEVELOPMENT IN THE PEOPLE’S REPUBLIC OF CHINA: Trends and Prospects 39 environmental impact analysis and modeling, undertaken evaluations, of many large nationally social impact analysis, environmental analysis, entities and line ministries. The CIECC is sustainability analysis, and implementation studies, systematically studying the policies and methods though in paractice not all of these are equally of evaluation, training staff from mulitple and conducted in a satisfactory manner. Building skill handbooks on evaluation practices. and experience in these areas for the PRC has been undertaken via a two-way flow of information and By June 1998, the CIECC had completed evaluations expertise — many persons from Cchina have gone of 58 national government projects and also a elsewhere to study these techniques, and many large-scale evaluation study of the projects consultants and experienced researchers in these constructed in accordance with the eighth state areas have traveled to China. five-year development plan (spanning from 1990 to 1995). It is now assisting the Asian De velopment Bank (ADB) to evaluate the performance of ADB’s whole portfolio in China. These studies can be characterized as general ly a combination of THE PRESENT SITUATION: performance measurement,financial analysis, cost- THE MAJOR A C T O R S benefit assessments, implementatin monitoring, and technical and engineering analysis. The CIECC The driving force for evaluation in China is the to date has done few systematic ex-post or impact massive and sustained surge in national development studies of large-scale national projects. and economic growth (the annual GDP has increased by 8% per year for the last 8 years). The A recent significant development is thatt, during attention and capability of the country to address the current government restructuring, the SPC evaluation questions comes form this concer with has been restructured as State Development development. Although most evaluation questions Planning Commission (SDPC) and the CIECC is comes from this concern with development. transferred from being the executing arm of the Although most evaluation is ex-post project SPC into an advisory group of the State Council. assessment, there is increasingly recognition that Compared to the SPC, the SDPC focusses much more on macro economic regulation and long-term evaluation issues are also embedded in all stages state development programming, and less on of the development project cycle. For this reason, annual investment planning. As a matter of fact, there is a growing awareness within China that the the annual investment plan of the SDPC covers evaluation function is applicable at all stages of only state financed large projects. the project cycle. There is now interest in linking evaluation to project and program formulation and In evaluation,however, the SDPC has an important implementation, and some ongoing evaluation has new duty of monitoring and supervising the already been undertaken, though comprehensively implementation process of state key projects. The doing so still infrequent. (cf. Valadez and Project Supervision Office of the SDPC, comprising Bamberger, 1994). mainly senior officials,is set up exclusively for this obligation. It has not yet been determined whether Before the reorganization of the government that the SDPC will conduct evaluation as the SPC did is now under way, the major national agency in before, but it is clear that if it will, only state China for project and program evaluation was the financed projects will be focused, and those projects State Planning Commission (SPC), whose major involving financing from bilateral and multilateral functions also include formulating macro regulation sources. It will not evaluate any projects wholly policies,preparing what large and medium projects financed by local governments with no funding need to be constructed each year all over the country. from foreign sources. This means that from now The SPC organized evaluation studies through on,local projects will need to be evaluated by local governments themselves. its Key Construction Department. The SPC organized evaluation studies directives, established The new functions of CIECC have not yet been policies and guilines for evaluation, and planed what defined. But it can be presumed that its role in projects or programs need to be evaluated and when. evaluation will by no means be weakened,but only The major executing agency for the SPC in project strenghtened. In fact, the CIECC has now a better appraisal and evaluation was the China institutional location than before in terms of International Engineering Consutling Company carrying out evaluation. Operating under the (CIECC). The CIECC, located in Beijing, has direct supervision of the State Council, it now 40 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing enjoys much more political clout and is almost evaluations for a number of projects in their own wholly independent of administrative agencies. As sectors,as required by either the SPO or their own a result, it can now access more easily information plans, with the assistance of their affiliated and data relevant to evaluation and can report research or designing institutes. directly to the State Council any evaluation findings and recommendations. It is essentially now in the The ministries and their institutes actively participate position to monitor the performance of the public in (and even sponsor) training seminars and workshop sector (cf. Mayne and Goni, 1997). Further, its of study tours to foreign countries on evaluation evaluation conclusions will hopefully be taken procedures and methodologies. Some of the more seriously by administrative agencies and ministries have developed their own evaluation project entities. However, there will be conflict of guidelines and directives, such as the Ministry of interest which will harm the objectivity of its communications, Ministry of Railway, Ministry evaluation work if the CIECC continues to do of Agriculture, Ministry of Health, and Ministry project appraisal. (Dr. HONG: WHY IS THIS of Water Resources. SO? CAN YOU EXPLAIN??) Admittedly, the evaluation capacity in the Another important agency, the State Audit provinces is still relatively weak, but a number of Administration (SAA), which in recent years has provinces have had in recent years good exposure made significant progress in the evaluation field, to evaluation. They have conducted self-evaluations also merits increased attention. Reporting directly as required by SPC and some international organ- to the premier, the SAA conducts audit of financial izations, participated in training courses on compliance and funds utilization as required by the evaluation methodology organized by central State Audit Law. The top management of the agencies, and some of them even have prepared SAA attaches great importance to evaluation work. their own evaluation guidelines. It will be a They have had extensive exposure to international significant challenge for China in the coming years practice and have a good understanding of the to effectively design and implement an inter- linkage between evaluation and auditing. The SAA governmental evaluation system that will allow has to date conducted considerable training for evaluation work to be linked and coordinated at their staff on evaluation methodology. They have the various administrative levels (cf. Rieper and also finished audits of financial and economic benefits Toulemonde, 1997). for 21 national key projects that were partially financed by lending from international organizations. One sector significantly influencing the development The SSA is now preparing guidelines,establishing of evaluation in China is banking. Many banks in data banks, and designing indicators for preparing China also evaluate projects they themselves have guidelines, and designing indicators for evaluation financed. They have explicit requirements regarding operations. According to its plan, the SSA will what percentage of their projects should be evaluated, have conducted a formal evaluation in 1998 and when to conduct an evaluation, what indicators to 1999. (DR. HONG. DO YOU KNOW HOW use, and what kinds of data are to be collected. MANY EVALUATION THEY PLAN TO DO Since many banks have been extensively commer- IN 1998?). The significant advantages of doing cialized during the past few years, their demand evaluation in the SSA is that it promises almost for evaluating their own lending operations is 100 percent independency from administrative tremendous. The Construction Bank of China agencies, holds the highest political clout, and is (CBC),formerly named the People’s Construction close to the top state leader. In its audit reports, Bank of China, for example, has developed systematic the SSA often makes critical but objective recom- guidelines and evaluated about 240 projects within mendations regarding what needs to be improved its portfolio. The State Development Bank (SDB), in investment management. China,will now, with as a non-commercial bank,has also issued an array the evaluation capacity of the SSA,be able for the of evaluation policies and guidelines. It also conducts first time to link up the budgeting proces with the evaluation regularly. In the banks, however, evaluation process (cf. Gray, Jenkins, and Segsworth, evaluations largely assess financial benefits, 1993). The SSA can collect information and data management performance, compliance with terms quickly and effectively, and provide good feedback of loans, and risks on loan recall. There is little in on its findings to the State Council and relevant the way of evaluation outcomes or impacts,studies administrative agencies. of stakeholder satisfaction, and sustainability analysis. The banks evaluate against criteria they There are evaluation activities in the line ministries have as lending institutions, evaluation policies as well. Many line ministries have conducted largely independently of the SPC. E V A L UATION CA PACITY DEVELOPMENT IN THE PEOPLE’S REPUBLIC OF CHINA: Trends and Prospects 41 The last key set of actor in evaluation within the information, (d) find appropriate means to convey PRC is made up of the bilateral and multilateral evaluation findings to the policy community, and development organizations, most notably the (e) create the development of evaluation as a World Bank, the Asia Development Bank, and sustainable effort. China is building the foundation. the United Kingdom’s Overseas Development However, there is no grand edifice in place. In the Administration (ODA). These three organizations, Chinese governmental structure and administrative in particular, have demonstrated sustained support hierarchy, several key tasks appear necessary at this for the development of an evaluation capability time if evaluation is to continue to develop. within China with their support, China has undertaken the following activities to strenghten First, it is vital for the PRC to have a strong central its evaluation infrastructure: organization for overall evaluation management s It has conducted studies of how other countries and coordination. Such a central organization in both the developed and developing world would carry out independent evaluations for state have established their national evaluation system financed key projects and state development and what are the possible options China to programs and portfolio, report relevant evaluation establish its own evaluation system; findings to the State Council, and disseminate s It has drafted evaluation guidelines, manuals, evaluation results to relevant stakeholders and even and handbooks; the interested public. Further, given the magnitude s It has financed senior Chinese officials to travel and diversity of the national administration, the to both developing and developed Countries to organization would also set comprehensive national study different evaluation systems; evaluation policies and guidelines so that any s It has provided training in evaluation to about one national organization unit undertaking an evaluation thousand officials, professionals, researchers; and will have a common understanding of policies and s It has organized a ministry-level seminar on methods necessar y. In addition, there is a strong evaluation. need for alignment among the key organizations involved in evaluation so that they share a common Many of these activities have been carried out understanding and build a common set of within the past four to six years, and there have approaches to different evaluation tasks. This already been notable results. First, the CIECC coordination has not existed to date, but many in and the SDB have both set internal units to focus these different sectors see the need to do so. exclusively on evaluation. Second, the CBC, similar banks,and a few sector ministries have established Second, it is important that the PRC establish evaluation divisions. Third, the SDPC has set up formal evaluation units, policies, and guidelines in a unit responsible for supervision of project imple- the ministries and banks. Such units would be mentation. Fouth, the SAA has begun to incorporate independent of their operational departments and evaluation into its benefit audit function, and adequately staffed and they might be required to Fifth, the Ministry of Finance has established a evaluate not only their own projects, but also function of cost-benefit analysis for state key projects. expand the analysis to include programs portfolios. Finally, in the area of knowledge dissemination, Such evaluations are an important instrument for some of those trained have now begun training the ministries and banks to maintain the quality of others on their own; and other central agencies, their portfolios as well as a basis for discussions major universities have incoporated on evaluation and decisions at the central level as to whether into their relevant modules, and some graduate further evaluations are required. students have chosen evaluation as the main subject of their degree dissertations. Third, it seems to be an appropriate time for the provincial-level and local governments to start building their own evaluation capability. The local governments have now almost 100 percent respon- WHERE TO GO FROM HERE sibility for their own investment activities, whithout interference from the central authorities. China has begun to (a) establish an evaluation Their investement portfolio are huge. Therefore, culture that legitimates serious inquiry into public the local governments need to establish an evaluation sector performance, (b) build the necessary expertise authority operating directly under the governor’s to undertake evaluation studies, (c) cultivate supervision. Such an authority would be responsible understanding and recognition in the program and for evaluating key projects and programs, drafting policy communities of the utility of evaluation evaluation policies and guidelines, organizing 42 E V A L UATION C A PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing evaluation training, and disseminating evaluation of large state owned enterprises. It is important to information. It appears essential to build a formal further disseminate such practices to the line evaluation function in the provincial operational ministries and local g overnments. departments as well, especially for those that have a large investment portfolio. Some departments Finally, the PRC needs to develop a supply of have already conducted evaluations and have well-trained evaluators for the many national relevant expertise and experience, but they do not ministries, provinces, and banks moving into the have a specialized evaluation unit and their evalu- evaluation arena. The supply at present is far too ation capacity needs to be strenghtened considerably. small to meet demand. Indeed, demand will probably outstrip supply for some considerable Fourth, the PRC needs to set up in the SAA an time in the future. Building training centers, auditing process of the evaluation function so that recruiting talented students and government there can be ongoing oversight and auditing of the employees, legitimating the professional status of evaluations undertaken within the line ministries evaluators, creating means of sharing experiences and evaluation policies and guidelines issued by and developents through journals and conferences, the central evaluation organizations, the relevant and linking evaluation skills to subject matter ministries, provinces, and banks. Evaluation units expertise will take considerable time and effort. But if need to be accountable and must adhere to standards China is to build the necessary capacity and expertise, of high performance, and the SAA has the right all these efforts need careful, sustained attention. institutional location and legitimate authority to supervise this. The oversight responsibility would be much like that performed by the U.S. General Accounting IN THE SHORT T E R M : Office, The Netherlands Court of Audit, the Swedish National Audit Office, and the Office of One task viewed as an important initial step to the Auditor General in Canada. the achievement of the evaluation goals set out above is the creation of a special central office, Fi ft h , the PRC needs to develop advanced which would have oversight responsibilities in the evaluation methods across these units and following areas: organizational entities. Much evaluation work to s Drafting of guidelines and procedures for the date has not been performed by such methods. ministries in the area of evaluation. Among the aspects that mostly need to be s Coordination of training, development of strenghtened are environment and social impact materials, and use of consultants; analysis, sustainability analysis, institutional s Development and maintenance of data banks assessment, program and portofolio evaluation, for its own work, the sector ministries,provinces, macroeconomic analysis, and evaluation of social banks, and the SAA; sector projects (cf. Valadez and Bamberger, 1994). s Building of evaluator networks and establishment Learning new approaches and bringing them into of a national evaluation association; evaluation requires training, building curricula, s Building the institutional capacity in universities translation of materials,and mentoring from those and research centers in order to ensure for the more experienced in evaluation. future supply of persons trained in evaluation; and Sixth, the PRC needs to introduce the monitoring s Dissemination of evaluation findings. and supervision function into its investment management agencies. Monitoring and supervision Given the movement in China toward a national provides direct examination and control of the evaluation system, the creation of such an office is quality of project implementation. Further, it an essential first step. At present, there is no helps collect data that are needed for evaluation. national coordination or oversight to ensure the Without good monitoring and supervision, it is best use of resources, prevent duplication, and set difficult to conduct good evaluation. The PRC has high standards. begun to recognize the importance of monitoring and supervision. Apart from the SDPC,the State Since the CIECC is now the advisory organization Council has set up a supervision office in the of the State Council and has relatively good expertise Ministry of Personnel specializing in monitoring in evaluation, such a central office could be well and supervising the performance of the managers placed in the CIECC. E V A L UATION CA PACITY DEVELOPMENT IN THE PEOPLE’S REPUBLIC OF CHINA: Trends and Prospects 43 IN THE LONGER T E R M : Tracking the changes and assessing the impacts associated with the development will necessitate The experiences of developed and developing an evaluation infrastructure, considerable expertise countries suggest that establishing national in both the subject matter and evaluation methods, evaluation system is a long and difficult process. It and development of routes for conveying evaluation takes system coordination and alignment that is findings into the policy communities. The second not easily achieved, even in small countries with source of demand comes from development relatively sophisticated public sectors. China is assistance given by the bilateral and multilateral not small, and its public sector is still developing. lending organizations. The loan portofolio of the PRC now represents billions of dollars. China is China can best approach this undertaking of the one of largest borrowers from the World Bank evaluation capacity building through a number of and the Asian De velopment Bank. The emphasis pilot projects targeted at sector ministries, by both the lenders and the Chinese government provinces, and banks with a strong emphasis on upon portfolio performance, indicators, and understanding what careful evaluation work evaluation of outcomes drives still more need for requires. This select group could test activities evaluation capability. related to capacity strenghtening, the building of methofological skills, development of reporting Finally, as China moves into the international formats, and training. The emphasis would be on arena, its policy communities will encounter new piloting different approaches to the institutional- evaluation developments,approaches,and forms of ization of evaluation in the public sector. In the utilization that will apply to China. There will be beginning, the emphasis would be more on encouragement to bring these developments home organizational learning than on organizational and test them out in the Chinese context. accountability, there may be a reluctance to test out new approaches-something essential in a pilot The major concern in the Chinese public policy phase. As lessons emerge from these pilot efforts, community is whether the initiatives and efforts the system can expand ministries and levels of described in these pages can be sustained. government. Over time, the balance between learning Building a national evaluation system in China and accountability can be worked through, and an will take decades. The infrastructure has to be evaluation culture can be developed. built piece by piece across ministries, provinces, and local jurisdictions. The first steps we have The demand for evaluation in china is great. described above are being taken, but sustaining Demand is being driven by social, technological, this marathon effort will take financial support, and environmental development of the country, technical assistance, the perseverance of officials, which is moving in a high and sustained level. and patience. 44 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing EVOLUTION OF EVALUATION IN THE PEOPLE’S REPUBLIC OF CHINA 7 Peter C.Darjes,Operations Manager ADB The process of building performance monitoring and evaluation capacity has been evolving in the PRC since the ear ly 1980s. It was,however, not until 1992, when the China National Audit Office (CNAO), the Ministry of Finance (MOF), the State Planning Commission, and other key central agencies, decided to develop specific proposals for recommendation to the State Council. The World Bank and the Bank have supported this process by providing a series of technical assistance for institutional strengthening and training. The World Bank supported a steering committee composed of representatives of the major public agencies under MOF leadership. This group drafted a proposal with various alternatives for an evaluation system. In 1994, the Bank provided a small-scale TA to develop project performance monitoring and evaluation capacity in the PRC.1 Under the TA, evaluation methodologies were developed and training was provided to selected Government officials. Similar objectives were pursued through TA to the State Development Bank.2 While the two TAs focused on selected agencies,3 the third Bank TA looked at evaluation as a governance function and therefore more as a generic activity.4 The TA provided a useful overview of the existing institutional arrangements,policies,systems,and practices for monitoring and evaluation in the PRC.The main objectives were to develop a plan of action to build long-term evaluation capacity, improve the institutional arrangements for better coordination among evaluation agencies, and harmonize the evaluation system, methodology, and practices to be used by all evaluation agencies. I.INSTITUTIONAL SET-UP 1. The evaluation system of the PRC involves many institutions. While CNAO has an oversight function for instituting evaluation at all government levels, its evaluation practice is confined to financial audits. Evaluation units have been established in most key central agencies. For example, the State Development Bank, Ministry of Construction, the China Engineering Consulting Corporation (CIECC), a Government owned consulting firm with close links to SDPC, and others have set up evaluation units. In all these cases,the evaluation units are being located at the highest levels of each organization to ensure independence from operational line responsibilities. 12 TA No. 2133-PRC:De veloping the Performance Evaluation Capability ofthe People’s Bank ofChina,for $100,000, approved on 9 August 1994. 13 TA No. 2664-PRC:Institutional Strengthening ofthe State Development Bank ofChina,for $500,000, approved on 16 October 1996. 14 The agencies covered by the TAs included the China National Audit Office (CNAO); State Planning Commission (SPC); China International Engineering Consulting Corporation (CIECC); State Development Bank (SDB) 15 TA No. 2821-PRC:Strengthening of Evaluation Capacity in the People’s Republic of China,for$500,000, approved on 4 July 1997. E VOLUTION OF EVALUATION IN THE PEOPLE’S REPUBLIC OF CHINA 45 2. SDPC is at the apex of the country’s planning urgency in the PRC. To stimulate domestic and performance evaluation activities. It is demand and to counter deflationary tendencies directed by the State Council to issue the in the economy the Government has embarked regulations and to commission evaluation of on a massive program of infrastructure invest- public sector programs. In the process of a ments. At the same time, poor implementation major reorganization in May 1998, SDPC results of large public-sector projects have has established an independent evaluation heightened the awareness of the Government department, the Key Project Inspectorate for more rigorous performance management Office (KPIO). Currently, 85 KPIO staff systems. In February 1999, a national conference monitors about 110 projects countrywide. As on project quality was prompted by a series of the number of projects will grow to eventually severe accidents and project failures relating to 700, the staff strength is projected to increase major investment projects. to about 200 in the medium term. SDPC’s evaluation work has been primarily focused on 6. In the PRC, despite the efforts exerted in the ex-post evaluation of projects. Thus far, very past, evaluation is still in its infancy. little work is being done in monitoring Therefore, the human resource requirements activities and evaluating operational perform- are significant and the need for both,qualified ance during project implementation. staff and training must be met as a matter of priority. Most KPIO staff has undergone some training at various PRC universities and in the context of external assistance. KPIO I I . FUTURE CHALLENGES management has, however, expressed concern that this training is inadequate given the tasks 3. While proposals have been made from time to ahead. Previous training activities are either time to devolve evaluation responsibilities to considered to be too short or not commensurate provincial public sector agencies, no formal with best practices. As KPIO is mandated links have yet been established. This is the with the training of future evaluators and challenge to be addressed by SDPC in the project managers, it must first focus on building future. The significance of performance up a nucleus of qualified trainers within its management and evaluation needs to be folds. Currently, there is no clear notion of the assessed in the context of effective governance. competencies and skills that would be It emanates from the delegation of authority required for project performance management. to subordinated line agencies that will be held The formulation of job descriptions will be a accountable for the accomplishments of tasks key task of the proposed TA. assigned to them. Thus, evaluation is a necessary corollary to results-based resource allocation. In the PRC, this thinking is slowly evolving. Currently, the emphasis of performance management is on inspection and control by I I I . PROPOSED ADB the central government. T E C H N I CAL A S S I S T A N C E 4. At the provincial government level, project performance management and evaluation is A. OBJECTIVES all but unknown. Apart from the central agencies 7. The overall objective of the TA is to continue mentioned above, there is no authority at the the process of building performance management provincial level that has a mandate or an appropriate organization to undertake project capacity within the PRC administration. The performance evaluation. In the process of focus will be on the central government level. decentralization, the Provincial Planning This will mean to improve the ability of Commissions were given project approval KPIO to develop its capacity and culture of authority. As a result, there are now a large evaluation. In this regard, the TA will act as a number of projects under provincial management. catalyst to enhance KPIO’s abilities,skills,and However, even at this level it is being increasingly knowledge, procedures and attitudes that will recognized that there should be a linkage enable it to improve the effectiveness of public between performance evaluation and project sector investments on a sustainable basis. planning, design and approval. Specifically, the TA will foster project performance management,including diagnosis, monitoring, 5. During the past two years or so, project and impact evaluation. A logical framework performance evaluation has assumed renewed for the TA is given in Appendix 1. 46 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing B. SCOPE coordination and guidance for all evaluation activities in the country. In the absence of 8. The scope of the TA will follow the recommen- these preconditions, it is difficult for evaluation dations and the action plan prepared under the methodologies to be harmonized and refined, previous Bank TA. As such, the TA will focus an evaluation culture to be formed,and for the on human resource development of KPIO. evaluation profession to be properly recognized. Towards this end, the scope will include: KPIO’s mandate includes all facets of impact evaluation, including reviews of project (1) Forming a partnership with one or more design, monitoring of benefit development, international partner agencies that have a bidding procedures, construction schedules, comparable mandate. This would most quantity and quality of physical progress as likely be a government department. It is well as costs, prices and fund utilization. considered that the most effective way of KPIO has shown a keen interest in adopting transferring know-how would be through the Bank’s Project Performance Management such arrangements; System (PPMS) and has translated the Bank guidelines into Mandarin. KPIO would (2) Identifying skills and key competencies appear as the most promising entry point required for KPIO staff.The foreign partner for transmitting that system to the PRC will help introduce appropriate procedures evaluation administration. for project performance management; (3) Based on the above activities, preparation of GETTING THE PROVINCES ON BOA R D a training curriculum and a training program; 10. The world over, the significance of evaluation (4) Implementation of training activities: as an element of governance has evolved in Training will center on seminars and the process of delegating responsibility and workshops to be conducted in the PRC. accountability from central government planning A limited amount of the TA funds will be levels to line agencies and subordinated provided for overseas training. All train- provincial administrative levels. For KPIO to ing activities will be programmed and become effective, the provinces must over carried out in close association with the time match KPIO’s efforts in undertaking foreign partner institution; project performance management. This should, as a start, include all projects under (5) Preparation of project performance evaluation central Government financing executed by the guidelines.The guidelines should include provinces, and should later cover projects the use of teaching methods and tools; financed by provincial governments alone. The longer-term need to appropriately (6) Evaluation of selected Bank-financed including the provincial administrative level projects as case studies; in performance management is addressed by the proposed scope of the TA. (7) Preparation in outline form of a program of activities for a follow-up phase of capacity building TA,which would focus on spreading LONG-TERM PERSPECTIVE project performance management practices to provincial governments, and; 11. Building an effective evaluation capacity in a country is a long-term developmental process (8) Provision of office and training equipment. and the process may take decades.The process has been evolving in the PRC since the early 1980s. During this relatively brief time span, remarkable progress has been achieved in the PRC.However, given the size and complexity I V. KEY ISSUES of the country, particularly with its ongoing economic transformation, a major effort APPROPRIATE ENTRY POINT would still be required to build an effective project and program performance evaluation 9. Institutionally, there is not a high-level central system covering all concerned entities in the authority in the Government to provide overall Government. E VOLUTION OF EVALUATION IN THE PEOPLE’S REPUBLIC OF CHINA 47 TRANSFERRING THE REQUIRED KNOW-HOW on technical assistance, it will largely be intended to assist staff to understand and 12. Project management will have to consider the master a process. Organisations interested in optimum source of technical assistance for this partnership arrangement will have to show each type of activity under the project. In this how they intend to develop a relationship based regard, a priority activity for the TA will be on mutual learning, equality and respect, with the selection of one or more organisations to sustainability of process as a long-term goal. enter into a partnership arrangement with KPIO. A partnership arrangement is more The partner agency will have a number of functions, complex and demanding than a twinning including the provision of technical assistance, arrangement. Twinning arrangements are professional development support and mentoring often based on an expert-client model. A part- for KPIO curriculum development and in-service nership approach may involve some twinning, staff, as well as the provision of work attachments. but takes a more developmental approach The partner organization(s) would have to be based on an analysis of the current and future selected through competitive bidding. This will skills and needs of both organisations. While not be easy given that the prospective organizations a partnership will have some necessary focus would likely be in the public sector. 48 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing LINKING EVALUATION TO POLICY FORMULATION AND BUDGET PROCESSES: Lessons Learned from Australia 8 Leonard J. Early, Dept. of Finance and administration, Australian Public Sector INTRODUCTION Australia has had a well-developed system of policy evaluation that has been firmly linked to the Budget process. The system developed over many years,but is now in a state of flux – its future is far from assured.The growth and possible future decline of the Australian model can provide some interesting lessons about development and maintenance of evaluation capacity in other countries. THE HISTORY There has been a long history of policy evaluation in Australia. A number of policy research bureaux (e.g. the Productivity Commission, the Australian Bureau of Agricultural and Resource Economics and the Bureau of Transport and Communications Economics) have had a major impact on policy formulation in Australia for many decades. 1 In addition, major public enquiries have frequently been commissioned to evaluate economic and other government policies. For example, the Vernon Committee2 undertook a major review of Australian economic policy in the early 1960s, and the Crawford Study Group3 evaluated industry policy in the late 1970s. Evaluation was given significant impetus with the election of the Hawke Labor Government in 1983.In the period 1947 to 1983, Labor had been in Government for only three years, when Gough Whitlam was Prime Minister, from 1972 to 1975. The Whitlam government never achieved legitimacy in the eyes of many Australians, and it was dismissed in controversial circumstances by the Governor General. The Hawke Government was led by young Ministers with tertiary qualifications in law, economics and commerce, who were determined not to repeat the mistakes of their predecessors. They intended to stay in office sufficiently long to stamp their philosophy irrevocably on Australian public policy. The Hawke Government had come to office with a well-articulated public service policy, designed to ensure that Ministers were firmly in control of the public policy agenda. Upon its election, the Government published a White Paper on public sector reform4 and embarked on a reform program, aspects of which included: s The introduction of program budgeting with its focus on program objectives and program performance rather than on inputs; and 1 Participating in other agencies’ evaluation activity became a major part of Department of Finance budget officers’work. By around 1993, the Department of Finance had to limit its involvement to major evaluations. 2 The Hon David Kemp, Minister Assisting the Prime Minister for the Public Service, Ministerial Statement,Reforming the public service to meet the global challenge, Public Service and Merit Protection Commission,1998. 3 This action was unprecedented in the Commonwealth Government in Australia. 4 Within a set of fairly loose controls. LINKING EVALUATION TO POLICY FORMULATION AND BUDGET PROCESSES: Lessons Learned from A u s t r a l i a 49 s Devolution of authority and responsibility to potential program improvements – including of program managers to enable them to achieve program objectives and performance information. the desired objectives. s Internal evaluations, undertaken entirely within agencies, usually short exercises aimed at improving program management and efficiency. THE INTRODUCTION OF A FORMAL EVALUATION POLICY By the middle of the 1980s, it was clear that further BEDDING DOWN reform was required if the Government’s objectives THE NEW A R R A N G E M E N T S were to be met. Program objectives and performance were not well integrated into the Budget process. In the early years,some agencies regarded the need Program resourcing did not depend on demonstrated to produce evaluation plans as an unnecessary program performance and, as a result, program bureaucratic requirement. The quality of plans managers did not have the incentive to produce was variable and, in many cases, evaluation good performance information. Program performance planning was not well integrated into strategic information was poor, and public service managers planning within agencies. Evaluation capacity in were not performance-oriented. Program budgeting agencies was mixed, as was the quality of evaluations. had not delivered the requisite cultural change in public sector management. Into the 1990s, a separate evaluation unit was established in the Department Of Finance to over- Further public sector reform in 1987 and 1988 sight the evaluation strategy, and to assist agencies included the introduction of a formal evaluation with evaluation plans, with advice on strategy. This strategy had a number of elements: evaluation methodology and (in some instances) s Each policy program was evaluated every three with evaluations. That unit delivered evaluation to five years. training to hundreds of policy analysts and s Each year, each portfolio submitted an produced evaluation handbooks. evaluation plan to the Department of Finance spelling out the portfolio’s evaluation program Evaluations were increasingly used and demanded over the next three years. by Ministers in taking Budget decisions. (Indeed, s Each new policy proposal submitted for Budget through the 1990s, it became increasingly common consideration included a proposal for future for the Expenditure Review Committee – the Cabinet evaluation of the initiative. sub-committee responsible for producing the s Evaluations were normally to be published. annual Budget - to refuse to consider policy proposals if scheduled evaluations had not been completed.) This strategy recognised the full range of Evaluation units (of varying sizes and capacities) evaluation activity. It did not only focus on more were established in most of the bigger portfolios. formal evaluation processes. It encompassed dif- ferent forms of evaluation, addressing program Most agencies quickly learnt that major Budget appropriateness, effectiveness and efficiency. For proposals were more likely to be accepted if they example, the strategy included coverage of: were supported by sound evaluation (particularly if s High level external evaluations (usually on that evaluation had also involved the Department politically more sensitive issues), requiring Of Finance and if that department supported particular expertise, complex analysis and the evaluation results). As a consequence, the possibly leading to significant change. Department of Finance was routinely involved in s Evaluations involving both the Department of steering committees of major evaluations5 and Finance and the line agency; usually taking evaluation became firmly embedded in policy less than one year and typically focused on development and in Budget processes. 4 For example: • The Department of Finance and Administration was formed by amalgamation of the Department of Finance and the Department of Administrative Services. On its formation in October 1997,the Department had 2,300 employees. As a result ofa series of reviews (some of which had been commissioned before the Department was formed) activities have been abandoned, re-engineered or contracted out. As a result,employment in the Department has now fallen to around 700; • The provision of IT services to all government agencies is being substantially outsourced to the private sector; and A new agency, Centrelink, has been created and required to compete with the private sector to deliver employment services to the unemployed. 50 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing By 1997, the Australian National Audit Office The government substantially rationalised central was concluding that evaluation plans and the quality controls on agency operations. Agencies became of evaluations was general ly satisfactory (although free to set their own wages and terms and conditions more needed to be done to ensure evaluation results of employment; financial controls were substantially were used to improve program performance). rationalised; and most central reporting requirements were rationalised or abandoned. However, at the same time, portfolio evaluation plans had exploded in length and complexity, until The winding back of central controls included reports of 120 pages or more were the norm. The abolishing the requirement to produce portfolio Department Of Finance and some agencies were evaluation plans. Agencies were to report (very becoming concerned at the cost of this bureaucratic briefly) evaluation plans in Portfolio Budget overhead, and beginning to question whether Statements. In practice, the specification of evaluation had become sufficiently embedded in evaluations in Budget documentation has been public service culture that the reporting burden patchy, with some agencies (most notably the could be significantly rationalised. Department of Finance) paying this requirement scant attention. The evaluation unit in the Department Of THE NEW PHILOSOPHY Finance has been disbanded and the department no longer sees its role as assisting agencies with The conservative Howard coalition government evaluation methodology (although it does continue was elected to power in 1996. The new government to sit on steering committees of significant was highly sceptical of the efficiency and effectiveness evaluations – often at the request of Ministers or Cabinet Committees). of the public service. It has stated6 that on coming to office it was confronted by: The Government has required managers to review s A public service with major inflexibilities in its all their activities to see if they should be discon- administrative, industrial and financial dealings; tinued, privatised, devolved to another level of s Costs of many processes that were high because government or otherwise improved, e.g. by market of a culture of compliance with detailed central testing. This entails a fundamental evaluation of rules and regulations; the appropriateness,effectiveness and efficiency of s A Commonwealth bureaucracy that lagged all government operations. There have been some behind other governments and the private significant changes, but most agencies are yet to sector in its efficiency and in the quality of its commence the process. management; and In 1999, the Commonwealth government produced s A cash-based accounting system that made its first accrual Budget. The accrual Budget is accurate assessments and comparisons of public produced in terms of outputs and outcomes. This sector activity impossible. provides a link from Budget resourcing through the outputs produced by agencies to the outcomes the The Government rapidly set out to address these government wants to achieve for the Australian people. perceived deficiencies. It dismissed six of the 17 portfolio secretaries (chief executive officers);7 Budget documents detail the proposed price, introduced a Budget which cut $A8 billion (or quantity and quality of agency outputs, and link around two per cent) from Commonwealth these to desired outcomes. The documents spell out Government outlays and introduced a sweeping detailed performance targets and performance against range of public service reforms. those targets is to be reported in annual reports. 6 Not surprisingly, the quality ofthis material is highly variable in this year’s Budget,with much progress yet to be made. 7 Government activities are classified either as Departmental outputs (the goods and services produced by Departments and agencies on behalf of Government for organisations or individuals) or administered items (controlled by the Government and managed or oversighted by Departments and agencies). Examples of Departmental outputs are policy advice, evaluation activity or the activity of payment of pensions or benefits. The benefits (e.g. unemployment benefits and age pensions) which are determined by Government and that are a statutory right of people who meet the eligibility criteria,are, administered items,as are most financial payments to the States and Territories. LINKING EVALUATION TO POLICY FORMULATION AND BUDGET PROCESSES: Lessons Learned from A u s t r a l i a 51 The concept of program essentially disappears. that departmental expenses comprise only around Thus the requirement to review programs every 10 to 20 per cent of the Commonwealth Budget, three to five years effectively lapses (although it any such development would entail obvious risks. has not formally been repealed). There is a rolling program of price reviews that effectively evaluate performance in delive ri n g Departmental and agency outputs. However, there LESSONS FROM AUSTRALIAN EXPERIENCE is a gap, in that there is no longer any formal requirement or process to regularly review admin- istered items. In implementing an evaluation strategy, Australia started with some significant advantages, including: s A history of evaluation at least of major policy initiatives; s A strong central statistical unit producing I M P L I CATIONS OF THE NEW PHILOSOPHY quality data; s A well educated bureaucracy, with a good leavening of policy and evaluation skills; Prior to the election of the Howard Government s A history of public provision of information in 1996, the efficiency of government arrangements on program performance; and had not been subject to much scrutiny or evaluation. s A sound Budget system, including four year The major instrument for improving efficiency forward estimates of Budget outlays and revenues. had been the efficiency dividend, which had been introduced in 1987. With the efficiency dividend, Even so, it took ten years for the evaluation strategy agency running costs were reduced by one per cent to become firmly embedded in the Australian public each year. While some agencies found this regime service culture. difficult, in fact it was highly ineffective in improving efficiency. Most agencies had little difficulty in Mackay has identified factors in the success of obtaining new resources through the new policy Australian evaluation policy as: process in the annual Budget, and managers had s The creation of an explicit, whole-of government little incentive to re-engineer or restructure their evaluation strategy; operations to improve efficiency or effectiveness. s The existence of a powerful central department Most government operations were effectively sheltered that has been a committed champion of evaluation; from any effective scrutiny or competitive pressure. s Sustained commitment and support the evaluation strategy over a decade; and Evaluation activity was mainly focused on programs. s Implementation of related public sector Few evaluations shone the spotlight on the management reforms that have given appropriateness,effectiveness or efficiency of those considerable autonomy to line managers and government operations that delivered the programs.12 that emphasise bottom-line results – providing incentives to conduct and use evaluation findings. This situation has changed dramatically. Attention is now firmly focused on the public I agree with this analysis as far as it goes. service and its effectiveness and efficiency. The However, to the list of factors Mackay identifies, intention is to subject the vast bulk of government I would add the existence of a strong supportive activity to competitive pressure – with agency group of key Ministers throughout the period. (I resourcing based on competitive prices of outputs see the consistent demand of the Expenditure rather than on the cost of their production. Review Committee (ERC) for evaluations as one of the most important reasons that evaluation The additional evaluation of government activities flourished in Australia over this period). is a welcome development. However, the risk is that this entails a shift in the focus of attention I think the Australian experience also illustrates away from the effectiveness of programs. Given the potential fragility of the evaluation culture. 52 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing While current Ministers are, if anything, even In my view, there is no doubt that a requirement to more firmly focused on performance than their report evaluation plans and performance is necessary predecessors, the assumption is that greater to get the requisite response in Departments and management flexibility, in tandem with increased agencies. However, this reporting requirement market discipline (including by privatisation) will should not be allowed to burgeon into a bureau- underpin performance improvement. cratic monstrosity. Reporting regimes should be kept simple. Evaluation resources in Departments Evaluation has a role in the process,but this role is and agencies should be directed to evaluating, not greatly diminished, in that the regular evaluation to completing lengthy reports for central agencies. of the appropriateness, effectiveness and efficiency Is it necessary for countries to “walk before they of the bulk of Government outlays is no longer can run”? In other words, is it necessary to first guaranteed. focus on provision of simpler performance information before moving to implement more A moot point is the extent to which evaluation sophisticated analytical techniques? I would suggest not – in my view, it would be desirable to move on will continue to flourish in this environment. both fronts simultaneousl y, albeit the adoption of Developments thus far suggest that the culture more sophisticated analytical techniques would is unlikely to change quickly in the bigger need to be at a pace that recognised the availability Departments and agencies where evaluation of skills to support the analysis. is firmly embedded – e.g. in those Departments that have established their own evaluation units. Public sector reform typically comprises a focus In other organisations, the stimulus to evaluation on outputs rather than inputs, and greater devolution is likely to come from the Department of Finance of authority and responsibility to line managers. and Administration in the annual Budget process. Performance objectives and performance information However, given that only around two per cent clearly have a role to play in this process. However, of Government outlays typically come up for the Australian experience suggests program managers detailed scrutiny in the annual Budget,this implies are unlikely to produce good performance information a significant potential reduction in evaluation unless it is an integral part of the Budget process. activity in some areas. However, the use of performance information in the Budget cannot be automatic. There will be a need of judgement in interpreting the performance information. Evaluation is necessary if perform- I M P L I CATIONS FOR OTHER COUNTRIES ance information is to give up its meaning. An interesting question is the extent to which For example, a failure to perform satisfactorily other countries might be able to learn from and could call for: emulate the Australian experience, particularly if s A reduction in Budget allocation (e.g. if the their starting environment appears less conducive program is proving too expensive and the outcome to success. does not justify the cost); s An increase in allocation (e.g. if the program In my opinion, the Australian experience does is a high priority for the Government and provide some useful guidance. It would suggest that cost is a lesser issue); or introduction of a systematic process of evaluation s No change (e.g. if a road program has been is more likely to be successful if: delayed by inclement weather). s There is an explicit whole-of-government policy championed by the Budget agency; The important point is that the appropriate s Ministers strongly support the evaluation policy Budget response can only be determined in and demand and utilise evaluations in the the light of evaluation and analysis of the Budget process; and performance information. Evaluation is necessary s Public sector reform has given line managers for a proper interpretation of the results. One needs sufficient autonomy to manage their resources to ask questions such as: to achieve specified objectives. s Are results good because targets were set too low? However, there is no miracle cure. Building an s Are results poor, despite good performance, evaluation culture requires sustained effort over an e.g. because the agency is poor at producing extended period. accurate and timely performance information? LINKING EVALUATION TO POLICY FORMULATION AND BUDGET PROCESSES: Lessons Learned from A u s t r a l i a 53 s Has there been goal displacement (e.g. lowering reasons for good or bad performance and evaluation service quality in other areas) in order to is the most effective means of shedding light on achieve the target? such questions. Sound evaluation both requires and increases the value of good performance information. In summary, one needs to understand the underlying The two go together. 54 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing PUBLIC PERFORMANCE EVALUATION AND IMPROVEMENT 9 Marc Holzer, Executive Director, National Center for Public Productivity, Rutgers University , NJ, USA PRESSURES FOR PERFORMANCE “Government must be accountable for results.”“Government must do more with less.” The ‘bottom line’ for government is that such statements, voiced repeatedly by policy makers, citizens, and the media, are deceptive in their simplicity. They raise some very basic questions: What are the desired “results” of government? What does accountability really mean? Faced with competing demands and expectations, and social issues that seem to defy measurement, the “results” of public sector initiatives are difficult, at best, to define. Although we might achieve consensus on broad platitudes, the “devil is in the details.” How can we develop an objective approach to accountability, to evaluation and improvement, which will be accepted as impartial, utilized as a means of adding value to public decisions, and applied to positively impact the quality of the lives of citizens? Our argument is that these multiple objectives can only be achieved if policy makers and citizens are actively involved in the process of performance measurement - the evaluative process of holding agencies accountable for their expenditures, actions and promises. Until recently, government accountability was largely a matter of financial accounting, hence its name. When public funds were appropriated, the key accountability questions focused on how much money was spent and on what specific items: personnel, supplies, travel, communications, etc. Today, the concept of governmental accountability has taken on a much broader meaning. According to Romzek and Dubnick (30) “Accountability is a relationship in which an individual or agency is held to answer for performance that involves some delegation of authority to act. Accountability mechanisms are the means established for determining whether the delegated tasks have been performed in a satisfactory manner or not. Thus governments,in an effort to demonstrate accountability, need to show their policy officials and service users: (1) what they are getting for their tax dollars, in terms of services (and sometimes products): public hospitals, roads, airports, libraries, water supplies, etc... (2) how efficiently and effectively their tax dollars are spent: cost per mile of roads built to certain standards, cost per pupil of students graduating from high school or university. (3) how such expenditures benefit their lives or the lives of those they care about: eradication or suppression of diseases such as encephalitis or AIDS; high sense of security in a neighborhood; safe and reliable water supplies. PUBLIC PERFORMANCE EVALUATION AND IMPROV E M E N T 55 This type of accountability holds government Performance measures can be quantitative (average responsible not only for its actions,but also for the response time) and qualitative (how safe people results of its actions - the impact of each service on feel in their parks during the day or at night). each citizen. Managers can develop performance measurement systems incorporating a variety of performance In order to emphasize to policy makers and service indicators, such as those defined by the users what they are getting for their tax dollars, Government Accounting Standards Board(4) in agencies need to be able to evaluate-measure and the United States: report- what they are accomplishing. Measures which are unreported,suppressed or poorly reported Input indicators. These measure the resources are virtually useless in improving performance. expended on a program, such as the amount of Well reported (i.e. well displayed and widely money spent or the total number of employee- distributed) performance measures emphasize how hours needed to deliver a service. well a government is doing at meeting its citizens’ needs and at living up to its commitment to Capacity indicator. These measure the ability provide quality services for the taxes people pay. of an organization to deliver services. Capacity Performance measures can help officials, public measures helps managers evaluate the level of managers, and citizens evaluate the impact that training (how recent, percentage of employees various expenditures have on the quality of their actually trained, level of training, etc.), the state of lives and the lives of those they care about. physical facilities (space, comfort,safety, etc.), the Performance measures also enable officials, public readiness of systems (Y2K compliant, computer managers and citizens to see the outcomes, or networking, memory and speed, etc.). results, of specific government services and to place a relative value on each service provided. Output indicators. These report the quantity of products or units of service provided to a service Evaluation and measurement of performance has population.They also include “workload” measures always been implicit in questions of outputs and that reflect the amount of effort expended to outcomes: Is crime increasing? Are the streets produce a product or provide a service. Examples cleaner? Is the air quality better? How well are our of output indicators include: number of meals children doing in school? In short, is the program delivered;miles of road paved;number of students producing as promised; as expected? What are passing the high school proficiency test. the results? Performance measures provide an opportunity to answer such questions with specific Outcome indicators. These measures report the data for evaluation, rather than the more common results of programs and services. Outcome indicators situation: subjective statements based on vague possess quantitative and qualitative aspects. Examples assessments of efficacy: the neighborhood seems of outcome indicators include: the number of safer. “Litter here is worse than ever.” “The schools individuals employed six months after participation in this town are very good!”In the absence of data, in a job training program or percentage of residents these statements are merely impressions. who frequently enjoy using a park. Efficiency and cost-effectiveness indicators. These measures focus on how a goal is achieved, rather than what was achieved. Specifically, PART I: DATA FOR efficiency indicators refer to the ratio of the level E V A L UATING PERFORMANCE of service (tons of refuse collected, number of meals delivered) to the cost, in dollars and labor, of providing It is helpful to think of performance measurement the services. They measure the cost per unit of an as a process or system of measures and procedures, output or outcome. Examples of efficiency indicators whereby organizations assess how well they are include: cost per meal delivered; cost per ton of doing compared to previous performance, to other garbage collected; cost per pupil educated. organizations (“benchmarking”), and in terms of how well they are achieving their stated goals Productivity indicators. These measures,according and objectives. A well-designed evaluation and to Ammons(1), combine the dimensions of performance measurement system should clearly efficiency and effectiveness in a single indicator. articulate service goals and objectives, define service For example, the number of meals delivered per outputs and outcomes, and specify the expected hour measures the efficiency; the number of quality levels for these outputs and outcomes. meals delivered on time (and warm) measures the 56 E V A L UATION C A PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing effectiveness. The unit cost (labor-hours) per on basic programs that need the most managerial time (and warm) delivery measures productivity. oversight and where accountability reporting is most important. Designing a good performance measurement system may be challenging to public managers not accustomed to measuring or setting performance 2 . STATE THE PURPOSE AND IDENTIFY THE DESIRED OUTCOMES targets. However, this effort should not be dismissed as too complex or too difficult. Developing a system Typically, a government ministry, department or involves an understanding of what the program agency initiates a strategic planning process to is trying to accomplish, who the main users or clarify its mission, goals, and objectives. Through customers are, and a basic knowledge of the current this process, the agency can identify the outcomes level of service. The Government Performance or results it wants to achieve through its programs. Results Act of 1993 in the United States represents A manager can only measure the performance of a an effort to institutionalize performance measurement program if its purpose is clearly stated. Preparing systems at the federal level(8a). The act requires a well-articulated statement of purpose is a very federal agencies to: important first step. Ideally, a clear mission statement is the starting point. If that is not available, a thorough s establish top-level agency goals and objectives, program description is a good place to begin. For including annual program goals; example, GASB (12) offers the following statement of purpose for a public transportation system: s define how they intend to achieve those goals,and; The basic purpose is to provide safe, dependable, s demonstrate how they will measure agency and convenient,and comfortable transportation services at program performance in achieving those goals. minimum cost to the citizens,including special clients groups such as the handicapped and the elderly. A good evaluation and performance improvement system should include the following seven-step system developed by the National Center for 3 . SELECT MEASURES OR INDICA T O R S Public Productivity at Rutgers University-Campus at Newark: A good system uses a few selected indicators to measure outcomes and performance.Most government programs that have established performance 1 . IDENTIFY THE PROGRAMS TO BE MEASURED measurement systems incorporate the indicators described above: input, capacity, output, outcome, To start with, programs to be measured must be efficiency and productivity. For example, in a clearly defined. Programs are groupings of routine sanitation department the input measures might activities aimed at providing support for specific be the amount of labor hours; the operating budget; public services. Groupings of individual activities the number of vehicles on the road. Output measures make up a program. For example, the following could include tons of refuse collected; number of activities-street resurfacing; street patching; seal households served; number of missed collections. coating; and curb repair-constitute a program that Efficiency measures could include labor-hours per is traditionally called street maintenance. Usually, ton of garbage collected or household served;dollars programs are defined by governments and are spent per 1,000 households; and productivity generally listed on an organizational chart indicators could include measures such as the cost contained in the operating budget. Programs relate per mile of clean streets; the cost per household of directly to the organizational structure and the twice weekly refuse collection. managerial areas of responsibility. Choosing what programs to measure is a matter of 4 . SET STANDARDS FOR PERFORMANCE AND judgement. On the one hand, programs should OUTCOMES (TARGETS FOR A C C O M P L I S H M E N T ) not be too few, so that only a small portion of services are covered or the information collected is Under this step, public managers should specify insufficient. On the other hand, too much reporting the conditions under which program goals and can be excessively costly, or overwhelming and objectives are met. Managers need to determine impractical. Generally, performance measurement what service effectiveness and quality means for a systems work best when they concentrate on particular program and explicitly state how they collecting limited but essential information about are going to determine whether the stated terms of PUBLIC PERFORMANCE EVALUATION AND IMPROV E M E N T 57 effectiveness and quality have been met. This planning to reevaluate goals and objectives and to involves comparing actual program outcomes or adjust priorities. Another use is in managing for results against some agreed-upon standards,including: results where outcome information is used to promote continuous improvement in program s previous performance (the percentage of residents operations and results. In terms of analysis for who feel safe in their neighborhoods this year action, a well-developed performance measurement compared to last year); and evaluation system will enable managers to spot weaknesses and challenges to program delivery, s performance of similar organizations (the as well as program strengths and opportunities percentage of residents who feel safe in their for improvement. neighborhood compared to the percentage in a neighboring community); s performance of the best organizations (the percentage of residents who feel safe in their PART II: PERFORMANCE neighborhoods compared to the percentage of residents who feel safe in their neighbor- MANAGEMENT FOR IMPROV E M E N T hoods in the recognized “safest communities” in the country); It is useful to distinguish between performance measurement and performance management for s pre-set targets, (next year 85 percent of all improvement. Performance management for improve- residents will feel safe in their neighborhoods, ment is broader than performance measurement. and in three years 95 percent will feel safe.) Pe rf o rmance management for improvement incorporates the results of performance measurement into general management practices. In the U.S. 5 . MONITOR RESULTS the National Performance Review (25a) defined performance management as “the use of performance Each accomplishment target should be monitored measurement information to help set agreed-upon on a continuous basis. Monitoring provides the performance goals,allocate and prioritize resources, manager with the results needed to decide whether inform managers to either confirm or change current or not the performance targets have been met. policy or program directions to meet those goals, Systematic and periodic monitoring gives the and report on the success of meeting those goals.” manager an opportunity to keep track of the operation of the program, and take corrective It is essential that performance measurement be action when necessary. Usually, monitoring will considered an inherent and indispensable part of vary depending on the program and target accom- the management for improvement process. plishments. For the most important programs and Measurement for measurement’s sake alone is services, monthly data collection and reporting insufficient. Managers can have stacks of data at systems that indicate results will be necessary. their fingertips, but unless they use that data to improve policy and procedures it is virtually useless. Osborne and Plastrik write in their book 6 . PERFORMANCE REPORTING “Banishing Bureaucracy”(27) “We have not listed performance measurement as an approach because A good performance measurement system reports we do not believe the act of measuring itself has program results on a regular basis. The report enough power to force fundamental change in public focuses on what was accomplished and what it cost the public. Reports should be brief, convey organizations. It is critical competent organizations information graphically, and include minimal need it... But some public organizations have measured explanatory information. Information should be performance for years, with virtually no impact.” presented in such a way as to facilitate comparisons over time, comparisons with similar organizations, As a means of achieving measurable impacts, we comparison with best programs nationwide and have found the following system to be an effective against pre-set targets. integration of measurement and improvement factors. This conceptualization, or “roadmap,” is based upon our research has to best practices, and 7 . USE OUTCOME AND PERFORMANCE INFORMATION in particular the experiences of award-winning Exemplary State and Local (EXSL) programs Information from an effective performance identified by our National Center for Public measurement system is regularly used in program Productivity (19). Today, to produce public services, 58 E V A L UATION C A PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing the best public organizations have developed Although citizen participation can ultimately multiple, reinforcing capacities. In particular, improve the level and quality of municipal service government agencies which have been formally provision by making services more responsive to recognized as high achievers, as state-of-the-art: the needs of citizens,it is often difficult to achieve. s apply quality management principles; While citizen involvement typically results in s use measurement as a decision making tool; effective policy and meaningful public sector, (32) s work hard to motivate employees; active citizenship is often perceived as burdensome, s adapt new technologies; and costly and time consuming (25, 33, 34). The s develop public-private partnerships. traditional,top-down,hierarchical model of public administration limits the role of citizen participation in performance measurement. Performance measurement systems may be easier to design and implement when citizens are excluded and PART III: GETTING POLICY MAKERS managers are the ones determining what will be AND CITIZENS INVOLVED measured and how it will be measured. Although easier to implement, such performance measurement Policy-level and citizen involvement in performance systems fall short in terms of measuring what measurement may help public managers stay matters most in a community. Citizen involvement focused on what really matters in their communities. increases the social relevance of indicators by Citizen involvement can increase the impact of combining facts—hard data—with value-how performance measurement by encouraging managers citizens feel. to look beyond traditional output measures (that often have little meaning to citizens) and instead Tangible benefits can be derived from effective focus on quality of life issues and community citizen involvement and participation not only in goals. When you think about it, why do managers performance measurement, but in public decision care about government performance? They want to making as well. Thomas (33) indicated in “Public improve the quality of service delivery and ultimately Participation in Public Decisions” that public the quality of life in the communities they manage. involvement can increase the effectiveness of public managers and the decisions they make. For example: The overall goal of involving policy makers and citizens in performance measurement is to build a) Decision quality may improve as citizens and lasting processes that involve citizens in assessing citizen groups add to the information available municipal performance so that government policies for making decisions. That information and services reflect the needs of the community. might prevent repetitions of many ill-advised public decisions. Policy makers, citizens and public managers, together, can establish performance measures that b) With citizens involved in making decisions, are meaningful to both citizens and managers. So, acceptance of the decision may increase, for example, instead of just calculating the tons of enhancing the likelihood of successful refuse collected, public managers might also ask implementation. policy officials and citizens to rate the cleanliness of their streets. Instead of counting the number of c) If citizens assist in service delivery, services may health workers in specific neighborhoods, public become more effective and more effiscient. managers could also ask citizens how well they feel in their own neighborhoods. d) As involvement increases,citizen understanding of governmental operations may increase The relevance of performance measures increases and criticisms of governmental agencies when managers incorporate citizens’ perceptions. may lessen, improving the plight of the A powerful management tool results when public beleaguered bureaucrat. managers combine or compare traditional output measures with outcome measures that reflect citizens’ If we were to revisit the seven-step performance perceptions.Meaningful measures that the average measurement system developed by the National citizen can understand provide citizens with the Center for Public Productivity, but this time opportunity to assess and improve government, incorporate citizen involvement in each of the and influence how government services can be made steps, the process would look something like this more responsive to community needs and priorities. (italics indicate changes): PUBLIC PERFORMANCE EVALUATION AND IMPROV E M E N T 59 1 . IDENTIFY THE PROGRAMS TO BE MEASURED 4 . EFFICIENCY MEASURES To start with, programs to be measured must be Set standards for performance and outcomes (targets clearly defined by citizens. Determining what for accomplishment). programs should be measured is the first step in identifying community priorities. In some Under this step, citizens and public managers should communities citizens will identify education and specify the conditions under which program goals recreation as essential programs. In other communities and objectives are met. Citizens and managers need it might be public safety and public transportation to determine what service effectiveness and quality that top the list of citizen priorities. Choosing what means for a particular program and explicitly state programs to measure is a matter of judgement and how they are going to determine whether the stated citizens should have an equal voice in determining terms of effectiveness and quality have been met. what programs are to be measured. Generally, performance measurement systems work best 5 . MONITOR RESULTS when they concentrate on collecting limited, but essential information, about basic programs that Each accomplishment target should be monitored need the most managerial oversight and where on a continuous basis by the management and staff accountability to the citizens is most important. delivering the service and the citizens receiving it. Monitoring provides the citizens and manager with the results needed to decide whether or not 2 . STATE THE PURPOSE AND IDENTIFY THE DESIRED OUTCOMES the performance targets have been met. Systematic and periodic monitoring gives the mana ger and Typically, a government department or agency, in citizens an opportunity to keep tabs on the collaboration with the citizens they serve, initiates operation of the program, and take corrective a strategic planning process to clarify its mission, action when necessary. goals, and objectives. Through this process, the agency and clients/citizens identify the outcomes,or results,they want to achieve through agency programs. 6 . PERFORMANCE REPORTING Citizens and managers can only measure the per- formance of a program if its purpose is clearly stated. A good performance measurement system reports program results on a regular basis. Reports are more than an internal management tool and are 3 . SELECT MEASURES OR INDICA T O R S made public.They are shared with elected officials, citizens, the media and other government watchdogs. A good system uses a few selected indicators to The report focuses on what was accomplished and measure outcomes and performance.Most government what it cost the public. Reports should be brief, programs that have established performance convey information graphically, and include minimal measurement systems incorporate the indicators explanatory information. Information should be described in the previous section: input, capacity, presented in such a way as to facilitate comparisons output, outcome, efficiency and productivity. over time, comparisons with similar organizations, comparison with best programs nationwide and Citizens and managers should determine the most against pre-set targets. Performance data should be appropriate measures for each program, rather presented in a way that is meaningful to citizens so than rely exclusively on the data collected by they can understand what is happening in their individual programs. Good program measurement neighborhood as well as the community as a whole. systems develop citizen surveys to assess citizen satisfaction and perceptions. For example, in a sanitation department the input measures might 7 . USE OUTCOME AND PERFORMANCE INFORMATION be the amount of labor hours; the operating budget; ( A N A LYSIS AND A C T I O N ) the number of vehicles on the road. Output measures could include tons of refuse collected; number of Information from an effective performance households served; number of missed collections. measurement system is regularly used in program Outcome measures might include the percentage planning to reevaluate goals and objectives and to of citizens who are satisfied with refuse collection; adjust priorities. Another use is in managing for the percentage of citizens who rate the cleanliness results, where outcome information is used to of their street as above average. promote continuous improvement in program 60 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing operations and results. A well-developed performance 5. Epstein, Paul D. Using Performance Measurement in Local Government: A Guide to Improving Decisions,Performance and measurement system will enable citizens and Accountability. National Civic League Press,1988. managers to spot weaknesses and challenges to 6. Fischer, R. J. (1994). “An Overview of Performance program delivery, as well as program strengths and Measurement.” Public Management 76: pp. S2-S8. opportunities for improvement. 7. Glover, Mark. (1992). A Practical Guide for Measuring Program Efficiency and Effectiveness in Local Government, Tampa, FL: Innovation Group Publication. 8. Government Accounting Standards Board, Service Effort and Accomplishment Measures. 1997. Norwalk, CT. FINDING A S S I S T A N C E 8a. Government Performance Results Act of 1993. Washington, D.C.: Government Printing Office. 1993. The field of public sector performance evaluation 9. Halachmi,Arie, and Marc Holzer, eds. “Productivity Measurement is changing rapidly. Fortunately, state-of-the-art and Management.” (Public Productivity Review, 44) San Francisco, knowledge and best practices are now more readily CA: Jossey-Bass,1987. available. The World Wide Web offers access to 10. Harvey, J. (1987). “Measuring Productivity in Professional dozens and dozens of helpful sites. Of course, all Services.” Public Productivity Review (44): pp. 29-38. such information must be adapted to specific 11. Hatry, H. P., Blair, L., Fisk, D., Greiner, J. M., Hall, J. R. Jr, & Schaenman, P. S. (1992). How Effective Are Your Community national and local needs. Services? The Urban Institute and ICMA: Washington,DC. 12. Hatry, H. P., Fountain, J. R., Jr., Sullivan, J. M., & Kremer, L. At the National Center for Public Productivity (1990). Service Efforts and Accomplishments Reporting: Its at Rutgers University we maintain web pages Time Has Come. Governmental Accounting Standards Board devoted to the improvement of public perform- (GASB): Library of Congress Catalog Card Number: 90-80879. ance, as well as links to best practices worldwide. 13. Hatry, Harry P., et al. Efficiency Measurement for Local Government Services. Washington,DC: The Urban Institute,1979. Those pages and links may be accessed at: 14. Hatry, Harry P., et. al. How Effective Are Your Community http://www.newark.rutgers.edu/~ncpp Services? Washington,DC: The Urban Institute,1977. 15. Hatry, Harry P., et. al. Measuring the Effectiveness of Basic Municipal Services. Washington,DC: The Urban Institute,1974. 16. Hatry, P. & Fisk, D. (1992). “Measuring Productivity in the Public CONCLUSION Sector”. In M. Holzer (ed.). Public Productivity Handbook. Marcel Dekker: NY, (pp.139-160). 17. Holzer, M. and Halachmi,A. (1996). “Measurement as A Means While it may be costly and time-consuming of Accountability.” International Journal of Public Administration, to include citizens or their representatives in 19 (11&12),(pp. 1921-1944). the measurement of government performance, 18. Holzer, Marc, (ed.). Public Productivity Handbook. Marcel ultimately the performance measurement system Dekker, 1992. developed will be more useful and meaningful. 18a.Holzer, Marc. “Focus: Productivity Improvement in New York The data collected will have an impact on policy State—the Science and Art of Capacity Building,” in Managing New York State, Number 1,1988. Albany, New York: Governor’s and program administration. The performance Office of Employee Relations. The ten steps contained in this measurement system, rather than focusing on article are based, in part, on Holzer, et.al., Managing for managerial accomplishments and administrative Productivity, including previous work by Constance Zalk. achievements,will address quality of life issues and 19. Holzer, Marc and Kathe Callahan, Government at Work, Thousand Oaks,CA: Sage Publications,1998. community goals and aspirations. Governments will be measuring to improve services, to make 20. Holzer, Marc and Vatche Gabrielian. “Performance Measurement and Improvement.” Newark, N.J.: National Center for Public government more responsive to the needs of the Productivity, 1998. citizens they ser ve. 21. Kymlicka, W. and Norman, W. (1995). “Return of the Citizen: A Survey of Recent Work on Citizenship Theory,” In Beiner, R. (ed) Theorizing Citizenship. Albany,NY: SUNY Press. REFERENCES 22. Managers Guide for Improving Productivity. Washington, D.C.: Office of Intergovernmental Personnel Programs, United States 1. Ammons, David N. Municipal Benchmarks: Assessing Local per- Office of Personnel Mangement,1980. formance and Establishing Community Standards. Sage,1996. 23. McGowan,R. P.,& Poister, T. H. (1985). “Impact of Productivity 2. Colwell, W. L., & Koletar, J. W. (1984). “Performance Measurement Measurement Systems on Municipal Performance,” Policy for Criminal Justice: The Federal Bureau of Investigation (FBI) Studies Review, Vol. 4,No. 3: pp. 532-540. Experience” Public Productivity Review VIII (3): pp. 207-224. 24. Milakovich, Michael E. “Total Quality Management for Public 3. Conference Proceedings of the 1993 Managing for Results: Service Productivity Improvement.” in Marc Holzer, ed., Public Performance Measures in Government. Austin,Texas, October 27- 29. Productivity Handbook, New York: Marcel Dekker, 1991. 4. Epstein, Paul D. (1992). “Measuring the Performance of Public 25. Mitchell, J. (1997). “Representation in Government Committees.” Service,” in M. Holzer (ed.), Public Productivity Handbook. Public Administration Review. March/April, Vol. 57, No. 2, Marcel Dekker: NY, (pp.161-194). p. 160-167. PUBLIC PERFORMANCE EVALUATION AND IMPROV E M E N T 61 25a. National Performance Review. Washington, D.C.: Government 30. Romzek, Barbara S. and Dubnick, Melvin J. “Accountability.” Printing Office,1997. International Encyclopedia of Public Policy and Administration. Jay Shafritz, Editor-in-Chief. New York: Westview Press, 1998. 26. Osborne, David and Ted Gaebler, 1992. Reinventing Government. pp. 6-11. Reading, MA: Addison-Wesley. 31. Schacter, H. L. (1997). Reinventing Government or Reinventing 27. Osborne and Plastrik, 1992. Banishing Bureaucracy. Reading, Ourselves: The Role of Citizen Owners In Making a Better MA: Addison Wesley. Government. Albany, NY: SUNY Press. 28. Performance Measurement: A Guide for Local Elected Officials. 32. Stivers, C. (1990). “The Public Agency as a Polis: Active Washington, D.C.: The Urban Institute,1980. Citizenship in the Administrative State,” Administration and 29. Poister, Theodore H., James C. McDavid and Anne Hoaglun Society, 22 (May),86-105. Magoun. Applied Program Evaluation in Local Government. 33. Thomas,J.C. (1995). Public Participation in Public Decisions. Lexington,MA: Lexington Books,1979. San Francisco: Jossey-Bass. 62 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing PART III A)Monitoring Evaluation in a Results-Based Management Approach B)Evaluation Initiatives in Science and Technology E V A L UATION C A PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing EVALUATION PARTNERSHIPS 10 Niels Dabelstein, Head of Evaluation Secretariat DANIDA,OECD/DAC WP Donor Government financed aid is expenditure of taxpayers money outside national borders.In addition, aid is often spent through partners or co-operating governments over which the donor government has no formal control. This is a feature distinguishing aid evaluation from evaluation of most other publicly funded programmes. For that reason alone, the accountability function may ver y well be the most important for aid evaluation. Not the least in periods of aid fatigue and pressure on government budgets, continued public support for aid expenditure is contingent upon aid agencies’ accountability and their ability to document aid results. Generally, aid evaluation performs several functions: it provides lessons for managers and staff of ongoing aid programmes thereby leading to improvements in implementation; it establishes information and documents experience that can be used in planning and designing future aid programmes;it is used by managers, strategy- and policy makers for revising existing — and for devising new — aid strategies and policies; and finally aid evaluation delivers information on efficiency and effectiveness of aid programmes, and thus ensures accountability towards politicians and the public. While these functions apply to both donors and partners, the scope differs substantially. Donors are primarily interested in evaluation the effectiveness, efficiency and impact of “their” investment. But even the best evaluations face difficulties when it comes to establishing causal linkages between the individual donor’s efforts and developmental changes. The individual donor’s evaluations, be they of projects, programmes, or sector support, run the risk of ascribing positive de velopment to a limited intervention, when the reality is that development is the result of the synergetic effect of all interventions by the developing countries themselves as well as the support provided by donors, enhanced or hampered by external factors. Our partners – the developing countries – have a wider scope: they would focus on overall performance of projects,programmes, and sectors, and even the entire economy. Evaluation is thus an aid to resource allocation and prioritisation, to sound management, and to accountability towards politicians and the public. In short you could say that donors evaluate development aid while developing countries evaluate development. Evaluation institutions exist in many developing countries, but most have little impact on policy and management decisions due to a number of barriers: Poor demand; lack of a culture of accountability (often related to ethics or corruption); absence of evaluation, accounting, or auditing skills; and lack of feedback mechanisms into decision making processes.The strongest barrier appears to be the lack of demand: Credible evaluation is a function of good governance i.e. demand for accountability more than of institutionalisation of evaluation and/or professional capacity. E V A L UATION PA R T N E R S H I P S 65 How, then, can donors contribute to enhancing focus, setting new demands for host-country evaluation capacity in our partner countries. evaluation institutions. Support to building evaluation capacity in developing s Regional, sectoral and programme/project countries has been on the agenda for several years, evaluations become more useful if they are and the DAC Working Party on Aid Evaluation based on a co-ordinated approach linked to a has promoted the concept actively, through several national evaluation system, particularly with regional seminars organised jointly with the respect to methodologies and data needs. regional development banks (Africa in 1990, Asia in 1992, Latin America in 1993, and in Africa These findings generally apply to both donor and again in 1998) and through the members’ direct recipient agencies. assistance, progress towards establishing credible and capable evaluation functions in developing While the donors can promote and facilitate countries has been slow. In recent years the multi- evaluation capacity building through collective lateral institutions have been particular ly active, as activities, the main support has been, and will witnessed by this seminar. continue to be, channelled through multilateral and bilateral donor agencies. The Study identified In 1996 the Working Party conducted a survey of several areas where donor agencies could strengten donor support for and experiences from evaluation and mainstream ECB work within their agencies capacity building. The Key findings of the study, and in collaboration with development partners: which I believe are still valid were: s Promote an agency ECB support policy or s Sustainable evaluation institutions need support strategy, particularly in view of new aid forms at the highest policy and management levels being introduced,including programme assistance and should be able to demonstrate its use to for institution and capacity building as part of these levels. The design of evaluation systems good governance initiatives at national and must also take into account the specific sectoral levels. government and administrative culture in the country/organisation. s Advocate and stimulate the evaluation issue in country dialogues and sector programme s Political advocacy and senior management assistance. demand should be preconditions for ECB supply activities and must be linked to the issue s Provide technical advice to operational units of governance.A long-term strategy is needed responsible for ECB support activities and for effective interventions in both cases. advise on training facilities and materials on s Sustainable and effective evaluation systems evaluation issues. must have a legal foundation or a firm statutory organisational regulation. s Support the establishment of twinning arrangements between other domestic evaluation s An evaluation unit’s independence from line man- institutions and host country institutions. agement is important, as is the security of career possibilities for evaluation staff and managers. s Assist in securing consistent evaluation methodologies and terminology in the ECB s The scopes of national-level performance support activities of the agency. evaluation and performance auditing systems are moving closer to each other, although the s Co-ordinate their evaluation programmes former is likely to be more closely integrated with host countries and other donors in in the planning process, while the latter tend order to optimise the use of resources and the to focus more on accountability to the policy- constrained capacity of recipient countries’ making level. The choice of approach may, evaluation systems. however, depend on other factors, such as political commitment, the legal framework s Arrange joint-evaluations with a genuine and institutional capabilities. participatory approach, where the needs of both parties are incorporated from the start s Development policy and aid tend to shift and where the capacity building element is from a project/programme to sector/policy taken into account specifical ly. 66 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing I will focus on the very last of these possibilities: DAC Member countries also use joint evaluations Joint-evaluation – or in current lingo: Partnership to improve partner countries’ evaluation capacity. in evaluation. The deeper partner country representatives are involved in evaluation process, the more they are In line with the emphasis on development exposed to new evaluation methods and approaches, partnership, local ownership, and good governance, and the learning process is smoother as a result. the donors should use the joint-evaluation as a vehicle to assist in developing an “evaluation The degree of partnership between partner and culture”.Thus the value of evaluation as a management donor countries will vary according to the objectives tool as well as an instrument for shared lesson and uses of the evaluation. In some cases, DAC learning and accountability could be demonstrated. Members have involved partner country represen- To succeed in this, donors need to co-ordinate tatives in the evaluation process by including them evaluations and ultimately let evaluations be co- in the evaluation team to define terms of reference ordinated by the recipient countries. (i.e. objectives, questions, scope, methods, uses, etc.). On other occasions, they took part in the evaluation process by assuming different roles such Although joint evaluations do take place they are as that of key informant, a member of a focus still infrequent, and have primarily concerned group, a researcher, an interviewer, a country donors’jointly financed programmes,or been eval- regional expert, or simply by gathering data. uations of multilateral agencies by groups of donors. Common to most of these evaluations is In most cases, partner country representatives have that the developing countries have played a minor - been involved in joint evaluations to gather and if any role in their planning and execution. Rarely analyse data. Generally the International Finance are the recipient countries involved until an evaluation Institutions (IFIs) have not included partner scheduled by the donor is initiated and mostly the countries in the planning and design phases of the recipient government is involved in providing evaluation cycle, while bilateral donors have done information to the donor but not in the analyses so more often. Some [For example, Denmark, the and final assessment of performance. Evaluation Netherlands, New Zealand, Switzerland and the programmes are prepared in response to agency European Commission] have involved partner needs for lesson learning and for accountability, representatives more fully in the evaluation process and is geared to the planning and programming by including them in the following phases: planning cycle of the agency, not to the planning cycles of and designing; gathering and analysing data; partner countries. identifying evaluation findings, conclusions and recommendations. Despite these difficulties the donor community considers it crucial to more fully engage with developing countries in a partnership strategy for evaluation.By jointly analysing opportunities both donor and partner countries enhance their sense of CONSTRAINTS ownership and feel responsible for results. Then, A1998 review of the DAC Principles for the evaluations of development assistance programmes Evaluation of Development Assistance already can be effectively carried out with the participation identified some of the difficulties encountered of donor and partner countries, as both would when implementing partnership in evaluation. share the interest in accountability for results,lessons These included recipients’unwillingness,or lack of learned and improved programme effectiveness. interest to participate in joint evaluations; time constraints and higher costs; communication Evaluations carried out with the participation of problems and delays; increased complexities and, partner country representatives are more effective occasionally, political obstacles. The fact that partner as they generally lead to a fuller use of the findings. countries have inadequate knowledge of donor In fact, joint-evaluations can generate multiple countries’ evaluation policies may also be one of the benefit, including: i) a better understanding of the possible impediments to partnership in evaluation. results;ii) greater acceptance or sense of ownership of the results; iii) enhanced credibility and validity Lack of partner government commitment to the of the findings; iv) increased sense of responsibly process;discrepancies between the donor and partner and incentives to follow the recommendations country’s objectives or reasons for carrying out the formulated by the evaluation;v) possibly improved evaluation; and insufficient evaluation capacity partner country’s evaluation capacit y. within the partner country have been identified as E V A L UATION PA R T N E R S H I P S 67 the most important impediments to effective Donor and partner countries’ governments should partnership in evaluation. be committed at senior level to a joint evaluation process. At the same time, Parliament,stakeholders, On the demand side, although many partner countries beneficiaries and civil society in both donor and have created an evaluation unit attached to the partner countries should be kept informed about planning or finance ministry, they are not yet the results of joint evaluations as so to enhance the strongly committed themselves to evaluation sense of ownership not only of the evaluation findings activities. Governments are not yet fully results but also of the development programme. oriented. The need for accountability, participation, transparency, rule of law has only recently been As participatory processes usually need longer considered necessary for improving governance time frame, it is crucial to plan sufficient time to practices. Moreover, the demand for evaluation carry out a joint-evaluation. It is necessary to plan comes more often from the donors community during the design phase, the number of evaluators (which has limited influence), than from the partner and financial resources needed, bearing in mind countries’ Parliament or taxpayers. that joint evaluations generally involve more people and call for more co-ordination. On the supply side, differences in donors’ evaluation and management requirements have hindered the When evaluation capacity is judged to be insufficient, development of evaluation capacity of partner countries. it is suggested to carry out some actions to support its development in partner countries simultaneously Public sector reforms can open new opportunities to the evaluation. Donors’ agencies dispose of to effectively stimulate demand and increase supply different tools to promote the development of of evaluation and performance management systems. evaluation capacity, notably “learning by doing” SIDA has suggested that, in order to develop evaluation training (e.g. their involvement in the definition of capacity effectively, development agencies could questions, data collection, methods); or technical also provide financial and technical resources to assistance to personnel, as part of monitoring conduct programme evaluation separately from programme/project, or other kinds of training. donors’projects.This could also create demand for evaluation independently from the donor needs. In sum, it is rare that donor and partner counties undertake joint evaluations in full partnership. The most important factors impeding effective joint- evaluations are lack of governments’ commitment THE FUTURE to partnership in evaluation.Different conceptions of the purpose of evaluations and limited time and It is important to keep in mind that to have resources availability. successful partnership in evaluation partner and donor countries should, at least partially, share I have attempted to point out the advantages of the same objectives. This normally calls for joint joint-evaluations, to identify the main obstacles or elaboration of the goals, and thus partner repre- constraints,and indicated a few ways in which the sentatives’ participation during the programming partners in development may enhance evaluation phases. For these reasons, evaluations should shift capacity through joint-evaluations. their primary objective away from the control and accountability functions, to become more a learning I hope that during the next 45 minutes we can instrument to improve development assistance identify practical ways of moving joint-evaluation performance and to learn lessons. a few steps further. 68 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing EVALUATION TOOLS AND METHODOLOGIES: Case Studies 11 Jacques Toulemonde, Director, Centre for European Evaluation Exper tise THE INTERNATIONAL MODEL OF PROJECT EVALUA T I O N Donor institutions have disseminated the practice of evaluation worldwide through a systematic review of the projects that they fund. This has given birth to a “model of project evaluation” that is typified hereafter. The evaluation follows the logical framework of the project,which means that it looks at activities, outputs, outcomes and the problem to be solved. All evaluation criteria are systematically applied: rationale, relevance, effectiveness,efficiency, and sustainability. Unintended effects are also valued. Donor institutions play the leading role in the evaluation system. They set up the evaluation agenda and the rules of the game. This does not prevent participatory methods to become more and more familiar. Participation typically extends to national authorities responsible for the project,to the project management and to beneficiary groups on the project site. Project evaluation and monitoring go hand in hand. Project managers periodically report on the progress of implementation, which generates monitoring data. As far as these data are properly filed, project evaluators can rely upon a sound basis of field data. Evaluation therefore consists of a quick external review. Additional collection of field data is typically limited to a few interviews and a site visit. The evaluation system follows the project cycle. It may include three different exercises: ex ante, on-going and expost. These three types of evaluation respectively correspond to the phases of project design, implementation and termination. The description above is a deliberate caricature. In fact, donor institutions have a more complex practice including joint evaluation with national authorities, country evaluations, thematic evaluations, coordination of donors’ evaluations, etc. Is it useful to typify the model of project evaluation in such a simplistic way? For the sake of this presentation, the answer is yes since my purpose is to set up a benchmark that everybody is familiar with. In order to make the above model workable the project under evaluation must fulfil several conditions. First, it must be reasonably simple. This means that it delivers a single type of output to one target group and that it aims at solving one major problem. Second, the project is supposed to involve one single line of stakeholders,namely the line, which connects headquarters of the donor institution to the recipients on the project site. Third, the project is under the responsibility of an identified management team that produces periodic reports and monitoring data. Finally, the project runs from A to Z with a beginning and an end. Its objectives should be made explicit from the very beginning. E V A L UATION TOOLS AND METHODOLOGIES: Case Studies 69 A EUROPEAN MODEL familiar with evaluation ten years ago. There was therefore a need to support the development of an OF POLICY EVALUA T I O N evaluation culture across Europe. This was the rationale for launching the MEANS program in This paper builds upon a very different evaluation the early 90s. model, which applies to a large policy of the European Union. Through a contrasted view of MEANS is a French acronym for “Methods for these two models, I will describe some problems Evaluating Actions of a Structural Nature” that arise when practice shifts from project evaluation (“Structural” because the policy aims at improving to policy evaluation. I will also propose methodological social and economic structures). This program, in recommendations for solving these problems. The which I have been closely involved, allowed for Cohesion Policy is a major stake for the European several years of research to be undertaken on Union. It principally aims at developing those evaluation practice in the field of social and regions that lag behind in terms of social and economic development. The main activities of the economic wealth. Many other goals are also pursued MEANS program were bottom up. It consisted of like reducing unemployment, promoting an envi- identifying problems and of designing solutions, ronmentally sustainable development, etc. This not of giving lessons. policy is the second biggest in terms of cost for the European taxpayer. It is implemented through; The outputs of the MEANS program comprise a hundreds of programs are managed and funded in body of methodological experiences, ideas and partnership with national and regional authorities. innovations which were presented in seven 50-60- page handbooks. The programme also published In this context, the term “program” applies to: (1) methodological reports and quarterly newsletters. a coordinated set of different types of projects (e.g. It organised training seminars and three European training courses, advice to enterprises, subsidies to conferences (the most recent, in Seville, was employers,etc.), of which (2) the objectives are clearly attended by just under 600 practitioners). An stated (e.g. placement of the long-term unemployed) effort was made to take full advantage of existing without implementation being defined entirely in experience and recent innovations, particularly in advance and, finally (3) which has a limited budget the following areas: measurement of employment and duration (generally five-years). effects, development of systems of indicators and evaluation of macro-economic effects. Through Within the framework of the European Cohesion the evaluations initiated at the European level and Policy, programs are particularly complex. They the work that has gone into building on these expe- typically encompass dozens of different types of riences, sufficient knowledge has now been acquired activities. Their global objective is like an umbrella to produce a comprehensive publication accessible to for several operational objectives and priorities. In a wide public. The result is a collection covering all most European regions, programs follow each aspects of the evaluation of socio-economic devel- other with incremental changes every five years.In opment policies (European Commission, 1999). this case, the concept of program cycle with a beginning and an end is somewhat artificial. For At the end of my long tour into this large European these reasons,I tend to consider that what is under policy, I conclude that it requires an evaluation evaluation has many features of a policy and I will model that is very different from that of project use this term hereafter in order to be better evaluation. Expressing my conclusion in a provocative understood by non European readers. way (that would certainly not be endorsed by European authorities),I would say that in the case The evaluation system of the European Cohesion of the European Cohesion Policy: (1) systematic Policy was created in 1989 and it is now ten years evaluation along the logframe technique is not old.The system is one of the most ambitious and workable, (2) systematic evaluation on every sites is comprehensive in the world. It makes evaluation not workable, (3) systematic relying upon monitoring compulsory in every European region at ex ante, data is not workable, (4) systematic following of interim and ex post stages. It puts much emphasis the policy cycle is not workable. on independent external evaluators. It establishes an obligation to evaluate that applies to the Union I develop these points in the next sections,together itself, but also to the fifteen member states and to with a series of recommendations that build on the hundreds of regional authorities. The least to say European practice and that help making policy is that all these authorities were not that much evaluation workable. 70 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing SCREEN THE POLICY WITH A LOGICAL SELECT KEY ISSUES FOR DIAGRAM OF EXPECTED IMPA C T S IN-DEPTH EVALUA T I O N One of the first steps of any evaluation is to understand Evaluation experience of the European Cohesion the cause-and-effects assumptions that connect Policy shows a wide variety of ways of addressing the activities and their expected impacts. This is evaluative questions. Fairly often, the evaluation easily done by using the logical framework if the team is expected to treat ten or more questions activities and expected impacts are in small numbers. within a few months. Other evaluations, by contrast, This is not so easy in the case of the European devote a whole year to the in-depth analysis of a Cohesion Policy. One single evaluation is often single question. supposed to assess thirty or more different kinds of activities and ten or more different kinds of The first option is in line with the model of project evaluation in which the donor institution tends impacts. Such a complex setting is not surprising to ask every possible question. In this case the for those who evaluate national policies. How many evaluator is asked a list of questions like: did the logframes should be developed for describing such various activities produce their intended effects a policy? Certainly too much for this technique to (effectiveness)? Were the effects obtained at be workable. reasonable cost (efficiency)? Was it worth trying to obtain this or that effect (relevance)? And what In order to tackle this problem,European authorities about unintended effects? Since the European have first recommended that evaluators and program Cohesion Policy typ i ca lly encompasses many designers establish a hierarchy of objectives as activities and many effects, the number of questions represented by an objective tree. This technique is multiplied in such a proportion that evaluators consists of a diagram in which the most global cannot convincingly answer all of them. They objective is placed on the far left, and the most often admit that “available data do not allow to detailed objectives on the far right. Each objective assess the impacts”,or, even worse, they answer the can thus be broken down into several sub-objectives questions without any evidence basis. which, in turn,are broken down into objectives on a more and more specific level. In practice, neither On the contrary, the European practice has shown the objectives nor the relations between them can that sound evaluation works tends to address a be articulated so simply. In almost all cases, a limited number of questions. This allows evaluators given activity may contribute to achieving several to derive their conclusions from an in-depth analysis, objectives of general interest. This is contradictory including collection of field data when necessary. with the structure of a “tree” and this calls for something resembling more a “network” or a “web” In the following example, the evaluation steering of objectives and activities. group considered several questions before deciding to analyze three of them in more depth. The story In order to address this problem, an alternative started in 1997 when public authorities in the French region of Franche-Comté had to evaluate technique has been proposed in the MEANS an industrial regeneration program partly funded collection. It is called: “logical diagram of expected by the European Union. The evaluation steering impacts”. The diagram sketches the main causal group devoted its first two meetings to the identi- links that connect the activities, situated on the fication and selection of evaluative questions. left, and their successive results, outcomes or Seven fields of questioning were identified. These impacts, situated on the right. Contrary to an were then discussed in order to formulate, for each objective tree which draws a simplified structure of of them, a realistic and useful evaluative question. objectives,the logical diagram of impacts attempts to mapping the causal links between the activities These questions drafted by the steering group under evaluation and their ultimate objectives. were as follows: (1) Which categories of projects have the best impact on the attractiveness of the In the European case, this alternative technique territory? (2) Has support to enterprise helped to has provided a workable means for clearly describing create or maintain jobs that are firmly rooted in the the logic of a complex policy. In addition, the area? (3) Have services for enterprises contributed diagram provides a means to quickly screen the sufficiently to the diversification of economic whole policy before deciding to scope some parts activity? (4) Does training serve to raise qualifications in more depth. or rather to make them more relevant? Which of E V A L UATION TOOLS AND METHODOLOGIES: Case Studies 71 these two expected effects is the most necessary? different activities in terms of contributing directly (5) Is the probable impact of higher education or indirectly towards a given objective. and research projects sufficient? (6) What is the effectiveness of the different categories of projects in Such analyses deserve to be selected when a given terms of creation and maintenance of employment? objective is considered as strategic (assuming that (7) Which categories of projects guarantee or improve the corresponding issue is most likely to gain from equal opportunities between men and women? an informed debate), or when an objective is controversial (assuming that the corresponding An in-depth study of all seven questions would issue is most likely to gain from the clarification have been too costly. It was therefore decided to of judgement criteria). The notion of thematic limit the number of questions to study in depth to evaluation calls to mind that of in-depth evaluation. three. Certain questions were eliminated so that In the framework of the European Cohesion there would not be duplication with other evaluations Policy, some thematic evaluations examine how and studies underway in the region. Finally, questions the same impact (e.g. the competitiveness of (1), (4) and (5) were judged priorities. SMEs) is obtained by various activities (type 3 of in-depth analysis) or how interventions financed The first type of in-depth analysis applies to a in a specific domain such as Research & cluster of similar activities (as for questions 2,3,4,5 Development produce several kinds of impact above). In this case, evaluators can easily implement (type 1 of in-depth analysis). surveys of beneficiary groups. Primary data obtained in this way provide indications relative to In the framework of the Cohesion Policy, thematic gross effects. The analysis of causality may also be evaluations have proven to be the most useful ones carried out, confounding factors may be identified, for policy-makers at European level. This confirms deadweight may be estimated, evaluation design that policy evaluation should focus on a few strategic may include comparison with non-beneficiary issues instead of asking all possible questions. groups. This type of analysis focuses on the success or failure of a given activity through its many different impacts. Such an in-depth analysis deserves to be undertaken I N VO LVE ALL RELEVANT PARTNERS for expensive activities in terms of budgetary IN AN EVALUATION STEERING GROUP appropriations (assuming the stakes involved in decision-making increase in proportion to the In those European regions where the culture of budget). It is also relevant for new or innovative public managers was old fashioned, the evaluation activities (considering that they are experiments, practice has first started as a bureaucratic exercise. which need to be validated by an evaluation). The A typical behavior was to hastily contract out second type of in-depth analysis focuses on a evaluation works at the lowest possible cost particular group of recipients, such as the long- (sometimes 0.01% of the expenditures under term unemployed,creators of new enterprise, small evaluation) and with the lowest possible ownership. farmers or women (as for question 7 above). The A current practice was to use standard terms of evaluation is performed from the point of view of reference, scarcely orientated towards regional these recipients. It analyzes their needs, it uses particularities, badly suited to the policy-makers their judgement criteria and it tries to understand expectations and hardly motivating for the managers. their interaction with the policy. In this situation, As long as the writing up of the terms of reference the evaluation produces con cl u s i ons on the has been considered an office job, it has been comparative impact of different activities that difficult to make them properly adapted to the directly address the group under study. It can also needs of evaluation users. show how these different activities reinforce or contradict one another with regards to the group This poor state of practice has quickly changed under study (synergy effects). The third type of and evaluation becomes a managerial exercise. in-depth analysis concerns an expected impact or a More time is invested in drafting the terms of category of impacts (as for questions 1 and 6 reference of exercises that cost up to 0.8% of the above).The question then applies to every activity expenditures under evaluation. In the best cases, under evaluation. It focuses on their respective the commissioner of the evaluation officially contribution to the production of the desired mandates a steering group from the outset and for impact. In this situation,the analysis concludes on the entire duration of the process. This group is no the comparative strengths or weaknesses of the longer a close shop that gathers funding institutions, 72 E V A L UATION C A PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing managers and operators. It is increasingly open A standard procedure for policy evaluation consists to other partners who hold various stakes (e.g. of performing specific evaluations of its components, somebody speaking with the voice of “environment” and then of synthesizing the conclusions at the in a policy that is mainly geared at economic policy level. Evaluation by synthesis has faced development). The steering group is then given serious methodological shortcomings when it has an active role in the preparation of the terms of applied to heterogeneous activities. Since the evaluation reference, which implies the organization of one or of each component uses its own indicators,analyses two or even three meetings before the external distinct impacts, and applies different criteria, the evaluator is chosen. This makes it possible to draft final synthesis can only be the juxtaposition of precise evaluative questions, to check that the partial evaluations. The very notion of synthesis essential (or embarrassing) questions have not therefore loses its meaning. An evaluation report been forgotten, to reject questions which are more can be produced in which a long chapter is devoted a matter of auditing or control, to select priority to each activity, but in which the overall conclusion questions, and to specify the judgement criteria is nothing more than a few lines with very poor or and norms. An extreme but salutary choice may be tenuous contents. the postponement or cancellation of the evaluation, when the steering group foresees that its results In these circumstances it is impossible to make will not be used. relevant recommendations for policy-makers. For example, it is not possible to justify recommendations This preliminary stage does not involve additional for the reallocation of budgets between activities workload. It is rather an alternative way of organizing because there is no argument showing that one is the evaluation process.In fact,the first meetings of more effective or successful than another. To produce the steering group provide an opportunity to structure genuine synthetic conclusions, the evaluators the evaluation process in a clear, fair and useful should apply common criteria at policy level. This manner, a task that external evaluators often recommendation, which is simple in theory, has undertake with great difficulties. proven to be hardly workable in the European model of policy evaluation. In most cases, evaluators tend to use the criterion of budgetary consumption, which has even been termed in Italy as the “capability DISTINGUISH POLICY EVALUATION to spend public funds”. The exact opposite of an evaluation culture! FROM PROJECT EVALUA T I O N In the framework of the European Cohesion Much attention is presently paid to measuring Policy, evaluation teams tend to view complex performance. Performance is often translated into policies as a series of components, and therefore a ratio of actual vs. expected outputs, which is also analyze each of these components separately. termed “completion rate”. The fact that an activity Components (or activities) tend to be evaluated in rapidly achieves its objectives may be an argument terms of their own success criteria. For example an in favor of its effectiveness or success, but this activity which consists of improving roads is argument is necessarily weak. Some activities have evaluated in terms of the total number of hours very cautious objectives while others have very saved annually by vehicles using the road. The ambitious ones. When such activities are compared, time saved can be globally related to the public the conclusions are distorted and the synthesis at investment so that comparisons can be made with policy level is impossible. road improvement activities in other similar regions. Finally, it has become clear that policy evaluations The indicator mentioned above is specific to a must address questions that are of policy interest. particular activity (improving roads), just as the Such questions are not related to the outputs of approach adopted is that of an evaluation specifically specific activities, or to their immediate outcomes. for the road transport sector. Specific evaluations Evaluative questions of policy interest are closely have been performed for all sorts of activities: related to the ultimate policy goals, i.e. macro- transport infrastructures,training of the unemployed, economic or macro-social development. These rehabilitation of run-down neighborhoods,support questions are difficult ones and they require long to innovation networks, etc. Volume 2 in the and careful evaluation studies before being credibly MEANS Collection presents numerous examples of answered. It has proven impossible to deal activities, with a description of their results and specific with such difficult questions for every activity on impacts and suggestions for appropriate indicators. every site. E V A L UATION TOOLS AND METHODOLOGIES: Case Studies 73 Consequently, a good practice is to distinguish two consultancy services provided, and the categories levels of evaluation. Evaluations of specific activities of businesses targeted. Individual interviews were address questions related to immediate outcomes then carried out with 20 operators. These interviews and apply judgement criteria that make sense for were used to describe in greater detail the strategies managers. These evaluations apply separately to developed by the operators and to establish a all sites and activities. At the policy level, evaluations typology of operators. Six operators were then address questions related to ultimate outcomes and selected for deeper case studies. A sample of 300 apply judgement criteria that make sense for policy- beneficiary firms was selected among the “clients” makers. Policy evaluation cannot reasonably apply of these six operators. Another sample of 200 to every activity and every site. On the contrary, non-beneficiary firms was also constituted, in they typically involve case studies and field surveys keeping with the matching pairs technique. in a limited number of sites. A second questionnaire was addressed to the beneficiary firms and to those in the matching group. A statistical comparison between the firms E V A L UATE WITHOUT MONITORING in the two groups provided quantitative indications on the effects of the programme, in terms of DATA WHEN NECESSARY turnover, productivity and investments. In order The task of evaluators is considerably easier when to avoid “biased” statements, the questionnaires they can rely upon a set of sound monitoring data comprised numerous and partially redundant related to inputs and outputs. In the case of questions (triangulated questionnaire).The survey the European Cohesion Policy, much effort has incorporated a search for deadweight effects. been made to develop evaluation together with monitoring. After ten years, it must be said that This example shows that evaluation can (and the state of practice is far from perfect. In most sometimes should) be implemented in the absence sites and for a majority of activities, it can be of monitoring. It shows that public authorities should considered that outputs are properly monitored, strengthen their capacity to undertake both exercises, both quantitatively and qualitatively. but that poor monitoring should not be accepted as an excuse for cancelling policy evaluation. In a near future, this may also be the case for job creation, an impact that is of strategic interest at policy level. Nevertheless most impacts are unlikely to be subject to systematic monitoring within the next decade. Unfortunately, these impacts give rise MERGE EX A N T E , ON-GOING AND to many evaluative questions of policy interest. Here comes the problem of evaluating without EX POST EVALUA T I O N monitoring data. Examples from the European In the evaluation literature, contradictory opinions practice show this is not impossible. are expressed on the timing of evaluation. Some authors consider that evaluation is only a retrospective For instance, this achieved in the case of the British exercise while others also attribute a prospective TEC program (TEC for Training and Enterprise dimension to it. In the context of European Council). This success story of useful evaluation Cohesion Policy, evaluation is clearly defined as an took place in the instance of an absolute lack of activity, which takes place before, during and after monitoring data. The TEC were financed by the British Department of Trade and Industry. They the implementation period. These three phases offered SMEs management consultancy, training, are related to the policy cycle and consist of ex ante and information services. The impact of this evaluation,interim evaluation and ex post evaluation. programme could not be expressed in jobs created, since it was aimed at enhancing productivity Given the uninterrupted sequence of policy cycles, which, in the short term, led to the destruction of the three exercises have been increasingly overlapping jobs. It was therefore necessary to evaluate the as the European evaluation system developed. programme in terms of other criteria such as Evaluations performed at different stages and for productivity, profitability or investments induced. different cycles tend to collide at the same time. None of these impacts had been monitored. The sequence of three evaluation phases in successive cycles creates overlaps that have to be organized as An initial questionnaire was sent to the 75 operators efficiently as possible to avoid any duplication in the Councils. This questionnaire concerned the of work. 74 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing The relative continuity of activities from one period an alternative technique (the Logical Diagram to the next makes it possible to use conclusions of Expected Impacts) has been successfully from the recent past to judge the relevance of the used for describing and quickly screening a new activities proposed. In the case of the complex policy. European Cohesion Policy, ex ante evaluations that prepare the 2000-2006 policy cycle had to be (2) Systematic evaluation on every site is not undertaken by regional authorities in 1998-1999. workable.A model of policy evaluation should By this time, the interim evaluations of the period better involve all relevant partners in a steering 1994-1999 were just finished. Some of the ex post group, require them to select key issues for evaluation for the period 1989-1993 took place at in-depth evaluation and to ask evaluative the same time. questions that clearly are of policy interest. In so far as evaluation must draw conclusions from (3) Systematic relying upon monitoring data is not the experience of preceding activities to improve always workable. Policy evaluators can and must future activities, an interesting solution is to work without monitoring data when necessary. establish an evaluation plan that covers several years and that more or less merges the three phases. (4) Systematic following of the policy cycle is not The idea is to identify the different possible workable. Policy evaluation should be organized evaluations and to establish their date and content through evaluation work-plans that merge ex in relation to the agenda of policy-makers and ante, on-going and ex post exercises. to the needs of other evaluation “customers”at var- ious levels. My personal opinion is that these lessons,learnt in the context of a large European policy, are of general value for other large policies in other continents. Are the corresponding recommendations TRANSFERABLE LESSONS transferable to other policy settings? This is a FROM EUROPEAN PRACTICE matter for discussion. Lessons and recommendations that emerge from the European practice can be summarized as follows: REFERENCES European Commission (1999) Evaluating socio-economic programmes, (1) Systematic evaluation along the logframe MEANS Collection,Luxembourg: Office for Official Publications technique has proven not to be workable, but of the European Communities. E V A L UATION TOOLS AND METHODOLOGIES: Case Studies 75 EVALUATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 12 Rustam Lalkaka, President,Business & Technology Development Strategies 1 . BA C K G R O U N D Science is intended to understand the world while technology is to help change it – hopefully, for the better. From a development standpoint, distinctions between high and low technology are irrelevant. What is important is that the product or service is useful, it is ‘friendly’ to society and the environment, and it contributes to the overall priorities of economic growth with equity, social justice, basic needs and employment. Further, knowledge at the root of science may be value free, but technology is conditioned by context and culture. A SHARED HERITAGE Our knowledge of inner and outer space is changing our understanding of the universe and our place in it. At the same time, we are witnessing the decline of the natural environment and the bio-diversity on which we depend for life itself. While the cold war may have ended, weapons proliferate and struggles continue by the world’s poor and its ethnic minorities for a better existence. What then does the coming millennium hold for human civilization? And what can the industrializing countries in particular now do to enhance their national innovation systems, in order to apply technology to the task of accelerating economic growth with fairness and stability? The historical perspective should be kept in view: For about 2,000 years, from say 300 BC to 1770 AD, countries which are today called ‘developing’ were at the forefront of applied science and many significant innovations moved from east to west. For the next 200 years, following the industrial revolution, countries today called ‘developed’ began to pull ahead, technologically and thereby economically and militarily. And over the last 50 years, the pace and pattern of technical change has altered sharply and the industrial countries have out-paced the rest of the world. Nevertheless, a dozen Newly Industrializing Countries now have the technical infrastructure for major innovation; and for all nations the advanced technologies – adapted, applied,and absorbed – can help improve their lives. This paper looks at the main dimensions of the innovation system and indicates some of the special considerations involved in making evaluations of S & T programs. Two case studies illustrate appropriate techniques. Such evaluation has to become integral to the a country’s governance, as the taxpayer has the right to ensure that the large sums of money for building the S & T establishment are being spent wisely and with maximum impact. E V A L UATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 77 2 . MAIN COMPONENTS OF 4. Technical support services required to enhance quality, raise productivity, lower production NATIONAL INNOVATION SYSTEM costs, and develop an integrated marketing- research-production system, especially the For the purpose of reviewing national Monitoring support services for strengthening technology- and Evaluation capacity, the innovation system can based small enterprises. be considered in terms of six sub-systems: 5. International cooperation through strategic 1. S & T policy comprising the setting of priorities alliances,design and production sharing in the and allocation of resources for basic research, advanced science-based technologies, com- supply chain, linkages to donor agencies and petitiveness in all economic sectors, and for development banks, and technical and financial other national priority concerns. collaboration while safeguarding the intellectual property of the inventor. 2. Innovation strategy, both short and long-term, based on the nation’s competitive advantage, Underlying the above sub-systems is the overarching the management of research and its commercial problem of mobilizing financial resources for S & T utilization, the roles of publicly-funded and (which is much more than R & D), particularly in corporate research, targeting the special needs poor countries where programs for the survival of of large rural and disadvantaged communities, its people must itself be the highest priority. the cost-effective acquisition of technologies, and the ‘deepening’ of technical competencies Each country has to design its own national innovation for adaptation, application, dissemination system based on its endowments and constraints, and, progressively, local innovation. its culture, climate and historical legacy. Table 1 3. Technical human resources including science indicates some key knowledge indicators in select- education starting in primary school and its ed countries.The expenditures on R & D are a proxy progression through technical universities and of the commitment to innovation but must be inter- continuous learning, vocational training, new preted carefully as they do not indicate the pro- disciplines such as the management of ductivity, the prioritization and inter-sectoral allo- technology and technological entrepreneurship cations of research efforts.More relevant today are development. The computing and communication the proportions of high-tech products in total technologies have a crucial role in developing manufacturing exports and the electronic infra- new ways to eradicate illiteracy. structure in a nation. TABLE 1: KNOWLEDGE-RELATED INDICATORS – SELECTED COUNTRIES GNP Military TVs PCs Internet R&D R&D High-tech. Royalty License Patent Patent PPP Expend. per per Hosts Personnel Expend. Exports Receipts Payments Appls Appls filed US$ % GDP 1,000 1,000 per 1,000 per 1 mil. % GNP % mfg US $ mil. US $ mill. filed by by Non- 1997 1995 1997 1997 1997 1985-95 1985-95 exports 1997 1997 Residents Residents 1997 1996 1996 Argentina 10,100 1.7 289 39.2 15.92 671 0.4 15 6 240 n/a n/a Brazil 7,430 1.7 316 26.3 9.88 168 0.6 18 32 539 2,655 29,451 China 3,070 2.3 270 6.0 0.16 350 0.5 21 55 543 11,698 41,106 Hong Kong 24,350 n/a 412 230.8 108.20 98 0.3 29 n/a n/a 41 2,059 Egypt 3,080 5.7 127 7.3 0.33 458 0.5 7 54 365 504 706 India 1,660 2.4 69 2.1 0.11 149 0.8 11 12 150 1,660 6,632 Jordan 3,350 7.7 43 8.7 0.79 106 0.3 26 n/a n/a n/a n/a Korea 13,430 3.4 341 150.7 37.66 2,636 2.8 39 252 2,413 68,446 45,548 Malaysia 7,730 3.0 166 46.1 18.38 87 0.4 67 0 0 n/a n/a Mexico 8,110 1.0 251 37.3 8.75 213 0.4 33 130 501 389 30,305 Poland 6,510 2.3 413 74.4 45.34 1,299 0.7 12 27 175 2,414 24,902 Singapore 29,230 4.7 354 399.5 187.98 2,728 1.1 71 n/a n/a 215 38,403 S. Africa 15,690 2.2 125 41.6 34.02 938 0.7 n/a 73 258 n/a n/a Turkey 6,470 4.0 286 20.7 4.30 261 0.6 9 n/a n/a 367 19,668 Germany 21,170 n/a 570 255.5 140.58 2,843 2.4 26 3168 4694 56,757 98,338 Japan 24,400 1.0 708 202.4 107.05 6,309 2.9 38 7303 9620 340,861 60,390 USA 29,080 3.8 847 406.7 975.94 3,732 2.5 44 33676 9411 111,883 111,536 78 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing Some major considerations in formulating an The evaluations of giant national undertakings effective evaluation for the above sub-systems are such as the Three Gorges Dam in China require outlined in Annex-1.These then form the basis for inter-disciplinary skills due to their enormous developing the needed evaluation capabilities. environmental, social and financial implications for the future. 1 . S & T POLICY 2 . I N N OVATION STRATEGIES At the national level, an appropriate strategy for developing countries starting the process of tech- For the longer term, the nation has to move from nology transformation could be a cascade of first, imitation to innovation, more so as the advanced importing equipment and know-how, acquiring proprietary technologies become difficult to buy. higher level design and operating experience, to This requires: Analyses of the strategic implications be adapted, improved and applied, and then on of global change and competitive advantage for to indigenous innovation and exports, both of niches in regional and international markets; technology-based goods and know-how itself. Realistic assessments of current capabilities and resources, to identify the gaps and take action In core technologies a nation (and a company) can on technical support services, human resource make good progress (and money) by using other development, special financing instruments; peoples’ innovations and purchasing embodied research in capital equipment, with a balance Formulation of new legislation providing fiscal between pa yments for intellectual property rights incentives for entrepreneurial activity and small and investments in adaptation, improvement and business promotion, for protecting intellectual reverse engineering. Consider the case of Japan property and the environment; Identification of which increased the number of technical collaborations strategic, generic technologies suited to local and fees paid by eight-fold in the period 1960–70; endowments, and ‘prospecting’ for concepts and concurrently in this period, it spent nine times research results capable of rapid patenting, more on assimilating and upgrading this technology, prototyping and market entry; and Promotion of to build manufacturing capability for the subsequent the culture of informal net-working, information export of products and technology. sharing and risk-taking. It may also involve a university linkage, some engineering, production However, with the galloping pace of technical and marketing capabilities, and almost always change all countries need to move more rapidly from investment-led growth to innovation-driven require a committed entrepreneur or champion to development. This calls for investment in science mobilize resources, start the venture, survive, and and technology – one’s own or spill-offs from oth- (with luck!) thrive. ers, the efficient import of know-how and equip- ment, and foreign investment that brings these in. Research management The building of a knowledge society requires the continuous commitment of civil society and the Perhaps the greatest invention of this century is deployment of significant resources in scientific, the invention process at corporate industrial educational and social fields. laboratories, starting with Edison’s in New Jersey. These big establishments of GM, Xerox and IBM Te c h n o l o gy for national priority concerns were downsized, but are now being rejuvenated to again become the prime movers of innovation. The majority of the world’s population still depends Even so, the R & D budgets of a single large on the traditional knowledge of its flora and fauna multinational corporation are larger than the total for food and medicine.This knowledge — and the national budgets of say China or Korea or India. related bio-diversity — have to be preserved and Indeed, the research expenditures of the developing supplemented by modern technology. Rural folk countries are only around 5 percent of the whole have a shrewd sense of what helps or hurts them, world. In this situation, these countries have to and governments as well as international development increase budgets where possible, enter into research agencies have not built upon their knowledge, with collaborations where appropriate, and importantly, poor results. In many situations, the traditional improve the productivity of their own research. can be upgraded by blending with the advanced, At the same time, technology for national security, For governments in developing countries, the defense preparedness and national harmony has to responsibility is to restructure the publicly funded be the foremost priority. research and technology organizations, improving E V A L UATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 79 the facilities and the incentives for researchers, and TORCH program for advanced technologies. making them more accountable in their perform- Now it is investing Rmb 4.8 billion in the 1998- ance and more responsive to the needs of business. 2000 period on the ‘Knowledge Innovation Institutes in industrializing countries operating for Project’, with the objective of raising its S & T system a decade or more should now recover at say, three- to rank among the world’s top ten before 2010. At quarters of their annual operating expenses the same time, it has much work ahead in revitalizing through contract research and services. Many now its Village and Township Enterprises and re-structuring claim to do this, some using creative accounting. its State Owned Enterprises. In most industrializing countries, the bulk of the formal R & D is done in publicly-funded laboratories, 3 . BUILDING T E C H N I CAL HUMAN RESOURCES much less in the technical universities and corporations. Nevertheless, the extent of improvisation and In countries at the leading edge such as Korea 20 manufacturing improvements taking place on the percent of tertiary enrollments are in sciences and shop floor are impressive. engineering while the Scandinavian and former soviet Union countries spend up to 8 percent of A case example is presented in Annex-2 of the their GNP on education. Countries with poor Benchmarking of Research and Technology technological capabilities enroll and spend less Organizations,(RTOs) towards improving their than half as much. In some cases, there is the performance. With financial generous support misallocation of resources resulting in high from the Danish and Canadian Governments,The unemployment among engineering graduates World Technological and Research Organization along side high illiteracy rates. organized the ‘Best Practices Project’ to evaluate and draw lessons from 60 RTOs, 20 each in the What has changed in this decade is the function of Americas,Europe-Africa, and Asia.The methodology the university: It is no longer teaching alone. It and results can be of interest to all in the pursuit of must be restructured to take responsibility for an improved research performance. active role in the more complex field of economic development,covering a portfolio of applied and basic Corporate innovation research,consulting and community services,specialized training, and distance learning and tech-based venture Everywhere, managements have to be receptive to formation. While it moves towards becoming an new ideas, to listen. Corporations in countries such ‘entrepreneurial university’, the corporation moves as Japan and US are adopting unconventional towards becoming a ‘learning enterprise’. innovative ways to stimulate creativity in their personnel. Further, as researchers better understand Within the research universities, the Media the physiological, psychological and biochemical Laboratory at MIT, Robotics Institute at Carnegie- roots of the creative process, the left-right-front- Mellon,and the electrical engineering department back functions of the brain, they now expect to be at Stanford have been the creators of continuous able to instill more creativity in more persons innovations. Some such as artificial intelligence, expert systems, fuzzy logic and biochemical computers The traditional wisdom is that the product must will become the industries of the 21st century. be based on the needs of the market, but where there was no existing market — as for the now Also being changed the world over is the education ubiquitous ‘Post-It’, paper clip, stapler and photo- curricula. New courses being introduced are on the copier — it takes extraordinary persistence to management of technology, which provides succeed. In promoting innovation, the large knowledge on key issues at the interfaces of science, enterprises are mimicking the small, by out sourcing engineering, business and civil society. Young and old supplies and services and breaking out into small are being exposed to entrepreneurship development, intra-preneurial groups, while the small are acting which seeks to transform the nascent entrepreneur like the big through alliances and virtual consortiums. into a successful enterprise-owner. And every person The flexibility and creativity of an entrepreneurial needs to be proficient in the English language and techno-venture may lead to more break-through computing, because without these you cannot work, innovations than can be generated by larger sized play, or participate in the expansion of Internet firms in many sectors. and e-commerce. China has had success in promoting innovation Interestingly, high school students in the US fare through its SPARK program for rural enterprises poorly in mathematics and sciences,but they seem 80 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing to acquire the questioning attitudes and computing effectiveness of China’s large public investment in skills, which produce good results at the university facilities such as technology parks and business level. The research universities have become incubators would point to interesting lessons for seedbeds for innovations, with the University of future developments in China as well as in other California system earning $ 61 million through developing countries. royalties and its 528 patents in 1997, while Stanford and M.I.T. each created 15 start-up companies. 5 . INTERNATIONAL COOPERATION 4 .T E C H N I CAL SUPPORT SERV I C E S In a world shrinking rapidly with the advent of instantaneous, low cost computing and communi- These cover the technical infrastructure of institutions cations, the S & T system — and those who operate and capability to support the innovation system. and evaluate it — have to think and act globally. They include productivity centers, metrology and total quality management, indigenous consultancy A major effort to promote such international organizations and services to strengthen small cooperation and strengthen endogenous technical enterprises, on which large proportions of employment, capacities was the UN Fund for Science & Technology incomes and exports depend. for Development. Independent evaluations of UNFSTD-supported technical cooperation programs Business development services in China (eg the National Remote Sensing Center and the Beijing Center for Computer Software While there is consensus that small enterprises and Research) showed that with catalytic funding will be the prime creators of employment and significant capacities were built.The main lessons: growth in the future, these ventures need special help. the success depended on the commitment and Various financial and non-financial mechanisms discipline of the Chinese institutuions and on are converging in a synergistic system. There is a vigorous monitoring of progress. current debate on the outreach, impact and financial sustainability of alternative means to strengthen International cooperation requires a renewed, new ventures. Such services have to be initially continuing dialogue between the poor and the subsidized,which the developing country governments rich, without arrogance, with the purpose of can ill afford and donors usually resist (although finding joint solutions and sharing the benefits of such support receives major state funding in human knowledge fairly. OECD countries). The ability of a BDS (such as an incubator) to 6 . E V A L UATION CA PA C I T Y replace the resources it consumes and become financially sustainable can be shown by an analysis In developing national capacity for science and of the flow of funds in and out of the system over technology related programs, their special charac- at least 5 years. Sustainability implies the ability to teristics have to be noted. continue achieving positive outcomes and the durability of the benefits achieved. Effectiveness Whether it is in the speed of computing and can be expressed in terms of all the benefits communications or the unraveling of the human derived at the whole system in relation to the use genome, technologies are advancing at exponential of all resources and the overall satisfaction of those rates. While the 19th century ushered in the involved. Outreach depends on the replicability of telephone, electricity and automobile, this century the embodied concept and the means of reaching has brought us near-costless and ubiquitous larger numbers of enterprises. The metrics and information flows.The evaluators of such systems criteria of assessing BDS performance require have to be abreast of current practices and coming common understandings by donors and governments trends. They have to be sensitive to the political, as well as by the businesses they serve. cultural, ethical and environmental consequences, the possible impacts and outcomes of the programs Starting in the 1980s China has built impressive being evaluated. networks of tech parks and incubators. Their performance has now to be assessed and enhanced. These call for inter-disciplinary teams of professionals Annex 3 presents a case example on Evaluating the with objectivity, impartiality and utmost honesty. Performance of Technology Business Incubators As S & T programs typically have long time horizons in Brazil. A similar approach to evaluating the and high investments,the judgements made in the E V A L UATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 81 evaluation process also have far-reaching conse- are more complex in science and technology quences and potentials for high costs as well projects as the consequences cross national as benefits. boundaries and there is seldom a consensus on the interpretation , whether it relates to global warming or biologically-engineered produce. The donors have the additional problems of being responsible to their own taxpayers and responsive to the real E V A L UATION OF WORLD BANK PROJECTS needs of aid beneficiaries. The World Bank has been recently involved in loans for industrial technology development (ITD) projects in China, India, Brazil, Korea and REFERENCES Golden, W. T. Ed. Science and Technology Advice to the President, Turkey. As these are significant loans they justify Congress and Judiciar y, Pergamon Press,1988 serious efforts on loan appraisals, monitoring Goldman,M. and Ergas,H., Technological Institutions and policies: progress, evaluating results and follow-up. In Their role in developing technological capability in industry, turn,the national and Bank M and E capacity has World Bank Technical Paper No. 383,19997. to be strengthened. Lalkaka, R. Applying technology for enhancing competitiveness, in Human Resource Needs for Change in R & D Institutes, WAIT- Evaluation lessons on policy lending for ITD projects RO-NISTADS,New Delhi,1996. indicate the characteristics of successful projects as: Lalkaka, R. and Wu Ming Yu, Managing Science Policy and Technology s Competitive environment and demand-led; Acquisition: Strategies for China and a Changing World, Tycooly, Dublin and UN Fund for Science & Technology, New Yo rk ,1 9 8 4 . s Private sector participation and use of strong Lalkaka, Manual on Planning and Operation of Technology Business specialized institutions and financial interme- Incubators, UNESCO, Paris,1999,under publication. diaries; Need for integrated approaches; Mackay, K., Evaluation capacity development, World Bank, ECD s Unhindered access to international sources of Working Paper No. 6,1997. technology and protection of intellectual Nelson, R. R. Ed. National Innovation Systems, Oxford University property; Priority of financing for tech-based press,New York,1993. small and medium enterprises as well as for WAITRO, Best Practices for the Management of Research and non R, D & E support institutions; Technology Organizations, Special Report,1999. s Further, the Bank should support a broad World Bank, Knowledge for Development, Oxford University Press, range of lower-income developing countries, 1998/99. not only the advanced ones. World Bank, World Development Indicators, Washington DC,1999. UNDP. Human Development Report, 1999, Oxford University Press, Measuring performance requires agreement on New York. methods and data collection among those being UNDP-SIDA, Measuring and managing results: lessons for development measured and those wanting the results.The tasks cooperation, 1997. 82 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing ANNEX 1 of the research portfolios to deal with the Issues in evaluations special needs of small and large enterprises? of innovation sub-systems 4. What are the incentive systems to promote research productivity and increased research activity in private and state laboratories? CHECKLIST FOR SUB-SYSTEM-1: S & T POLICY 5. What needs to be done to introduce a program of bench-marking the performance 1. What promotional measures are being taken to of research institutes? see Annex 2. build a national consensus on the role of science- 6. How does the policy and process of international based development in strengthening the economy technological and financial collaborations and improving the lives of its people? stimulate (or restrict) the acquisition of selected 2. What are the country’s short, intermediate technologies? and long term goals to which the innovation 7. If a specific research program is under evaluation, system must be organized to contribute? how do the outputs and outcomes compare to 3. How can the processes of formulating the the original design? national S & T strategy be made more effective 8. How will the research results be scaled-up and through better involvement of all stake- taken to market? Will the benefits be replicable holders in rural and urban communities, civil and sustainable? society and defence establishment, particularly 9. What are the mechanisms at the public, women, youth, ethnic minorities and other university and private research institutes for disadvantaged groups? assisting the commercialization of research? 4. How are the legislative bodies that formulate the policy instruments and regulations being prepared for the content and implications of CHECKLIST FOR SUB-SYSTEM-3: their actions? T E C H N I CAL HUMAN RESOURCES 5. What are the present arrangements for providing advice on S & T issues to the executive and 1. What is the Government priority and legislative authorities in government? support for science and technology related 6. What methodologies are appropriate to make education and training at all levels,in schools, candid evaluations of the effectiveness of a university, continuous and distance learning, particular type of inter vention? vocational training? 7. What are the experience and qualifications of 2. How are the new computing, CD-ROM persons on inter-disciplinary teams for specific and Internet technologies being applied in evaluation tasks and how can their competencies preparing students, formulating training be strengthened? materials and methods? 8. How, by whom and when will the basic 3. What are special measures to improve proficiency information and data needed for evaluation in English language and computer literacy at purposes be collected and analyzed? all age levels, including senior citizens? 9. What measures are available for dealing with 4. How is tertiary education being prepared for complaints and disagreements with the findings the new millennium? and recommendations of the evaluation teams? 5. What programs are underway to create the ‘entrepreneurial university’ and the ‘learning enterprise’? CHECKLIST FOR SUB-SYSTEM-2: INNOVATION STRATEGY 6. What special programs and methods are being used to develop new curricula specific to local 1. What is the overall allocation for research and needs and conditions, such as ‘Management of development in the national budget, and how technology’ and Entrepreneurship development’? is this distributed among the industry, social 7. What are the special programs targeted at services,agriculture, health,education,defence education of women, ethnic minorities and and other sectors? disadvantaged groups? 2. What are to be the roles of basic and 8. What structural changes are being made in applied research, scientific break-throughs the research establishment, services and private and incremantal innovation,public,corporate, sectors to motivate technically competent persons university research? to work in the country rather than migrate? 3. What is the organization system in public What is being done to attract expatriate laboratories and the means for prioritization nationals to return to their country of origin? E V A L UATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 83 CHECKLIST FOR SUB-SYSTEM-4: 8. How can the national metrology, standards T E C H N I CAL SUPPORT SERV I C E S and quality system be strengthened for international competitiveness? 1. What are the main objectives and outputs 9. How are the national capabilities being of the service being evaluated? Importantly, developed for preserving the environment? what are the longer-term outcomes expected? 2. What provisions have been made in the original design and subsequent implemen- CHECKLIST FOR SUB-SYSTEM 5: tation for the collection of the data needed INTERNATIONAL COOPERATION for measuring effective performance, see Annex 3. 1. As a matter of state policy, is the S & T 3. Have the financial and technical resources activity such as can benefit significantly by been provided to achieve the desired linkages to other countries or donor agencies? results? Ample finance can be a disincentive 2. What will be the main purposes of interna- while scarce resources can mean that the tional cooperation: developing local skills, acquiring special equipment? Bench-marking project managers must continuously and learning from others? mobilize additional money to the neglect of 3. In pursuit of better international relations, their substantive tasks. would the project experience and results 4. How do the operations of the TSS fit in to obtained be shared with other interna- the larger strategy for the development of tional agencies, other developing countries? the concerned sector? What are the political 4. What are the benefits and costs related or other hidden agendas which the sponsors/ to forming an international association or financiers of this service seek and why? other mechanism for collaborative efforts 5. What are the levels of competence and in the future? training of the managers? What are the levels 5. What mechanisms are in place to attract of autonomy and accountability? t e ch n i cal and financial collaborations 6. If the service is in the public sector in selected S & T sectors? To better adapt, and receives public subsidy, does this apply and improve the technology acquired? affect private sector or non-governmental 6. How can the S & T system be enabled to providers of the same service? share its experiences with other countries, 7. How are national engineering and man- developed and developing? Is there a agement consultancy capabilities being national technical assistance program to promoted? assist other countries? 84 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing Annex 2 whereby comparisons are made on productivity, quality and good practices between a chosen Case Example: Benchmarking comparable set of RTOs. the effectiveness of Research & Benchmarking requires: an evaluation of the Training Organizations (RTOs) research process, the systematic collection of information, the continuous measurement of Scientific research and technological development performance, and the long-term commitment account for the bulk (up to two-thirds) of total to improve RTO effectiveness. annual S & T expenditures in many countries. The productivity of research and the performance of The RTO model can be conceptualized as in research institutes, largely supported by public Figure A2-1 below. funds in developing countries, are of importance to the national economy. As technological change In the WAITRO project, funded by DANIDA/ is moving at exponential rates, these RTOs can Denmark and IDRC/Canada, 60 RTOs were provide major support on training, research and examined (6 in Europe, 10 US/Canada, 20 Asia, consultancy services for small enterprises, which 10 central/south America and 14 in Africa) by an need to upgrade their total operations. international project team consisting of experts from Canada, Denmark and India. A comprehensive study has been undertaken by the World Association of Industrial and The following processes were benchmarked: Technological Research Organizations (WAITRO) 1. RTO governance (ownership, legal structure, on benchmarking of RTOs, that is, identifying governing board’s constitution, size and and measuring those practices which enabled powers,mission and vision ,l evel of autonomy, RTOs to enhance their operative and strategic mandate on sectors or regions). efficiency. Benchmarking is a systematic process 2. Financial management (funding methods, FIGURE A2-1: THE RTO MODEL SOCIETAL CONTEXT GOVERNMENTAL POLICIES RTO Ownership & INFRASTRUCTURE Managememt NETWORK: Partners, MARKET: Business colleagues, competitors Clients, beneficiaries, development (Local,national,international) funders Services Human resources; Facilities; Finance E V A L UATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 85 level, dependence on state support, flexibility, The WAITRO team has developed a consultancy retention of surplus/loss). manual of diagnostic tools for transformation 3. Services (types such as basic/applied, testing, efforts at RTOs,including check-lists and procedures training,consulting, identification of needs, quality). on strategic planning, change management, self 4. Business development (activities,effectiveness assessment,technology and industrial audits. RTO in attracting business, rewards, promotion, transformation excercises have been conducted in costing, affordability, prioritization). several countries based on the WAITRO findings. 5. Organizational and project management Their success depends in good measure on the (structure, team selection, unit responsibility to explicit administrative and financial support given meet RTO goals, management and follow-up) to the change process by all the stakeholders. 6. Capability building (staff development, capacity- building methods,funding of training/equipment). The process follows the trajectory [visualized in 7. Personnel management (Recruitment process, Figure A2-2]. below promotion, compensation, motivation, staff evaluation, internal communication). FOLLOW-UP A C T I O N S 8. Networking (relations with other technology providers and with industry to better serve them) Implementation of a bench-marking process 9. Policy and programs (Role in formulating requires the candid analysis of current practices national S & T and industrial policies, leveraging and the progressive move towards an organizational government programs). transformation. This is not easy. People resist change, particularly if the change is imposed on As precise information was not available to them. Real Change happens only when the quantitatively measure performances,broad macro- researchers from the bottom-up and the leadership level indicators had to be used as proxies. These can both recognize the need for change, can jointly included average growth of income from clients, ratios implement a constructive process rather than of grants to total income, expansion of the RTO in assign blame, and the leaders can motivate and services and territory served, and recognition in communicate a vision for the future in which terms of national and international patents and everyone wins. publications. The above indicators were given weighted ratings, for a total of 100 points. Concurrently, the research managers need special orientation on new techniques of busi- The main lessons learned regarding good RTO ness management under conditions of global practices are summarized as follows: change. This is best done at special training courses focused on the needs and conditions s Despite differences in culture, there is a com- of RTOs. monality of problems such as poor patronage from industry, declining support from government An RTO benchmarking network calls for sharing and difficulties in commercializing research. of information which in turn requires confiden- tiality and trust among the parties concerned. A s Reliable management information systems are data base of best-practice indicators (with the prerequisites for implementing good practices. secret codes rather than actual names of RTOs) can then be made available to those participating in s While RTOs from North America,Europe and the network. the industrialized Asian countries were more successful overall,many effective practices were The next step would be to develop cooperative found in RTOs from developing countries. projects between small enterprises and the RTOs, s The best practice emerged when the RTO where enhanced research performance can be applied was focused on the needs of their clients. to the real needs of the small industry sector. FIGURE A2-2: THE RTO PROCESS DIAGNOSIS PLANNING IMPLEMENTATION By local/external experts Frank diagnosis of practices Identify expertise needed Seminars & diagnostic tools Adaptation of good practices Find funding, compare results Develop report for Board Communication/consensus Develop long-term plans FEEDBACK 86 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing Annex 3 to greater business opportunities and promote spin-offs. Good measures of performance of an Case example: Evaluating the performance incubation system are the medium-term benefits accruing to small businesses, sponsors, local of Te c h n o l o gy Business Incubators (TBIs) community, region and nation, Figure A3-1. The overall system assessment requires that donors FRAMEWORK FOR PERFORMANCE A S S E S S M E N T make provision for — and pursue — the collection of the needed information by the management The majority of incubation programs worldwide team, on firms in the facility and those leaving as can be characterized as “public-private partnerships” well as other parameters. While some of the in which initial (and often continuing) financial coefficients can be calculated readily, others require support is received from the state bodies. Many complex social benefit-cost estimations. governments consider them as part of the business infrastructure, and the evidence indicates that the annual taxes and other benefits from regional INSTITUTIONAL A N A LYSIS OF BIOMINAS AND PA R Q T E C economic development more than offset the capital and operating cost subsidy. Private sector The location, implementation, governance and participates when it sees that the program will lead management factors of the Biominas and ParqTec FIGURE A3-1: ASSESSMENT OF INCUBATOR IMPA C T S , EFFECTIVENESS AND SUSTAINABILITY Taxes Income State finance Government Community Bank/venture capital Federal/state/city Private sector Seed Capital Credit; Equity Policy Infrastructure Support Mentoring SPONSORS/MANAGER Entrepreneur Enterprises INCUBATOR Goods; Services Innovations Faculty; Graduates Change Employment Replication University Culture National/export market Research Institute Skills Entrepreneurship Work force University-market links I. Impact/Outreach II. Effectiveness III. Sustainability 1. Enterprises created 7. Employment per net $ subsidy 13. Revenue surplus (6 years) 2. Survival rate of enterprises 8. Taxes paid per net $ subsidy 14. Services cost recovery 3. Jobs generated (6 years) 9. Income, sales & exports 15. University-business links A. in incubated/affiliated firms 10. Research commercialized 16. Stakeholder satisfaction B. in graduated firms 11. Disadvantaged groups 17. Tenant/graduate satisfac. C. indirect jobs 12. Incubator expansion 18. Changes in culture 4.Enterprises reached 19. Enhancement of skills 5. Replication of “pilot” model 20. Leveraging state policies 6. ‘Extra-curricular’ activities 21. Enhanced self-esteem E V A L UATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 87 incubators are reviewed below. The main charac- agencies such as FAPEMIG and SEBRAE; teristics are summarized in Table A3-1.Incubators and a vibrant private sector with BIOBRAS as a need to be sited where there is a strong business world-class insulin producer. Biominas incubator infrastructure with availability of scientific talent, occupies a prime site of 10,000 sq m land adjacent good living conditions, and positive government to the publicly-funded, research laboratory — and community support. On these counts, both CETEC. It is a custom-built building with excellent the Biominas and ParqTec incubators have good biochemical laboratories. locations. Both plan major technology parks linked to the incubators. ParqTec: The city of Sao Paulo is Brazil’s major business hub. Some 230 km to the north-west is Biominas: Minas Gerais (MG),the second largest the city of Sao Carlos, population 200,000, which industrial state with its capital at Belo Horizonte, has the distinction of having the highest density has traditional strengths in minerals, mechanical of PhDs in science/engineering — one for every and automotive manufacture, and now in bio- 230 inhabitants. Industry is concentrated on technology. MG has an impressive technical consumption goods and mechanical appliances. It infrastructure, with universities such as UFMG, has two public universities and private learning Vicosa, Ouro Preto and Uberlandia; research centers, laboratories for cattle and animal protein institutes Rene Rachou, and FUNED; support development, and over 70 enterprises in aero- TABLE A3-2: PRO FORMA INCOME AND EXPENSE AT PARQTEC AND BIOMINAS INCUBA T O R S All figures in US$ ,000. Rate R/$ =0.84 PARQTEC BIOMINAS 1996 1997 1998 1997 1998 REVENUES Rental Income 24.4 38.6 39.4 28.9 81.7 Fees from Tenants 2.1 2.1 2.5 3.1 12.9 Utilities Cost Recovery 9.2 18.5 18.5 — — Partnership with SEBRAE 139.4 62.2 239.4 186.0 186.0 TOTAL REVENUE 175.1 121.4 299.8 218.0 280.6 EXPENSES Manager 27.7 38.6 32.9 36.0 54.0 Admin Assist/tech adviser 0.0 0.7 4.5 10.0 5.0 Lab coordinator — — — 7.0 13.4 Receptionist/Secretary 0.8 2.5 4.2 7.3 14.3 Other Professionals 22.8 26.8 33.4 37.9 48.4 Fringe Benefits 5.9 8.4 10.9 — — SUB-TOTAL STAFF 57.3 77.0 85.9 98.2 135.1 Bldg. Maintenance/Cleaning 31.9 70.6 70.6 8.0 17.5 Utilities/Telephone (net) 18.5 24.4 23.5 30.8 48.4 Travel & Promotion 1.3 0.8 20.2 3.2 3.2 Supplies 0.7 1.3 0.7 4.5 9.3 Audit & legal 9.1 9.1 20.2 3.2 4.9 Insurance 5.0 3.8 4.7 — — Publications 56.3 — — Interest/debt repayment — — — 30.4 39.7 TOTAL EXPENSES 123.7 187.0 282.1 178.3 258.1 OPER. SURPLUS (DEFICIT) 51.4 (65.6) 17.7 39.7 22.5 88 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing nautics, informatics, new materials and robotics. The figures in Table A3-3 below should be Fundacao ParqTec de Alta Tecnologia houses considered as preliminary; as it is often difficult to the CINET and SOFTNET incubators together get data from privately held firms on sensitive with related facilities. topics such as sales, payroll and taxes. At ParqTec, the tenant firms have 69 employees while 17 (of FINANCIAL VIABILITY AND OUTREACH the 21) graduated firms have 168, making a total of 237 direct jobs. Estimates of income and expenses As ParqTec has been in operation at its present location since 1990, it has more results to evaluate Estimates of income and expenses for the two in compared to Biominas,which has been operating incubators are shown in Table A3-2 below. Such only since July 1997 in its new permanent facilities. data is hard to collect in most countries. Annual The 1997 estimate of public capital and operating revenue from services (in relation to total) is subsidy for ParqTec and the personal and corporate considered low. Support through the partnership taxes payable would be approximately as follows, with SEBRAE constitutes more than half of based on the limited data available: revenue, on average. Total jobs (with employment multiplier of 1.5*) 357 Both Foundations plan to achieve greater financial Capital cost subsidy per year self-sufficiency through the development of (20-year straight line depreciation) $ 19,150 Technology Parks. Biominas has initiated the feasibility analyses and business planning for a Operating subsidy per year (average of last 3 years) $147,000 biotech related park adjacent to the existing Capital and operational incubator-CETEC complex. subsidy per year $166,150 The São Carlos Science Park is on a 172,000 Total subsidy over 7 years $1,163,050 square meter property that it owns in a prime Subsidy cost per job industrial location. Planning and design for the (excluding jobs in affiliates) $ 3,258 first 3,500 square meter building has been Estimated payroll & corporate taxes by tenants & graduated firms $1,054,320 completed along with a master plan for the balance Return on public investment of the property. The first structure will house the as taxes per year $6.34 per $ subsidy ParqTec headquarters as well as incubator modules * Indirect employment multiplier based on ranges for similar for 64 additional tenant enterprises. The master economic activities in the U.S. development plan also includes industrial sites for lease to technology-based enterprises as well as two multi-tenant buildings and a convention TABLE A3-3: JOBS AND T A X E S , 1 9 9 7 ( A P P R O X I M A T E ) , US $ center, designed to establish ParqTec Biominas Cost effectiveness and stakeholder satisfaction Jobs (tenants The evaluation methodology in this case example and graduates)* 237 92 uses a combination of qualitative description, quantitative analysis, and stakeholder perceptions. 1997 payroll $1,854,000 $1,030,040 The stakeholders interviewed for this purpose 1997 sales $9,846,990 $2,558,320 were the public and private sponsors as well as the incubated enterprises themselves. 1997 payroll taxes payable $ 463,500 $ 258,510 Businesses incubated and jobs created 1997 corporate taxes payable $ 590,820 $ 153,500 An approach to evaluating the effectiveness of Total taxes $1,054,320 $ 412,010 incubation programs is to look at the number of businesses incubated, the success rate, and the Initial Investment number of jobs created by incubated firms. As in incubator $ 383,000 $1,940,000 noted, both incubators have to aggressively recruit * This includes current tenants plus the one graduate tenant at more tenants and affiliates as well as increase the Biominas and 17 graduated firms at ParqTec for whom information throughput of graduating businesses. is available. Taxes are estimated at 25% on payroll and 6% on sales. E V A L UATION INITIATIVES IN SCIENCE AND TECHNOLOGY PROGRAMS 89 The subsidy per job should decline at ParqTec as development, and business training on site. more firms graduate and continue to expand, and as additional incubator space becomes available. For mixed-use incubators, which typically have LESSONS LEARNED much larger areas and less services for tenants, the subsidy cost per job can be much lower. A point to To summarize, the ParqTec and Biominas incubators note is that while the investment is made once, the studied have had positive impacts and outcomes jobs continue, and it is useful to think in terms of on their respective city and state economies in “job-years” in the stream of benefits. nurturing entrepreneurs and creating sound enter- 1997 taxes realizable from sales and payroll of prises with good survival rates. ParqTec has generated ParqTec tenants and graduates could be about six employment with public subsidy of around US$ times the subsidy. 3,258 per job, without including jobs in affiliates. The estimated return in the form of taxes could be Performance evaluation by incubator tenants and graduates about $ 6 per dollar of public subsidy. Biominas: Representatives of the present enterprises The linkages to universities and research institutes and the one graduated were asked to evaluate the have resulted in commercialization of some technologies. effectiveness of the incubator as well as the advantages The sponsors and tenants at both incubators have and disadvantages in being tenants. All persons interviewed felt that the program is of value to expressed satisfaction with the results achieved, them. The major benefits expressed were as follows: particularly the help in marketing, business planning, help in dealing with bureaucracies resulting in and securing government permits. Both are helping faster permits; valuable assistance in marketing their government sponsors in promoting technological and faster time to market for new products; development together with other social aspects such excellent infrastructure and labs; interaction with as reinforcing the cultures of entrepreneurship and other tenants; and legal assistance. university-research-business cooperation. ParqTec: The incubator tenants and graduates That being said, Biominas and ParqTec have the i n t e rv i ewed expressed satisfaction with their major challenges ahead of enhancing their experiences. The major benefits cited were its: operational effectiveness through innovative good location for a startup venture, access to activities and creative financing, increased occupancy facilities such as labs. telephone, internet, and fax and higher fees for quality services, with more service, valuable marketing assistance received, affiliate companies and anchor tenants, in order to legal assistance for incorporation and patent reduce dependence on state subsidies. 90 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing REVIEW OF DRAFT OF SCIENCE & TECHNOLOGY EVALUATION STANDARD OF CHINA 13 Chen Zhaoying, Vice President,NCSTE, China 1 . BA C K G R O U N D The science and Technology Evaluation Standards of China (hereinafter referred to as the Standards) is developed in accordance with the following demands: s Demands from evaluation performers (the people who “do” evaluation) Evaluators/evaluation organisations need unified standards in order to regulate their evaluation practice, to standardise the planning and regulate their evaluation, to raise level of science and implementation of evaluation, to raise the overall level of science and technology evaluation, to ensure the evaluation quality and to reduce and avoid disputes. s Demands from evaluation users (the people who “use” evaluation results) Through the Standards the evaluation consignors and users can better understand science and technology evaluation in order to utilise the evaluation results. s Demands from government agencies Mrs. Zhu Lilan, Minister of Science and Technology pointed out at the National Technical Innovation Conference that we should establish a scientific and impartial evaluation system, implement it according to law and gradually standardise our science and technology evaluation. 2. CHARACTERISTICS A SELF-DISCIPLINARY DOCUMENT The Standards is a professional self-disciplinary document. The formulation of the Standards comprises an important part of the management of science and technology evaluation. The Evaluation Standards Committee is responsible for the interpretation and revision of the Standards, but has no power to enforce the implementation. In accordance with the Standards, member organisations of science and technology evaluation, such as federations and associations, may conduct self-management and self-control of the evaluation development and development disciplinary actions against the violation of professional conduct and code of ethics. THE ACCEPTED EVALUATION STANDARDS WITH A BINDING FORCE FOR ENFORCEMENT According to international practice, evaluation standards can be approved by relevant government agencies as an accepted evaluation standard with a binding force for enforcement. REVIEW OF DRAFT OF SCIENCE & TECHNOLOGY EVALUATION STANDARD OF CHINA 91 The initiative and development of science and by other professional associations for it was legislated technology evaluation in China have been and approved by the government as an accepted supported by the government. The evaluation on appraisal standard Almost all the customers in the government investment in science and technology United States require that the appraisal assignments will occupy an important place in the overall should comply with USPAP. The authority of science and technology evaluation for a period of USPAP has been widely acknowledge by government time in the future. The administrative department agencies and companies in the United States. It has of the government should approve the standard or become the basis for formulating an international part of it for use of relevant evaluation organisations appraisal standard. as a reference. Since the promulgation of USPAP on April 27, 1987, it was revised 13 times. Significant changes THE STANDARDS FOR PROFESSIONAL PRACTICE have been made while the major structure remains. As a standard of professional performance, the Standard is a condensation of the theories and INTERNATIONAL PROPERTY APPRAISAL STANDARDS (VO L . 9 4 ,9 5 ) practices of science and technology evaluation, reflecting the appraisers’understanding, viewpoints International Property Appraisal Standards was and experiences of professional evaluation. Evaluation formulated and promulgated by the International work requires both creativeness and strict compliance Evaluation Standard Committee. The framework with the standards, and a combination of professional is designed in accordance with the objectives of responsibilities and morality. property appraisal activities and the derived values. SCOPE E V A L UATION STANDARDS OF OECD COUNTRIES The Standard is a standard for science and “Evaluation Standards for Science, Technology technology evaluation profession. Any evaluation and Innovation Projects” OECD-1990 (excerpt). activity relating to science and technology including planning, achievements,projects,organisations,policy and personnel can be covered by the Standard. G OVERNMENT AUDITING STANDARDS (UNITED STATES AUDITING A D M I N I S T R A T I O N ) s Financial audit EFFECTIVE T I M E F R A M E s Audit relating to financial affairs s Performance audit The Standard is used not only to address the prob- s Comprehensive audit (contract audit) lems in present evaluation activities, but will also play roles for a long period to come. Therefore, pertinence, continuity should be adhered to during FRAMEWORK OF CHINA’S ASSET APPRAISAL STANDARD SYSTEM the formulation of the Standard. The Chinese Evaluation Association is now making research on and formulating the assets appraisal standards in the light of USPAP. 3 . MAJOR REFERENCE SYSTEMS FOR FORMULATION OF THE STANDARDS 4 . FRAMEWORK AND CONTENTS UNIFORM STANDARDS FOR PROFESSIONAL A P P R A I S A L The differences of evaluation activities in different PRACTICE (USPAP) OF THE UNITED STATES counties are reflected in their market economy conditions, legal and management systems while USPAP is a professional standards influential not the basic evaluation methods and techniques have only in the United States, North America, but all little differences all over the world (refer to the over the world. It is a standard document with a International Assets Evaluation Standards Vol. self-disciplinary force for the profession formulated 94&95). The Standard takes the internationally by professional appraisal association and recognised adopted open frame structure, which can ensure the by appraisers, appraisal organisations and appraisal overall stability of the Standards and accommodate clients. The effectiveness and influences of the the changing conditions of evaluation so that the standards surpass the ordinary standards formulated Standards will be improved step by step. 92 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing FRAMEWORK AND CONTENTS OF SCIENCE s Requirements for evaluation report AND TECHNOLOGY EVALUATION STANDARDS s Utilisation of Evaluation results s Quality control of evaluation activities P a rt I. The main body s Archiving of evaluation activities. The first level:basic standard The Third level:statements on standards This part is composed of basic principles, codes of Statements on standards forms as an organic part ethic and terminology of science and technology of the Standards,which further identify, intemperate evaluation. The basic standard shows the features and explain the evaluation standards and have the of science and technology evaluation and serves as same force as the text of the standards. As the a basic and general standard for science and statements are flexible, they should be stipulated as technology evaluation profession. It is the basis required and after being well conceived. The statements for guiding and regulating all kinds of technical evaluation activities. The basic standard sticks to can be upgraded into part of the standards according the basic rules of evaluation profession, and at the to the practical situation and maturity level during same time, has its own characteristics of technical the process of the revision of the standards. The first evaluation. The main contents include: edition of the statements on standards contains s Standard of professional conduct. the following: s Principle of independence and impartiality, s How to maintain independence, objective and principle of avoidance and misleading, confi- impartial? (three elements endanger the inde- dentiality, and fees. pendence). s Professional morals (activities violating the s How to justify that the appraisers capabilities professional morals). meet the requirements of the evaluation s Capabilities and qualification of the appraisers/ assignment? evaluation organisations,the structure of appraisers’ s Different requirements and characteristics of competency and capabilities,continuing education different types of evaluation activities. and training of the appraisers. s Major concepts and methods of classification s Alternatives (requirements for using the for science and technology evaluation in alternatives). foreign countries. s Basic concepts and terminology of science and s When to use the alternatives regulations in the technology evaluation. process of evaluation? What are specifically The second level:technical standard required for the evaluation report when the alternative regulations are used? Composed of the major evaluation procedures, and s How to make use of comments from the the obligations and rights of the related parties of evaluation results caused by the limits? the evaluation, the technical standards are the basic part of the Evaluation Standards. It describes P a rt II. R e f e r e n c e s the main principles, key issues and points to be noted in the planning and implementation of the Guidelines for the implementation of t ypical evaluation, and stipulated in detail the requirements science and technology activities and contents of the evaluation report. The main contents of the technical standard include the The following is some references, cases and advis- evaluation report. The main contents of the technical es which have no binding forces for appraisers in standard include: light of some typical science and technology eval- s The roles,obligations and rights of the parties uation activities. involved in the evaluation s Evaluation on science and technology programs s Evaluation procedures s Evaluation on project relevance s Preparation of the evaluation s Evaluation on project acceptance s Evaluation planning s Quality control of individual evaluation The preconditions/presupposition of evaluation, evaluation standards, evaluation index, reliability Explanation on the background,concepts, of evaluation, rating system of evaluation methods and processes of the development of the s Information collection, data reviewing and sorting. Science and Technology Evaluation Standards s Selection of evaluation methods. s Multi-index comprehensive evaluation model, Introduction of some representative evaluation case study, comparative study. standards and norms of both domestic and abroad REVIEW OF DRAFT OF SCIENCE & TECHNOLOGY EVALUATION STANDARD OF CHINA 93 5 . METHODS AND PROCESSES OF PA R T I C I PATION AND T R A N S PARENCY OF THE STANDARDS FORMULATION THE DEVELOPMENT OF THE STANDARDS Importance has been given to the participation of the appraisers and clients in the process of the STRESS ON FUNDAMENTAL W O R K formulation and revision of the Standards. It is planned to set up an evaluation standards committee The development of American Uniform Standards at proper time and promulgate the draft standards for Professional Evaluation Practice (USPAP) over to the public for comments and suggestions so as the past ten-odd year has given us much enlightenment. to improve it to a top level of the profession. The First and foremost, USPAP is a condensation and early participation of people working in this summary of the theories and practice of evaluation. profession and other related fields enabled the A complex system engineering, USPAP is developed universality of the standards and laid a good foun- on the basis of sound theories and practice. In the dation for the understanding and implementation process of formulating the Evaluation Standards of the standards after it is formulated and revised. for Science and Technology of China, much attention is given to the fundamental work, especially the Publicity should be given to the standards after it research on evaluation theories and the summing-up is formulated and revised, especially the updated of experiences and cases, in order to lay a sound information. The revised Uniform Standards for basis for the development of the Standards. Professional Evaluation Practice should be published each year. The revised part should be notified in the preface in order to allow the appraisers and DEVELOP THE STANDARDS ON THE BASIS OF INTERNATIONALLY customers to understand the latest development ACCEPTED EVALUATION PRINCIPLES; TAKE IN THE NEW CONCEPTS and make a better use of the standards. AND PROGRESS OF THE INTERNATIONAL EVALUATION PROFESSION The practice of international evaluation profession REVISE THE STANDARDS REGULARLY AND IMPROVE IT STEP BY STEP shows that the basic evaluation principles in all the countries,mainly the code of conducts and professional As the evaluation standards need to be revised and morals, are almost the same, although different reprinted with the deepening of the evaluation countries have their own national conditions and practice and the changing of the situation, it different types of evaluation activities. Base on the should be formulated to meet the basic demands at internationally accepted basic evaluation principles, first rather than covering everything. New contents the Standards generalised the experience and should be added to the Standards and revision research results of the evaluation practice in China, should be made step by step. In this way we can and took in the new concepts of and the latest eventually produce an evaluation standards for science progress of the theoretical research on international evaluation profession. and technology with Chinese characteristics, which is influential and has a strong binding forces. Generally speaking, standards are developed by two As evaluation theories are not perfectly established ways:inductive and deductive. The inductive way is at present, many basic definitions and principles to form specific rules, first according to experiences are not clearly identified. We should not finalise and general practice, and then formulate general the present research results of evaluation. Instead, evaluation standards. From specific to general,the only the matured theories and experiences can be standards formulated this way are descriptive. absorbed in the evaluation standards. Making standards in the way of deduction is to identify the generally accepted concepts first and The Evaluation Standards Committee will make then formulate the specific evaluation standards. timely revision according to the situations of practice From general to specific, standards formed this and the implementation on the basis of reviewing way are normative. Judging from the evaluation the evaluation theories and practices. The revision standards formed this way is normative. Judging will be illustrated in news media and professional from evaluation standards of different countries journals in order to obtain sufficient attention and and regions,the standards would not be consistent understanding of the public. The standards,then, if they are formulated just on the basis of experiences will always reflect the latest theories of the evaluation and general practice. Based on the conceptual structure profession and remain proper consistency to serve of evaluation standards of other countries, the as the guidelines of evaluation practice. In summary, Standards are formulated with methods of combining China is still at the first stage in formulating the deductive and inductive ways. The experiences and evaluation standards for science and technology. lessons learned in the science and technology evaluation Therefore the standards will not be made perfect practice in China are fully related in the Standards. to cover everything at one step. 94 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing PART IV ECD Opportunities for National Initiatives and International Cooperation E V A L UATION C A PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing GUIDE TO DIAGNOSING ECD NEEDS: Video Speech 14 (Full Presentation is available in booklet form) Keith Mackay, Senior Evaluation Officer, OED World Bank DEVELOPMENT OF M&E SYSTEMS WITHIN A GOVERNMENT — DIAGNOSING ECD NEEDS + READINESS s Danger if we take too simple an approach and assume M&E a “good thing” and its merits should be obvious to all development assistance. s World Bank experience (and other dev’t asst agencies) is that most forms of capacity building are difficult to achieve, and require long-term efforts if they are to be sustainable. s This also applies to ECD. s A key lesson from experience is that to be successful/sustainable we need to understand broader institutional environment. s Another key lesson is that there is no single, best model for ECD. • ECD has to be tailored to each country’s circumstances. • One motivation for ECD is that many of the most advanced economies in the world t ypically have some t ype of national system for M&E. • Countries such as Australia, Canada,USA, + various European countries for instance. • But when you compare different countries, you find that their approaches differ considerably. s Increasingly, a national M&E system is not being seen as a stand-alone activity, but as part of sound governance: • performance measurement is seen as counterpart to performance management; • the multiple uses of a national M&E system can provide support to a number of different public sector reforms. In Ghana for instance, there are a number of related and mutually- supporting reforms: • there is a medium-term expenditure framework (MTEF) a – e this sets out funding allocations for each type of activity for every ministry over next 3 years. The MTEF also focuses explicitly on planned government outputs,and this necessitates the accurate measurement of actual outputs; • there are performance improvement plans a – e prepared by each ministry and setting out particular initiatives designed to help build a client- oriented service culture in the civil service; • and there are also employment contracts for senior officials a-e these ensure a close focus on what results (what performance in terms of promised outputs) are expected from senior officials over the coming year; • and a big push to decentralize government to the district level a – e this creates GUIDE TO DIAGNOSING ECD NEEDS: Video Speech 97 a greater need to be able to measure gov- do to make a national M&E system is a ernment performance at the regional and Government decree, plus some training district levels, to provide greater support to • but really, our experience tells us that this district government management. will not be enough • Ghanaian government (GOG) officials recognize that national + sectoral M&E s But another key lesson we have learned is systems can support all these reforms. that there has to be real demand æ real • actually, GOG officials increasingly under- commitment — by a Government to using stand that M&E does more than support M&E if a system is to be developed in a the reforms: they are coming to realize sustainable manner that a number of the reforms will not be • An example of this is our work with the fully successful unless the government’s Government of Benin to develop performance performance is measured more systematically indicators to support budget decision-making and rigorously. — as part of an MTEF • the Finance Minister in Benin is strongly com- s So in your country, helps if you identify the mitted to this use of performance indicators different uses + different users of M&E info, and the different types of M&E s Consideration of types of M&E — and of the different users and uses of them — has big s Different types fill different roles: implications for how you structure M&E 1. Basic financial data on government spending functions and implications for who plans or on different activities, projects, sectors commissions M&E, who conducts it, and • these data are complementary to basic socio- who uses it. economic statistics or development indicators 2. Performance monitoring/indicators on govern- The World Bank has prepared an ECD ment inputs, processes and outputs Diagnostic Guide1—to help Governments and 3. Cost-benefit analysis analysts in addressing these issues and in developing 4. Formal program evaluation an action plan for ECD. 5. More qualitative reviews 6. Performance audits a – e typically done by s There are eight key steps : a National Audit Office • the Guide provides checklists of questions and issues to help you s Different uses of M&E findings include: • it’s really what we call a SWOT (strengths, 1. Policy formulation and planning to help weaknesses,opportunities and threats) analysis clarify the pros and cons of alternative types of policy inter vention; 1. Identify key ministries and their formal 2. A second is resource allocation in budget processes so M&E findings can be one relationships input to Government decision-making, • ministries with major role in performance along with a number of other influences management — i.e., in resource alloca- on government; tion decisions and in ongoing manage- 3. To support management via greater results ment of sectors + projects orientation — the learning function; • e.g., key central ministries and large 4. Accountability—this is the quid pro quo in line ministries response to devolution of authority to managers. 2. Diagnose public sector incentives and the s The need to understand which of these different unwritten “rules of the game” uses of M&E likely to be most important in • it is these unwritten rules + incentives, your country. not formal decrees and procedures,which determine the behavior of ministries, s It also helps to understand which different line managers and other civil servants. potential users of M&E actually want this info. • so what is extent of autonomy + authority of line managers? s A mistake which we evaluators make is to be • what are rewards + incentives for good supply-driven: we may think that all we need performance? 1 Keith Mackay, Evaluation Capacity Development: A Diagnostic Guide and Action Framework, ECD Working Paper Se ries No 6, January 1999,The World Bank. 98 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing • what are sanctions for poor performance? on key issues and dimensions: • so, this Step helps answer the question: • demand and supply “does a performance culture exist?” – where is existing demand for M&E, If not, “what are the roadblocks to and how can it be consciously built up achieving it?” by “winning hearts + minds”? – where is existing supply of M&E,and 3. Find out how budget resource allocation again, how can it be built up? and line management decisions are actually • different types of M&E; taken a – e the realit y, not the rhetoric • evaluation infrastructure and systems • this Step looks at actual systems,structures, – what planning systems would be roles and info flows desirable to help decide which areas of Government should be evaluated? 4. Next step is to investigate the extent of – what mechanisms are necessary to influence of M&E on budget and line ensure evaluations are completed in management decisions timely manner? • again, “what is the reality?”, not the rhetoric – what mechanisms would be set up to provide M&E findings to users of them? 5. Map out the evaluation activities and capa- • possible support from development bilities of ministries and other organizations assistance agencies • this step focuses on supply of M&E,and – this could include technical assistance and on processes for making M&E findings advice, training, twinning arrangements; available support for ECD diagnoses, etc; • think of timelines for an action plan,and 6. Map out the evaluation activities and contri- sequencing and speed of implementation bution of development assistance agencies – need some flexibility in approach • development agencies often focus on – but helps to set indicative targets for building M&E for donor-funded projects, 1,3, 5 and 10 years in the future; and and this may have contributed to national • sustainability a – e how sustainable do M&E capacities you think your ECD efforts might be? • and they might have been involved in – what are some of the risks and threats supporting other relevant capacity- to it? building work in the past, in areas such as building national statistical systems s This is a detailed and systematic approach; it helps ensure that your ECD work takes full 7. Understand the major public sector reforms account of opportunities and possible barriers underway or completed in recent years æ these may well provide synergies with s I regret not being with you in person to discuss ECD a – e as Ghana, for example, illustrates these issues —I’m sure your debate will be interesting and I wish you every success with 8. Map out an action plan for ECD, focusing your ECD efforts. GUIDE TO DIAGNOSING ECD NEEDS: Video Speech 99 PART V Meeting Wrap-Up E V A L UATION C A PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing MEETING WRAP-UP 15 Christine Roth,Evaluation Advisor, UNDP Presentations and discussions during the 1999 Beijing Conference on Evaluation Capacity Development gave us a glimpse of the ever-evolving environment we are living in. At the dawn of the 21st century, in a world of increasing globalization, competition, and reduced public resources, scrutiny is growing for governments and development agencies to demonstrate value for money in public services. As a consequence, the field of evaluation is changing rapidly to deal with the new demands placed on it. First and foremost, evaluation is being recognized as an essential part of good governance and public sector reform. With the advent of public sector reforms, governments and aid agencies have begun to revisit their monitoring and evaluation (M&E) arrangements, looking for ways to develop effective learning systems and promote transparency and accountability. One of the key elements of this reform has been the adoption of results-based approach to management (RBM). The Conference focused particularly on the interface between RBM and monitoring and evaluation in connection with public sector reform and good governance. The shift to RBM requires a focus on development results, or outcomes, rather than inputs. In this paradigm, the continuous assessment of whether intended outcomes are being achieved is critical. M&E becomes a central function in a results oriented environment. Moving to outcomes also requires the development of strategic partnerships and the concerted efforts of many actors in society. Methodologically, this approach is raising several issues, such as attribution and accountability inter alia. During the past two days,we have engaged in lively discussions on some of these methodological issues as well as broader topics including: s The Independence of the evaluation function s The necessary link between evaluation and decision-making s The balance between learning and accountability s The challenges facing evaluation • Political environment • Systemic problems • Internal capacities (human and financial resources) s The role of partnerships • Joint evaluations with governments, aid agencies and development banks • Involvement of stakeholders and civil society in evaluation exercises Of the many constraints hindering the establishment of effective monitoring and evaluation systems at national level, we have collectively identified the following: s Weak demand for evaluation s The lack of political commitment and appropriate institutional support for evaluation s The lack of standards and criteria for evaluation s The lack of local expertise and resources for evaluation ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the New Millennium 103 As a first step to lifting these constraints, the concept evaluators engaged in development related activities of networks has emerged in our discussions as a and will support the national efforts in strengthening powerful instrument to mainstream the culture evaluation capacity de velopment. of evaluation in the public sector by allowing: (1) exchange of experiences, (2) access to best Last,an opportunity appeared during this meeting practices, (3) sharing databases. for the development agencies in general,UNDP in particular, to maximize the synergies between the Concretely, several countries have committed to work of development banks – the Asian establishing evaluation networks at the national Development Bank in this particular case – and of and regional level. Sri-Lanka and China are two national entities to promote the role of evaluation examples and support will be provided as needed in good governance and public sector reform. by the international community. As we move forward in the field of ECD, At the international level, both UNDP and the Beijing will represent one of the milestones World Bank, in agreement with several other that has helped furhter refine our understanding donors are promoting the establishment of an of issues and contributed in shaping the inter- international development evaluation association. national community’s game plan to meet the This association will help to form links between challenges ahead. 104 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing PART VI Annexes E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from an International Conference in Beijing ANNEX I: List of Participants CHINA Mr. Sun FUNQUAN Professor NCSTE – National Center for Science and Technology Evaluation P.O. Box 3814 Beijing, People’s Republic of China100038 Tel: (86) 10-685 14841/685 15544-3805 Ext.3825 Fax: (86) 10 68514854 Email: tac@public3.bta.net.cn Mr. Li XINNAN Professor, Vice-President NCSTE – National Center for Science and Technology Evaluation P.O. Box 3814 Beijing, People’s Republic of China100038 Tel: (86) 10-685 18087/685 15544-3805 Ext.3818 Fax: (86) 10 68514854 Email: lxin@public.bta.net.cn/tac@public3.bta.net.cn Ms. Chen ZHAOYING Professor, Deputy Director Member of Executive Committee, International Association for Technology Assessment and Forecasting Institutions P.O.Box 3814, Beijing, P.R. China 10038 Tel: (86)10 68515544/66515544 Fax: (86)10 68514854 Email: tac@public3.bta.net.cn Mr. Fang YAN Head, Evaluation NCSTE –National Center for Science and Technology Evaluation P.O. Box 3814 Beijing, People’s Republic of China100038 Tel: (86) 10-685 18087/685 15544-3805 Ext.3818 Fax: (86) 10 68514854 Email: tac@public3.bta.net.cn DEMOCRATIC PEOPLE’S REPUBLIC OF KO R E A Mr. Kim SEON-BIN Director Public Welfare Technology R&D Division Ministry of Science and Technology Government Complex-Gwacheon Seoul, Democratic People’s Republic of Korea, 427-760 Tel : (82-2) 503 7606 Fax :(82-2) 749 1417 Email : kimsb@mostws.most.go.kr ANNEX I: List of Part i c i p a n t s 107 FIJI Mr. Leonid Nukhimovich KOMAROVER Head, Economic Analysis and Ms.Litia MAWI Prognosis Department Director of Management Improvement Office of the Prime Minister of Public Service Commission the Kyrgyz Republic 29 Greene Street, Howell Road 205 Chuy Avenue #401 Suva, Fiji Islands Tel: 22 29 24 Tel: (679) 386 001 Fax: 66 13 82/66 25 77 Fax: (679) 301 714 Mr. Sabyrbek MOLDOKULOV Mr. Yuxue Xue Deputy Chairman, State Committee on Assistant Resident Representative Foreign Investments UNDP Fiji Private Mail Bag 58a Erkindik Bvld, #210 Suva, Fiji Tel: (996-312) 22 32 92 Tel: (679) 312 500 Fax: (996-312) 66 10 45 Fax: (679) 301 718 Email: registry@undp.org.fj Ms.Elzira SAGYNBAEVA National Programme Officer UNDP Kyrgystan INDIA 90 Chuykov Street Dr. S.P. PAL Bishkek 720000, Kyrgystan Advisor, Planning and Tel: (906-312) 666100 Evaluation Organization Fax: (996-312) 660557 Planning Commission Email: root@undp.bishkek.su; Fo.kgz@undp.org Yojana Bhavan Parliament Street New Delhi - 110001 M A L AY S I A Tel: (91 11) 3710162/91 11 221 3653 Fax: (91 11) 371 7681/91 11 372 5491 Dr. Selva RAMACHANDRA Assistant Resident Representative KAZAKHSTAN UNDP Malaysia Ms. Svetlana CHERIMISSINOVA Wisma UN Deputy Head State Programmes Division Block C, Damansara Offices Complex Regional Policy Department Damansara Heights Ministry of Economy 50490 Kuala Lumpur, Malaysia 33 Pobedy Prospect/52 Abai Str, Tel: (60 3) 253 1020/ (60 3) 255 9122 Astana, Republic of Kazakhstan Fax: (60 3) 255 2870 Tel: (7-3172) 11 81 34/29 Fax: (7-3172) 11 81 36 Mr. Ab.Alim BIN ZAKARIAH Principal Assistant Director Economic Planning Unit KYRGYSTAN Prime Minister’s Department Blok B6, Aras 2 Miyegombo ENKHTSETSEG Kompleks Jabatan Perdana Menteri Head, Programme Support Unit Pusat Pentadbiran Kerajaan Persukutuan UNDP, Kyrgystan 62502 Putrajaya 90 Chuykov Street Tel: (603) 88882767 Bishkek 720000, Kyrgystan Fax (603) 88883768 Tel: (906-312) 666100 Fax: (996-312) 66055 Email: root@undp.bishkek.su MONGOLIA Mr. Chorobek IMASHEV Deputy Minister of Finance Mr. Rentsen BATMEND The Kyrgyz Republic Principal Officer, Cabinet Secretariat 58 Erkindik Blvd., #305 Room 374 Tel: (996-312) 22 74 55 Government House, Fax: (996-312) 66 16 45 Ulaanbaatar 12, Mongolia 108 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing Dr. Dungersuren SUKHJARGALMAA Mr. Kithsiri ASOKA KASTURIARACHCHI Assistant Resident Representative Assistant Resident Representative UNDP Mongolia UNDP Sri Lanka 7 Erhuu Street 202-203 Bauddhaloka Mawatha Sukhbaatar Region Colombo, Sri Lanka Ulaanbaatar, Mongolia Tel: (94 1) 582630 Fax: (94 1) 581 116 Tel: (976 1) 327 585 Email : asoka.kasturiarachchi@undp.org Fax: (976 1) 326 221 THAILAND Email: fo.mon@undp.org/registry@undp.org.mn Ms. Netnarumon SIRIMONTHON Assistant Resident Representative and NEPAL Programme Management Specialist Mr. Devendra Raj Panday UNDP Thailand Chairman United Nations Building Nepal South Centre 12th Floor Tripureswar Tel: (66 2) 280 0519 Khathmandu Fax: (66 2) 280 0556 Email: fo.tha@undp.org P.O. Box 8242 Tel: (977) 1 247 656 Fax: (977) 1 240 685 VIETNAM Email: nesac@npl.healthnet.org Mr. Nguyen DINH CUNG Director, Macro Economic Policy Department Central Institute for Economic Management PA K I S T A N 68 Phan Dinh Phung Ms. Shaheen MASUD Honoi, Vietnam Joint Secretary, Economic Affairs Division Tel: (84 4) 845 8241 Ministry of Finance and Economic Affairs Fax: (84 4) 856 6795 Address: MISSING TEXT MISSING TEXT THE UNITED NATIONS DEVELOPMENT PROGRAMME (UNDP) Tel: (051) 9201 706 Fax: (051) 920 5971 / (051) 921 0734 Mr. Khalid MALIK Director, Evaluation Office UNDP, Room DC1-478 PHILIPPINES One United Nations Plaza Mr. Ernesto BAUTISTA New York, NY 10017 Assistant Deputy Resident Representative United States UNDP Philipppines Tel: (212) 906 5095 NEDA Sa Makati Building Fax: (212) 906 6627 106 Amorsolo Street Email: Khalid.Malik@undp.org Legaspi Village Mr. Antonio MOLPECERES 1229 Makati City Learning Resource Advisor Metro Manila, Philipppines 304 E 45th Street Email: ernesto.bautista@undp.org New York, NY 10017 Room FF-1104 Tel: (212) 906 6724 SRI LANKA Fax: (212) 906 5120 Email: Antonio.molpeceres@undp.org Mrs. Nanda ALAHAKONE Director, Post Evaluation Unit Ms. Christine ROTH Ministry of Plan Implementation & Evaluation Advisor UNDP, Room 476 Parliamentary Affairs, 3rd Floor One United Nations Plaza "Sethsiripaya" New York, NY 10017 Battarumulla United States Sri Lanka Tel: (212) 906 5722 Tel: 863 954 Fax: (212) 906 6627 Fax: 862 485 Email: Christine.roth@undp.org ANNEX I: List of Part i c i p a n t s 109 THE WORLD BANK OPERATIONS EVALUATION DEPA R T M E N T / ASIAN DEVELOPMENT BANK (ADB) WORLD BANK INSTITUTE Mr. Peter C. DARJES Operations Manager Mr. Keith MACKAY Asian Development Bank Senior Evaluation Officer Address: OED Room G 3212,The World Bank Tel: (63-2) 632 4111 1818 H Street, N.W. Washington, DC 20433 Fax: (63-2) 636 2161 Email:pdarjes@mail.asiandevbank.org U. S. A. Tel: (202) 473 7178 Fax: (202) 522 1655 CONSULTANTS Email: kmckay@worldbank.org Ms.Annette BINNENDIJK Mr. Ray RIST P.O. Box 146 Evaluation Advisor Basye, Virginia 22810 USA WBI Room G 3192,The World Bank Tel: (540) 856 2624 1818 H Street, N.W. Washington, DC 20433 Email: annetteb@shentel.net U. S. A. Tel: (202) 458 5625 Mr. Leonard J. EARLY Fax: (202) 522 1655 Department of Finance, Australian Public Sector Email: rrist@worldbank.org 33 Bringley Cr. Fraser ACT, 2615, Australia Tel: (61 2) 6259 5390 Fax: (61 2) 6259 5382 Ms.Jody ZALL KUSEK Email: learly@pcug.org.au Evaluation Advisor WBI Room G 3192,The World Bank Mr. Marc HOLZER 1818 H Street, N.W. Washington, DC 20433 Executive Director, National Center for Public U. S. A Productivity, Rutgers University, NJ, USA Email: Jkusek@worldbank.org 7th floor Hill Hall 360 King Blvd Newark, NJ 07102 Tel: (973) 353 1351/5504 Ext.23 OECD Fax: (973) 353 5907 Email: mholzer@pipieline.com Mr. Niels DABELSTEIN Head of Evaluation Secretariat Mr. Adil KHAN DANIDA/OECD/DAC WP UNDP Senior Advisor Taj Samudra Hotel Danish International Development Assistance, 25, Gale face Centre Road Ministry of Foreign Affairs Colombo 3 2, Asiatisk Plads, DK 1448 Copenhagen, Sri Lanka Denmark Tel: (00941) 889190 Tel: 45.33.92. 00.39/92.00.00 Fax: 94.1. 446348/330509 Fax: 45.33.92.16.50 Email: akhan@sltnet.lk Email: Niedab@um.dk Mr.Rustam LALKAKA Mr. Shoko KIMURA President, Business & Technology Evaluation Division, Development Strategies Economics Cooperation Bureau, 1441 Third Avenue, Suite 5A Ministry of Foreign Affairs, OECD/DAC, Japan New York, NY 10028, USA 2-2-1, Kasumigaseki, Chiyoda-ku Tel: (212) 452 1678 Tokyo 100, Japan Fax: (212) 535 0060 Email: rlalkaka@earthlink.net Tel:81 3 35 92 83 42 Fax:81 3 35 93 80 21 Mr. Jacques TOULEMONDE Director Ms. Fiona SHERA Center for European Evaluation Expertise Economic Advisor C3E, 13 bis Place Jules Ferry-F69006 DFID, United Kingdom Lyon, France Tel: (44) 171 917 0545 Tel: 33 (0) 4 72 83 78 80 Fax: (44) 171 917 0561 Fax: 33 (0) 4 72 83 78 81 Email: F-Shera@dfid.gov.uk Email: toulemonde@c3e.fr 110 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing ANNEX II: Conference Agenda International Conference On Evaluation Capacity Development Beijing, People’s Republic Of China,27-28 October 1999 Wednesday 27 October 1999 8:00- 9:00 Registration MORNING SESSION 09:00- 10:00 Opening Ceremony: Session Chair:Mr. Li Xinnan, Vice President, Chinese National Center for Science and Technology Evaluation, RPC (NCSTE) Opening Speech: Ms. Deng Nan, Vice Minister, Ministry of Science & Technology Address: s Mr. Khalid Malik, Director, UNDP Evaluation Office s Ms. Ma Dexiu, Director, Industry Department, State Commission for Development & Planning s Mr. Ray Rist, Senior Advisor, World Bank Institute s Mr. Zhan Jingtao, Director, Public Expenditure Department, Ministry of Finance s Mr. Yu Liegui, Director, Industry & Transportation Department, Ministry of Finance s Mr. Wang Jianzeng, Director, Science & Technology Department, State Commission for Economy & Trade 10:00 - 10:15 Launching of the Handbook “Result-Oriented Monitoring and Evaluation” Mr. Qi Rang, Director, Development & Planning Department, Ministry of Science & Technology Mr. Khalid Makik, Director, Evaluation Office, UNDP 10:15 - 10:30 Break THE EVO LVING CONTEXT FOR EVALUATION CA PACITY DEVELOPMENT Session Chair:Mr. Ray Rist,Advisor, World Bank 10:30 -12:15 Statements: The New Dimensions in ECD in International Development Mr. Khalid Malik, Director, UNDP Evaluation Office ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the Ne w Millennium 111 Background paper: The Asian Perspective in Evaluation Capacity De velopment: The Challenges of the New Millenium Mr. Adil Khan, Senior Advisor, M&E UNDP/UNOPS, Sri Lanka Background paper: The S&T Evaluation in China: Practices and Roles Ms. Chen Zhaoying, Vice President of NCTSE Background paper: Aid Evaluation/A donor perspective Mr. Niels Dabelstein, Head of Evaluation Secretariat DANIDA, OECD/DAC WP: Questions and Answers 12:15-12:30 Objectives of the Conference Mr. Antonio Molpeceres, Learning Resource Advisor, UNDP 12:30-14:00 Lunch AFTERNOON SESSION: PERFORMANCE MANAGEMENT IN THE PUBLIC SECTOR Session Chair:Representative of Ministry of Finance, PRC 14:00-14:45 Background papers: Results Based Management:A Donor Agency Perspective Ms. Annette Binnendijk, Consultant to the Evaluation Office, UNDP Respondents: Mr. Fang Yan, Chief of Project Evaluation, NCTSE National Delegations: Mongolia, India Questions and Answers 14:45-15:30 Background paper: Linking Evaluation to Policy Formulation and Budget Processes Mr. Ray Rist, Senior Advisor, World Bank Institute Respondents: Mr. Len Early, Department of Finance & Administration, Australian Public Sector Ms. Xu Yaoling, Vice President, NCSTE Questions and Answers 15:30-15:45 Break Announcement for reception Session Chair:Mr. Wang Jianxin,Director of Evaluation Division, Planning Department,Ministry of Science & Technology, RPC 15:45-17:30 Background paper: Evaluation Capacity Building in the People’s Republic of China Mr. Peter C. Darjes, Asian Development Bank Respondents: Mr. Zhao Lu, Director of Science Division, Public Expenditure Department, Ministry of Finance National Delegation: Sri Lanka, Fiji Background paper: Public Performance Evaluation and Improvement Mr. Mark Holzer, Executive Director, National Center for Public Productivity, NJ, USA Respondents: Ms. Jody Kusek, Advisor, World Bank Institute Mr. Gu Wenxing, President, Shanghai Science Research Institute Questions and Answers: Wrap up UNDP Evaluation, Fang Yan, Lalkaka 18:30 - 21:00 Reception given by UNDP Evaluation Office 112 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing Thursday 28 October 1999 MORNING SESSION : A) MONITORING AND EVALUATION IN A RESULTS-BASED MANAGEMENT A P P R OA C H B) EVALUATION INITIATIVES IN SCIENCE AND T E C H N O L O G Y Session Chair:Mr. Khalid Malik,Director, UNDP Evaluation Office 09:00-9:45 Background paper: M&E and RBM: Balancing Learning and Accountability Concerns,the UNDP Experience Ms. Christine Roth, Evaluation Advisor, UNDP Evaluation Office Respondents: Mr. Dai Guoqing, Director, Finance Division, Conditions Finance Dept., Ministry of Science & Technology National Delegations: Kyrgyzstan Questions and Answers 9:45-10:30 Background paper: Evaluation Partnerships Mr. Niels Dabelstein, Chairman Working group on Aid Evaluation, OECD/DAC Respondents: Br. Yang Sai, Director, Beijing Kehuan S&T Development Center National Delegation: Vietnam, Pakistan Questions and Answers 10:30-10:45 Break Session Chair:Ms.Li Maoming, Senior Advisor, NCSTE 10:45-11:30 Background paper: Evaluation Tools and Methodologies:Case Studies Mr. Jacques Toulemonde, Director, Center for European Evaluation Expertise Respondents: Mr. Du Zhanyuan, assistant Director, Development & Planning Department, Ministry of Development & Planning Department, Ministry of Science & Technology National delegation: Russia, Malaysia Questions and Answers 11:30-12:30 Background paper: Evaluation Initiatives in Science and Technology Programs Mr. Rustam Lalkaka, former Head of UNFSTD Respondents: Mr. Wang Jianxin, Director, Evaluation Division, Development & Planning Dept., Ministry of Science and Technology National delegation : Korea Background paper: Review of Draft of Science & Technology Evaluation Standard Ms. Chen Zhaoying, Vice President, NCSTE Respondents: Mr. Zhang Zheng, Director, Wuhan Science & Technology Evaluation Center National delegation: Kazakstan Questions & Answers 12:30-14:00 Lunch AFTERNOON SESSION: ECD OPPORTUNITIES FOR NATIONAL INITIATIVES AND INTERNATIONAL COOPERATION Session Chair:Mr. Ray Rist,Advisor, World Bank Institute 14:00-14:45 ECD opportunities for National Initiatives Video Speech: Guide to Diagnosing ECD Needs: Mr. Keith McKay, Senior Evaluation Officer, World Bank ASIAN PERSPECTIVE IN EVALUATION CA PACITY DEVELOPMENT: The Challenges of the New Millennium 113 Questions and Answers 14:45-15:45 Group Work: National needs and planned steps for ECD National delegations, UNDP, World Bank and Asian Development Bank as facilitators 15:45-16:15 Current trends and future steps for ECD at the national level Presentations by rapporteurs of working groups 16:15-17:00 Break Session Chair:Ms. Chen Zhaoying, Vice President,NCSTE 17:00-17:15 Meeting wrap-up: Ms. Christine Roth, Evaluation Advisor, UNDP Evaluation Office 17:15-17:45 Concluding remarks: s Mr. Ming Tinghua, Chairman, Invention Association of China (former Director of Chinese Patent Office) s Mr. Ray Rist, Senior Advisor, World Bank Institute s Mr. Yang Qiwen, Director, Basic Research Department, Ministry of Science & Technology s Mr. Khalid Malik, Director, UNDP Evaluation Office s Mr. Li Xinnan, Vice President of NCSTE, Vice President of NRCSTD Friday 29 October, 1999 18:30-21:00 RECEPTION HOSTED BY THE NCSTE: CLOSING CEREMONY, GREAT HALL OF THE PEOPLE Chair:Ms.Li Maoming, Senior Advisor, NCSTE Presence of senior officers Mr. Zheng Guoan, Director, Ministry of Science & Technology 114 E V A L UATION CA PACITY DEVELOPMENT IN ASIA: Selected Proceedings from the International Conference in Beijing UNITED NATIONS DEVELOPMENT PRO G R A M M E © Evaluation Office, 2000 Evaluation Office United Nations Development Programme One United Nations Plaza New York,NY 10017, USA Tel. (212) 906 5095, Fax (212) 906 6008 Internet: htt p://intra.undp.org.eo or htt p://www.undp.org/eo