55853 Fifth Conference of the Latin America and the Caribbean Monitoring and Evaluation (M&E) Network Challenges in Monitoring and Evaluation: An Opportunity to Institutionalize M&E Systems June 2010 Gladys Lopez Acevedo Katia Rivera Lycia Lima Helena Hwang Editors Document of the World Bank and Inter-American Development Bank The content of the papers on thisreport are the responsibility of the authors and do not necessarily represent the views of the World Bank or the Inter-American Development Bank (IDB). The World Bank and the IDB do not guarantee the accuracy of the information included in this publication and accept no responsibility whatsoever for any consequence of their use. The World Bank © 2010 by The International Bank for Reconstruction and Development / The World Bank. 1818 H Street, N.W. Washington, D.C. 20433, U.S.A. Internet: www.worldbank.org E-mail: viconferencia@worldbank.org All Rights Reserved Printing and Manufactured in Washington, D.C. First Printing: June 2010 The findings, interpretations, and conclusions expressed in this book are entirely those of the authors and should not be attributed in any man- ner to the World Bank, to its affiliated organizations, or to members of its Board of Executive Directors or the countries they represent. The World Bank does not guarantee the accuracy of the data included in this publication and accepts no responsibility for any consequence of their use. The boundaries, colors, denominations, and other information shown on any map in this volume do not imply on the part of the World Bank Group any judgment on the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development/The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; Telephone: 978-750-8400; Fax: 978-750-4470; Internet: www.copyright.com. The World Bank Edition: 1st Cover: Larry Mayorga Lopez-Acevedo, Gladys (Trust Fund TF096261-SFLAC) Contents Abbreviations and Acronyms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii Acknowledgments. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Foreword. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Welcome Address . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix Conference Overview. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Session 1: Challenges to Evidence-Based Decision Making: The Role of Monitoring and evaluation and Other Factors. . . . . . 1 Monitoring and Evaluation System in the Minas Gerais State Government: Aspects of Management . . . . . . . . . . . . . . . 1 Transparency Index of Public Entities: Experience in Evaluating Risks of Corruption in Government Administration at National, Departmental, and Municipal Levels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Challenges for Evidence-Based Decision Making: The Experience of Chile and Chief Features of Its Design, Implementation, and Use of Information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Session 2: Institutional Arrangements for Monitoring and Evaluation Systems: International Experiences . . . . . . . . . . . . . . . . . 25 Monitoring and Evaluation in the Government of Canada: Lessons Learned from 30 Years of Monitoring and Evaluation Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 The Monitoring and Evaluation System in Sri Lanka: Experiences, Challenges, and the Way Forward . . . . . . . . . . . . . 33 Institutional Arrangements for International Monitoring and Evaluation Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47 Institutionalizing Monitoring and Evaluation by the Public Service Commission of South Africa: A Continuing Journey . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50 Session 3: Institutional Arrangements for Monitoring and Evaluation Systems in Latin America . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Challenges and Opportunities for the Institutionalization of Regional Institutional Agreements for Monitoring and Evaluation Systems in Latin America: The Experience of Costa Rica. . . . . . . . . . . . . . . . . . . . . 57 U.S. Monitoring and Evaluation Trends: Using Performance Information to Improve Results. . . . . . . . . . . . . . . . . . . . 61 Institutional Arrangements for M&E Systems in Latin America: The Case of Mexico: Performance Evaluation System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66 The Creation of an Evaluation System of Higher Middle Education in Mexico. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75 Latin American Monitoring and Evaluation Systems and Their Governance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82 Special Session: National System for Performance Management and Evaluation (Sinergia). . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89 National Management and Results Evaluation System Achievements, Challenges, and Perspectives. . . . . . . . . . . . . . . . 89 Session 4: Monitoring and Evaluation Systems: Capacity Building for Monitoring and Evaluation and Academia-Government-Civil Society Partnerships . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 Conceptual Framework for Developing Evaluation Capacities Building on Good Practice in Evaluation and Capacity Development. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 iii M&E SYSTEMS AND M&E TOOLS The Experience of the Latin American and Caribbean Institute for Economic and Social Planning in Evaluation Training. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Strengthening Monitoring and Evaluation Capacities through the Design and Implementation of National Results-Based Monitoring and Evaluation Systems: Lessons Learned in Africa, Asia, and Latin America. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113 Session 5: Institutional Arrangements and related Policies to ensure the quality, access, and use of performance information. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 Challenges in Monitoring and Evaluation: An Opportunity to Consolidate the M&E Systems. . . . . . . . . . . . . . . . . . 123 Performance Information: How Web 2.0 Can Help . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 Institutional Arrangements and Policies of Monitoring and Evaluation Systems to Ensure Quality, Access, and Use of Information . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Select Presentations from the Fourth Meeting of the Latin American and Caribbean Monitoring and Evaluation Network. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 Institutionalizing Impact Evaluation: Lessons from Latin America and Other Regions . . . . . . . . . . . . . . . . . . . . . . . . 139 Building a Monitoring and Evaluation System Based on Results for Social Development . . . . . . . . . . . . . . . . . . . . . . 144 Impact Assessment of Regional Development and Peace Programs in Colombia. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151 Progress and Challenges of Monitoring and Evaluation Systems in Latin America and the Caribbean. . . . . . . . . . . . . 156 The Brazilian Monitoring and Evaluation Network: A Report on the Creation and Development Process . . . . . . . . . 164 References and Additional Bibliography. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173 Figures, Tables, and Boxes Figure 1: M&E System in the Public Policy Cycle. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 Figure 2: Strategic Map, Minas Gerais State Government . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Figure 3: An Overview of the Canadian M&E system. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Figure 4: Organization and Implementation of M&E in a Government Department . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Figure 5: MfDR in Sri Lanka. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Figure 6: Whole-of-Government Approach. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34 Figure 7: Snapshot of the Agency Results Framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36 Figure 8: Agency Performance Measurement Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 Figure 9: Data Capture at Source. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42 Figure 10: MfDR and the Paris Declaration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43 Figure 11: Performance Evaluation System, First Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67 Figure 12: Results-Based Management Strategy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68 Figure 13: Performance Evaluation System, Second Stage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 Figure 14: Budget Programming Structure . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73 Figure 15: National Baccalaureate System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 76 Figure 16: Milestones of Reform, 1991–2009. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90 Figure 17: Results Road Map. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 Figure 18: National Development Plan Goals. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Figure 19: Annual Evaluations, 2006–10. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 Figure 20: Goals Management System. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 Figure 21: Evaluation Principles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98 Figure 22: Levels of Capacity Development. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 Figure 23: Five Dimensions of Governance. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 Figure 24: Structure of the Government M&E System in India. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128 Figure 25: Linking Outputs, Impacts, and Outcomes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 Figure 26: Different Measurement Techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132 iv C ontents Figure 27: Three Pathways for the Successful Institutionalization of IE. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 Figure 28: Results-Based M&E System, 2004–07. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 Figure 29: Results-Based M&E System, 2008–12. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149 Figure 30: RDPP Change Theory. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 Figure 31: Management Cycle Pillars. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 Figure 32: MfDR Score by Pillar and Country Group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 Figure 33: Brazilian M&E Web Site . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 Figure 34: Growth of Membership in the Brazilian M&E Network. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 Figure 35: Dashboard of the Brazilian M&E Network. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 170 Table 1: Logical Framework Matrix. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Table 2: Working Design for Evaluation of Programs and Institutions (ex post), 2000–09 . . . . . . . . . . . . . . . . . . . . . . 16 Table 3: Uses and Actors in Ex Post Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19 Table 4: The Evolution of M&E in Canada. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Table 5: Drivers for M&E. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 Table 6: Key Features That Reinforce Credibility and Quality of the M&E System. . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Table 7: Implementing the M&E System . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Table 8: Building M&E Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Table 9: Changes Introduced through NCESP 3294. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 91 Table 10: Channels of Communication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 Table 11: Evaluation Principles and Levels of Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102 Table 12: Evaluation Capabilities. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103 Table 13: Tips to Reinforce an Enabling Environment for Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106 Table 14: Tips to Develop an Institutional Framework for Evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Table 15: Tips for Stakeholders to Develop Skills and Knowledge of Evaluators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108 Table 16: Training Evaluation Levels. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 116 Table 17: Governance Indicator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124 Table 18: Modification of Indicator. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125 Table 19: Measurement Techniques. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130 Table 20: Annual External Evaluations, 2004–07 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 Table 21: Country Ranking by Degree of MfDR Development. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158 Table 22: MfDR Pillars by Country Group. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159 Table 23: M&E Indicators by Country Groups. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 161 Box 1: Basic guidelines for collection, review, and appraisal of evaluation data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Box 2: Structural independence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100 Box 3: Selecting the head of evaluation. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 Box 4: Definitions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 Box 5: Comparison of Web 2.0 and Government 2.0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131 Box 6: Juana’s story. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Box 7: Alternative responses to the social dilemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 Box 8: Information dissemination. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 Box 9: PRODEV survey results. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 163 v Abbreviations and Acronyms AAPSCOMS Association of African Public Service Commissions (South Africa) ADB Asian Development Bank AEP Annual Evaluation Program AFREA African Evaluation Association AIM Africa Impact Evaluation Initiative ANIF Asociación Nacional de Instituciones Financieras (Association of National Financial Institutions) AOPs Annual Operational Plans ARF Agency Results Framework BNB Banco do Nordeste do Brasil CAPAM Commonwealth Association for Public Administration CCCI Internal Control Coordination Commission CEE Centre of Excellence for Evaluation CENEVAL Centro Nacional de Evaluación para la Educación Superior (National Center for the Evaluation of Higher Education) CERCAPAZ Cooperación entre Estado y Sociedad Civil para el Desarrollo de la Paz CES Canadian Evaluation Society CGU Office of the General Comptroller CLAD Latin American Center for Development (Centro Latinoamericano de Administración para el Desarrollo) CMA Comprehensive Management Analyses CONEVAL Consejo Nacional de Evaluacion de la Politica de Desarrollo Social (National Evaluation Council on Social Development Policy) CONPES Consejo Nacional de Política Económica y Social (National Economic and Social Policy Council) CREAM Clear, Relevant, Economic, Adequate, and Monitorable CTMA Technical Monitoring and Evaluation Chamber (Câmara Técnica de Monitoramento e Avaliação) DEPP Departamento de Evaluación de Políticas Públicas (Public Policy Evaluation Department) DFABM Department of Foreign Aid and Budget Monitoring DfID U.K. Department of International Development DNP Departamento Nacional de Planeación (National Planning Department) DPR Departmental Performance Report ECB Evaluation Capacity Building ECD Evaluation Capacity Development vii M&E SYSTEMS AND M&E TOOLS ECLAC Economic Commission for Latin America and the Caribbean EDMS Electronic Documentation Management Systems ePMS Electronic Project Monitoring System ERO Education Review Office(New Zealand) EIS Evaluation Information System EMS Educación Media Superior (Higher Middle Education) ENLACE Evaluación Nacional del Logro Académico en Centros Escolares (National Evaluation of Academic Achievement in Academic Centers) FEB Federal Expenditure Budget FLACSO Facultad Latinoamericana de Ciencias Sociales (Latin American Faculty of Social Sciences) FMAP Financial Management, Auditing, and Procurement FNE Northeast Financing Constitutional Fund GAO U.S. Government Accountability Office GPRA Government Performance and Results Act (United States) GTZ German Cooperation Agency HME Higher Middle Education HMEES Higher Middle Education Evaluation System HMESP HME Scholarship Programs IAL Incentives-Alignment [program] ICRME Intersectoral Committee for Results-Based Management and Evaluation ICT Information Communication Technology IDB Inter-American Development Bank IDEA Institute for Development in Economics and Administration IE Impact Evaluation IEG Independent Evaluation Group 3IE International Initiative for Impact Evaluation IFAD International Fund for Agricultural Development ILPES Latin American and Caribbean Institute for Economic and Social Planning INEE Instituto Nacional de Evaluación de la Educación (National Institute of Evaluation of Education) IPDET International Programme for Development Education Training IRHME Integrated Reform of Higher Middle Education ISMGA Information System for Monitoring Government Action JPF João Pinheiro Foundation JPR Joint Progress Review KPI Key Performance Indicators LFA Logical Framework Analysis LFM Logical Framework Matrix MAF Management Accountability Framework viii A bbreviations and A cronyms MAPSESI MatrizAnual de Programación, Seguimiento y Evaluación Sectorial e Institucional (Annual Institutional and Sectoral Programming, Follow-up, and Evaluation Matrix) MBC Competitive Brazil Movement (Movimento Brasil Competitivo) MDS Ministry of Social Development and Hunger M&E Monitoring and Evaluation M&E-MR Monitoring and Evaluation-Managing (Management) for Results MDGs Millennium Development Goals MEC Monitoring and Evaluation Commission MEU Monitoring and Evaluation Unit MfDR Managing (Management) for Development Results MR Managing (Management) for Results MIDEPLAN Ministerio de Planificación Nacional y Política Económica (National Planning and Economic Policy Ministry) MIP Management Improvement Program MPFA Modernization of Public Financial Administration MPI Ministry of Plan Implementation MPOG Ministry of Planning, Budgeting, and Management MRI Matrix of Results-Based Indicators MRRS Management Resources and Results Structure MTEF Medium-Term Expenditure Framework NACH National Anti-Corruption Hotline (South Africa) NAO U.K. National Audit Office NBS National Baccalaureate System NCESP National Council of Economic and Social Policy NDP National Development Plan NDS National Development Strategy NES National Evaluation System NGO Nongovernmental Organization NIEE National Institute of Evaluation of Education NIST National Institute of Standards and Technology (Canada) NMRS National Management and Results System NONIE Network of Networks for Impact Evaluation NOR National Operations Room NPD National Planning Department NPHI National Public Health Institute NPS New Programmatic Structure NREGS National Rural Employment Guarantee Scheme (India) NRP National Reform Program ODA Official Development Assistance ix M&E SYSTEMS AND M&E TOOLS OECD/DAC Organisation for Economic Co-operation and Development/Development Assistance Committee OED Operations Evaluation Department OMB Office of Management and Budget PART Program Assessment Rating Tool PES Performance Evaluation System PET PRODEV Evaluation Tool PIE project implementation unit PMDI Integrated Minas Development Plan (Plano Mineiro de Desenvolvimento Integrado) PPAG Multiyear Government Action Plan (Plano Plurianual de Ação Governamental) PPE Public Policy Evaluation PPX Performance and Planning Exchange PRODEV Program to Implement the External Pillar of the Medium Term Action Plan for Development Effectiveness PROGRESA Education, Health, and Nutrition Program (Programa de Educación, Salud y Alimentación) PROMAP Programa de Modernización de la Administración Pública (Program for the Modernization of Federal Public Administration) PSC Public Service Commission RBB Results-Based Budget(ing) RBB-PES Results-Based Budgeting-Performance Evaluation System RBBS Results-Based Budget System RBM Results-Based Management RBMES Results-Based Monitoring and Evaluation System RBMS Results-Based Management Strategy RDPPs Regional Development and Peace Programs (Colombia) RF Results Framework RMAF Results-based Management and Accountability Framework SAMEA South African Monitoring and Evaluation Association SEADE Sistema Estadual de Analise de Dados SEDESOL Secretariat of Social Development (Secretaría de Desarrollo Sociali) SEEMS Sistema de Evaluación de la Educación Media Superior SEMS Subsecretaría de Educación Media Superior (Under Secretariat of Higher Middle Education) SEP Secretaría de Educación Pública (Secretariat of PublicEducation) SEPLAG Minas Gerais State Planning and Management Secretariat SFC Federal Internal Control Bureau SFGG/SISMEG System for the Follow-Up of Government Goals SFP Ministry of Public Administration (Secretaría de la Función Pública) SGPGM System of Government Programming and Goal Management SHCP Ministry of Finance and Public Credit (Secretaría de Hacienda y Crédito Público) x A bbreviations and A cronyms SINERGIA National System for Performance Management and Evaluation (Sistema Nacional de Evaluación de Gestión y Resultados) SLEVA Sri Lanka Evaluation Association SMART Specific, Measurable, Achievable, Relevant, and Time-bound SMS School Management Service SNCG Secretariat of the National Comptroller General SOPS State of the Public Service (South Africa) TBS Treasury Board Secretariat TCU Federal Government Auditing Court UNDP United Nations Development Programme UNEG United Nations Evaluation Group UNESCO United Nations Educational, Scientific and Cultural Organization UPEPE Unidad de Planeación y Evaluación de Política Educativa (Planning and Evaluation of Educational Policy Unit) xi Acknowledgments T he papers in these proceedings were Sector Manager for Poverty and Gender; and Roberto Gar- presented at the Fifth Annual Conference cía López, PRODEV Coordinator for the Inter-American of the Latin American and the Caribbean Development Bank (IDB). The Advisory Committee for Monitoring and Evaluation Network, which the Fifth Conference included Gladys Lopez Acevedo (LC- took place in Bogotá, Colombia, on Novem- SPP), Fernando Rojas (LCSPS), Manuel Fernando Castro ber 5 and 6, 2009. The conference topic was “Challenges (IEGCS), Diego Dorado (DNP), HeraldoLaguzzi (IDB), in Monitoring and Evaluation: An Opportunity to Insti- and Katia Rivera (IDB). The team acknowledges the sup- tutionalize M&E Systems.” The event was hosted by the port from Indu John-Abraham (LCSPP), Irene Clavi- Colombian National Planning Department (Departamento jo (LCSPP), Lucy Bravo (LCSPP), Pedro Arizti (LCSPS), Nacional de Planeación, DNP), with the support of the gov- Henry Forero (CITPO), Jaime Saavedra (PRMPR), and ernment of Spain. Elizabeth Tello (DNP). This publication includes three presentations from the We thank our colleagues from the World Bank and the Fourth Annual Conference, which took place in Belo Hor- Inter-American Development Bank who traveled and con- izonte, Brazil, on November 25, 2008. tributed to the conference and the speakers and partici- The Fifth Conference was organized jointly by the pants whose willingness to learn and share their experienc- World Bank’s Latin America and the Caribbean Region, es played an important role in the success of this event. The the Inter-American Development Bank, and the National presentations were submitted by participants at the confer- Planning Department of Colombia. Guidance for the orga- ence and later prepared for publication by Gladys Lopez nization of the Conference was provided by Louise J. Cord, Acevedo, Helena Hwang, Lycia Lima, and Katia Rivera. xiii Foreword E ffective policy making requires informa- The 2009 Conference was hosted by the Colombian tion on whether governments are doing things National Planning Department (DNP). The partnership right and whether they achieve the results with DNP was extremely fruitful for all sides. Country intended. Strong monitoring and evaluation ownership of the event was demonstrated by high Colom- (M&E) systems provide the means to compile bian participation, about 150 professionals and high gov- and integrate this valuable information into the policy cycle, ernment officials from DNP, the Colombian Ministry of thus providing the basis for sound governance and account- Finance, and the presidency. The Colombian government able public policies. The Latin American and Caribbean also took this opportunity to launch a national chapter of (LAC) Monitoring and Evaluation Network was estab- the M&E Network. lished by the World Bank and the Inter-American Devel- We thank World Bank and IADB colleagues who trav- opment Bank (IDB) (Program to Implement the External eled and contributed to the conference. There was an un- Pillar of the Medium Term Action Plan for Development precedented level of cooperation within the Bank to make Effectiveness [PRODEV]) to promote and strengthen the this year’s conference an international success. The World use of M&E systems in Latin America and the Caribbean. Bank-Independent Evaluation Group (IEG) and the Since 2005, the M&E Network has promoted knowledge IADB-OVE were active partners in the organization of the and a forum for discussion by practitioners in key aspects conference, providing inputs in terms of technical content of the institutionalization of M&E systems at national and and other resources. Also, a large number participants from subnational level. several sectors and regions (South Asia, Sub-Saharan Af- The Fifth Annual Conference of the Network took rica) attended the Conference. We are extremely positive place in Bogota, Colombia, from November 5–6, 2009. about this broad based and multiregional engagement. We The Conference gathered more than 350 practitioners, hope it will continue in the future and that the annual con- government officials, academia, and international organiza- ference is increasingly as a key knowledge event for M&E. tions (IDB, World Bank, the Food and Agriculture Organi- A virtual social networking platform was also launched zation, Comisión Económica de América Latina, RELAC) at the V Conference to enable the region’s M&E Network from more than 30 countries around the world, and an participants to share information on good practices, events, additional 300 people watched the event by Webcast. The evaluation methodologies, and other resources for M&E. evaluations from the conference indicated a positive feed- The World Bank and the IADB have played a catalyzing back from participants, with 93 percent of the respondents role in setting the platform, but the goal is for M&E Net- ranking the conference as excellent or good. Support from work members also to contribute to the dynamics of the the Government of Spain is greatly acknowledged for the Web site. Already more than 600 members have signed up organization of the Conference. in the social network. The social network and the confer- xv M&E SYSTEMS AND M&E TOOLS ence Web site featured 13 interviews of the leading speak- The fifth session analyzed the development of M&E ers from the Conference—Brazil, Canada, Chile, Colom- capacities and the alliances between the government, aca- bia India, Italy, Mexico, New Zeland, Peru, South Africa, demia, and civil society. Various presentations created the Spain, Sri Lanka, and the United States. The interview pro- right atmosphere for a fruitful exchange of ideas. vide a comparative overview of the main features of the The sixth session dealt with the institutional arrange- M&E frameworks developed in these countries. ments and policies of M&E systems to ensure quality, ac- cess, and use of information. The core portion examined Structure of the publication and of the ways to obtain reliable, clear, and timely information. conference The seventh session of the conference focused on the ex- The objective of the Fifth M&E Conference was to dis- change of information about the experiences and challeng- cuss challenges in institutionalizing M&E systems and us- es that surround the creation of a national chapter of the ing M&E information to support planning and budgeting M&E Network. Brazil presented their progress in establish- decisions as well as to enhance government transparency ing a national network. The lessons and recommendations and accountability. were useful for Colombia which launched their national The structure of the present publication follows the M&E chapter during the conference. structure of the Fifth M&E Conference agenda, which included seven sessions. The first session was devoted to Emerging messages from the conference the challenges facing evidence-based decision making: the The varied presentations highlighted a tremendous growth role of M&E. The analysis looked at experiences from in recent years in both the supply of and demand for quali- Minas Gerais, Brazil, Peru, the Mexican National Evalua- ty M&E initiatives. This growth is spurred on by increased tion Council on Social Development Policy (CONEVAL), demand for public sector effectiveness and transparency, Mexico, and Chile. The goal was to show experiences in by the increased need for information frameworks and sys- which M&E has influenced resource allocation and the tems to increase program quality, and on the supply side by modification, strengthening, or elimination of policies or the growing availability of M&E technologies and instru- programs. ments. Several countries have created high-level agencies The second session focused on institutional arrange- devoted to M&E (South Africa, Mexico, Colombia), and ments for M&E systems at the international level, and have strengthened their legal and regulatory frameworks to enabled participants to know, examine, and identify the require regular M&E and to ensure the public dissemina- advantages and difficulties of different institutional ar- tion of information (Mexico, Peru, Spain, India, New Zea- rangements used in government management in Canada, land, Chile, and Australia) and adopted innovative M&E Sri Lanka, Spain, and South Africa. tools to strengthen their budgeting and planning exercises. The third session addressed the institutional M&E ar- The Brazilian state of Minas Gerais is using benchmarking rangements in Latin America and discussed experiences in tools to better plan and monitor their development plan. the region, considering the characteristics, advantages, and Mexico introduced a Menu of Evaluations two years ago, disadvantages of M&E systems depending on whether they which consists of seven different types of evaluations (stra- fall under or outside the executive, and how central M&E tegic, impact, consistency and results, indicators, policy, systems relate to and complement their sectoral counter- and process), which has allowed a greater use of M&E in- parts. Presentations included Costa Rica, the United States, formation by Congress, the Ministry of Finance, and the Mexico, and a comprehensive look at the region. Presidency. The special session discussed the achievements and The growth in supply and demand presents tremendous challenges of the Colombian National System of Evalua- opportunities, but also underscores some critical challenges tion of Results-based Management (SINERGIA) in its 15 and priorities for the future. years of operation. The analysis considered its overall ef- First, and perhaps most important, is the need to more fects and the way it has influenced the formulation and consistently link M&E results to policy planning and bud- implementation of policies and/or programs in Colombia. get processes and thereby to ensure that the increased in- xvi foreword vestments in this area are sustained and used. This will re- Next steps quire institutional incentives as well as enhanced use of The network has been actively supporting three emerging methods, such as business intelligence, benchmarking, tar- national M&E networks. The Brazilian Chapter of the Lat- get setting, and quality data to ensure that the M&E in- in American M&E Network already has a large number of formation is integrated into the decision-making processes. members (http://redebrasileirademea.ning.com/) and is or- Second, it will be important to continue developing ca- ganizing a second annual meeting to be hosted by the Ban- pacity for M&E. Training is needed both for the users of co do Nordeste in June 2010. Discussions are under way to M&E information (Congress, government officials, legis- create a sectoral subnetwork in Brazil around health M&E. lation), as well as for its producers (National Statistics Of- Colombia launched their national network and first meet- fice, academia, and government officials). There are some ing. Mexico is in the process of setting up one. The ini- recent efforts to increase capacity through Regional Knowl- tial focus of these networks is on sharing knowledge and edge Centers (an initiative led by IEG) and to develop and promoting learning and capacity building on M&E among strengthen national M&E chapters (an initiative led by subnational entities. The network is supporting these na- LAC). tional chapters and providing training in various M&E tool. Third, to continue encouraging the growth of M&E We hope the enclosed publication serves to continue systems and their innovative uses for policy making, several fostering knowledge in M&E in the LAC Region and even speakers noted the importance of creating more carrots and beyond. Five years have passed since our first meeting held fewer sticks for implementing agencies to do M&E. For ex- in May 2005 and judged by the growing interest of partic- ample, the United States recently launched a new initiative ipants and the quality of the presentations, we can all be under President Obama to create a fund to encourage pro- pleased with the results obtained thus far. The Sixth M&E grams to undertake evaluations and to implement the rec- Conference will take place in Mexico City on August 25– ommendations from improvement plans. Efforts such as 27. This event will be hosted by CONEVAL, a regional these should be expanded. leader in the area of M&E and poverty monitoring. Fourth, while laws on access to public information have As suggested at last year’s conference in Bogota, Colom- been introduced in some countries in the region, most of bia, the theme for this year’s conference will be using M&E the best practice remains outside the region and there is a results in the public policy cycle. The conference will high- clear need to strengthen regulatory frameworks for public light specific sectoral policy uses for M&E results, as well as information access. the role of civil society and parliaments in producing and Finally, several speakers discussed the challenges of us- using M&E results. Two parallel pre-conference workshops ing M&E systems in weak institutional or low-income also will offer participants the opportunity to learn more countries or regional entities (states, provinces) where core about CONEVAL’s structure and activities, as well as a series building blocks for public sector effectiveness and trans- of seminars of good practice M&E techniques and tools. For parent governance are lacking (effective budget systems, the first time this year, the agenda reflects a regionwide call planning mechanisms, core data, and so on). Efforts to for proposals among the network members and beyond to understand how to promote data development and design identify many of the conference presenters. Hereby, we ex- useful M&E systems for fragile environments should be tend an invitation to continue working and partnering with supported. countries toward the institutionalization of M&E. Louise Cord Roberto Garcia Lopez Gladys Lopez Acevedo Sector Manager PRODEV Coordinator Senior Economist Poverty Reduction and Gender Unit Inter-American Development Bank Poverty Reduction and Equity Latin America World Bank World Bank xvii �Welcome Address I t is an honor and a privilege for us to have you data, and figures, for which the latest indicators are al- here. I hope you enjoy both the region and the city, ways available. and take new perspectives and news of Colombia Since we are in the last year of this government (which back to your countries. I would also like to thank has been in power for seven years), it is essential for the NPD the Inter-American Development Bank and the to evaluate a broad range of public programs. This process World Bank for their efforts to bring us together once a will contribute to maintaining and adjusting the programs year and keep us up to date on what is happening in other that are underway, regardless of changes in government. To parts of the world. ensure continuity and adjust and correct problems or flaws, This type of event is a fantastic opportunity to learn it is important to demonstrate both technically and objec- about the good and bad aspects of others countries’ expe- tively how each program works. One of this administration’s riences. In Colombia, we have learned a great deal from main priorities is to leave the incoming administration the our neighbors and we are obviously delighted to be able to products, data, and information to improve planning and share our experiences and challenges, and the problems we further incorporate the newly acquired knowledge. have faced in this and other areas. We have achieved enor- I am not an expert on monitoring and evaluation, so I mous progress in the area of monitoring and evaluation, will leave the technical discussion up to you but I would which is crucial to public policies, not only for the depart- like to take this opportunity to welcome you to Bogotá. I ments that implement them but also to ensure that this in- am sure that the meetings will be productive and enjoyable formation is available to society. The National Planning for all of you. Finally, I would like to thank Diego Dorado Department (NPD) is continuously uploading informa- and his work team and all of you for gathering this enor- tion to its Web site so that citizens can track and evaluate mous range of representatives from all over the world here the results of government programs, an essential aspect of today. modern democracy. We are also working to develop forms of verification Thank you to guarantee that programs are evaluated and that this information is useful to legislators and, above all, poli- Esteban Piedrahita Uribe cy designers. Colombia faces the challenge of having a Director General of National president whose management model is based on results, Planning Department, Colombia xix �Conference Overview T his is the fifth year that an increasingly Why are we here? large network of academics, public officials, Why are we here? In one way or another, we are all involved and experts have met to discuss, share expe- in public endeavors. We are here to improve people’s qual- riences, and reflect on the challenges and op- ity of life. That is the ultimate goal. To achieve this, we are portunities in this vast world of monitoring attempting to urge each one of you, in Colombia and in and evaluation (M&E). The first two meetings were held each of your countries, to use scientific procedures to im- in Washington, D.C., with more than 70 participants; the prove your decision-making processes, and to insert M&E next meeting was held in Lima; and then Belo Horizonte; tools into project cycles and policy cycles. How can we in- and now, Bogotá. The first meetings were dominated by sert our efforts in policy cycles? presentations and agendas defined by international entities. To do this, we first have to know what is important for Since we left Lima, we have taken steps to make this an people’s well-being. Is it to reduce infant mortality? Im- activity with ownership and participation by the communi- prove citizen security? Children’s malnutrition levels? In- ties interested in these topics. crease young people’s access to Internet? It is essential that This year, the network is taking yet another step. After we define the results that are the most important for taking successfully joining efforts with the João Pinheiro Founda- action. And our first challenge is to have reliable data that tion (JFP) and the Minas Gerais state government in Bra- will allow us to measure whether or not we make progress zil, we now have included the Colombian National Depart- toward these results. ment of Planning and Synergy as a partner, in an experience Once we have identified our critical areas of interven- that has been very productive. It is no accident that we are in tion, we need to know how much we should advance. Bogotá, nor was it when we were in Belo Horizonte. It is not For this reason many of you are in the business of es- just that the organizers were clever enough to bring us here to tablishing quantitative goals, and we are all involved in enjoy a fantastic city like Bogotá. We are here because things this. It is not the same to say that our goal is to reduce are happening in Colombia. In Colombia, interest in M&E poverty, as to say that we are going to reduce poverty can be found at the different levels of public administration, by 10 points in five years. This definition of quantita- in the various departments and municipalities, and in both tive goals, at all levels, is a key instrument that we use, the public and private sectors. And, despite the complex in- but we do not use it enough. The focus on vigorously es- stitutional challenges, the public sector has a commitment at tablishing goals entered the development world through various key points to strengthen planning, to establish goals, the Millennium Development Goals, but now it should and to carry out M&E to provide essential feedback to the be part of all planning and monitoring systems. Howev- design and formulation of public action. This conviction and er, and this is an interesting point, the establishment of commitment cannot be found just anywhere. quantitative methods is not as common as it should be— xxi M&E SYSTEMS AND M&E TOOLS in part, because it requires commitment and clearer ac- How to make progress in inserting M&E in the policy cycle countability. is a complex challenge, but that is why we are here. And this matter of goals is not an easy task. If we set a goal of reducing poverty by two points per year, is that a Monitoring and evaluation is increasingly lot or a little? If we attempt to reduce infant mortality from important 30 to 25 per thousand in five years, will this mean that our Policy discussions in Colombia and Latin America in gen- country’s performance will be excellent or only mediocre? eral have been gradually changing in recent years. There is What we call benchmarking in English—which makes it a clear consensus on the need for macro stability, low infla- possible to determine whether a certain performance is ade- tion, and openness to international trade. But there is al- quate, good or bad, by making a comparison with what has so consensus in many of our countries that huge gaps and been done in other countries or regions—is something that major inequalities persist between different groups of the we do very infrequently. Setting goals, and setting them in population in terms of access to services, access to opportu- an informed way, is complex, but it is the first critical step nities, and differences in access to good nutrition for chil- in M&E systems. dren, to a good education for young people, and to good Now, if we know what the goals are, we have a bet- jobs for adults. ter idea of what we should be monitoring. But to move The development of increasingly more efficient public forward with these goals, we need to implement policies policies is indispensable to closing these gaps in opportu- that work and that will be useful in reaching these goals. nities. That is why government efficiency is at the center of The problem is that we always have lots of programs and the discussion. And this points to the need to ensure that it is not always clear how they will contribute to achieving the methodological tools and institutional mechanisms our established goals. The link between the school break- that will be discussed here will contribute to the process of fast program and nutrition, or the link between the occu- designing and formulating policies. Only to the degree that pational training program and the employability of young we are able to move forward in this task of ensuring that all people, are two examples of processes, of functions that are the accumulated information generated by M&E systems not totally clear. We are using logical frameworks and some has an impact on policies will we know that we are doing other tools, and we have made some advances along these a good job. lines, but we still have a long way to go. But it is difficult to move away from speaking about the In order to identify the policies that will allow us to global crisis we are facing (which possibly we have recent- make these goals a reality, or to know whether the current ly been moving out of, although it has undoubtedly halt- set of policies are working, we establish evaluation systems, ed or at least decelerated what we were gaining in terms evaluations of different types that allow us to have a better of growth and poverty reduction). And this raises an addi- idea of what works and what does not. tional challenge beyond the structural challenges in public policy and fiscal policy. These challenges vary among Latin Does monitoring and evaluation matter? American countries and throughout the world. And this takes me to the last point, which is feedback. Does In some countries—unfortunately only a few—it is fis- the work of M&E that you are carrying out matter? Is it cally possible to increase spending. And M&E tools can as- useful for anything? It is useful if the decision-making pro- sist us in making this spending more effective, to ensure cess is going smoothly. Succeeding in making program eval- that funds are being used correctly and on the right things. uations and the information generated have an impact on In other contexts, we are already seeing the need to re- decisions regarding ways to expand, perfect, or close pro- duce spending. But how can we reduce spending, given grams is complicated and is the challenge faced by many of the needs of our populations, given the increasingly great- us who are here. A challenge for many of us is to establish er need to provide opportunities for all—a task in which forums for making evaluation results known, and to estab- the state has a critical role? Spending cuts require the use of lish institutional mechanisms through which finance min- a scalpel, not a sledgehammer or a guillotine. And this can istries and congresses can request and use this information. be done only if we know what works and what does not. “A xxii C o n f e r e n c e O v e rv i e w scalpel and not a sledgehammer,” President Obama said in under the leadership of the Departamento Nacional de Pla- his campaign. The M&E tools that we will be talking about neación (National Planning Department, DNP) in Colom- here are precisely that scalpel. bia precisely as we are meeting here. And, that is why we are going to be making a call to the IDB and the World Without information, we have nothing Bank to support the formation of national networks in oth- (capacity building) er countries. At the World Bank, we consider contributing to capacity And we are going to support the development of plat- building at all levels to be one of our main missions. Better forms that facilitate exchanges of knowledge and infor- M&E systems will make it possible for policy design and mation between members of this Latin American net- implementation to be based on scientific evidence of what work and members of national networks, with a much works and what doesnot. more aggressive use of the Internet and of new means to Although all of us who are here today agree on many promote exchanges among all of you. As an example, ev- basic principles, determining the appropriate M&E system ery single World Bank employee can be watching this for each specific country depends on local conditions. It event right now, in any World Bank office anywhere in would be fantastic to have a perfect model of the organi- the world. zation of an M&E system that could be transplanted from I want to close by expressing my appreciation to the one country to another. But that is not possible, and it is IDB, to Roberto García López and Mario Marcel personal- not feasible or appropriate. We know a single route is not ly, for their constant spirit of collaboration and for enthu- the answer. siastically maintaining this collaboration with the World That is why the central role of the Inter-American De- Bank for five years now. And, second, I want to thank the velopment Bank (IDB) and the World Bank is to help cre- DNP and Diego Dorado personally, as well as Director Es- ate and share knowledge to support the process of educat- teban Piedrahita, for their support and trust, and for their ing the critical masses of people who can help to build the hospitality in this marvelous city of Bogotá. best system in each country. In countries like Brazil, Chile, I also wish to thank the Colombian government, in gen- Colombia, and Mexico, advances have been made in this eral, and all of you who are fervently dedicated to this task process, but we must keep going, and the road is even lon- of improving policies in your countries. I want to thank ger in other countries. And this is the reason we have con- you for allowing us the opportunity to assist in this task. tributed to forming national M&E networks, such as the Brazilian network that was created in a meeting like this Jaime Saavedra Chanduvi one a year ago, and about which we are going to hear to- Sector Manager, Poverty Reduction and Equity morrow, and like the one that is beginning to take shape World Bank xxiii �Session 1: Challenges to Evidence-Based Decision Making: The Role of Monitoring and evaluation and Other Factors Monitoring and Evaluation System in the a substantial budget deficit that had existed since 1996,” Minas Gerais State Government: Aspects of in which “the state apparatus was being systematically ru- Management ined” (Anastasia 2006, 13). In describing the background to management shock, Anastasia(2006, 15) emphasiz- Introduction es how the Management Shock Project became “the main This paper describes the monitoring and evaluation (M&E) guiding principle underlying the Minas Gerais government system implemented in the state of Minas Gerais in 2007, program.” In other words, the project enjoyed support and based on the Management Shock Project of 2003–06. The the political will to implement it. This, in large measure, M&E system in Minas Gerais consists of (1) a planning proved essential to the successful implementation of the process (as an important element when determining priori- government’s measures. ties),(2) institutional alignment instruments(such as results In 2003, the Minas Gerais state government introduce- agreements, strategic management, and results commit- dresults-based management. The Neves administration in- tees), (3) benchmarks and strategic project goals and their troduced (1) the linking of planning and budgeting, (2) a final results (outputs and outcome indicators), and (4) the number of strategic projects(intensively monitored by a feedback process (an effort to allow for ongoing learning team of project management experts from the GERAES and reinvention of the modus operandi in public policy). project office), and (3) results agreements and performance evaluations. Foundations of the Minas Gerais model: The Minas Gerais planning process included “double Management shock 2003–06 planning—a strategy based on a dual approach, with short- The idea that Minas Gerais needed a management shock term strategies for the first year, while simultaneously work- emerged from diagnostic work by the Minas Gerais Devel- ing on a medium and long-term strategic plan” (Guimarães opment Bank (Banco do Desenvolvimento do Minas Gerais, and Almeida 2006, 44), and the linkage of planning and BDMG), which concluded that “any lasting reform of the budgeting to ensure that the strategy is viable in the short state apparatus implies a ‘management shock’ and a revamp- run(Oliveira, Almeida, and Guimarães 2006, 67). This two- ing of the social insurance system” (BDMG 2002, 219). pronged approach to planning can be described as follows: The expression “management shock” was first intro- The state strategy rested on two pillars: fiscal balance duced during the governorship of Aécio Neves (2003–06). (current strategy) and a development agenda (future strat- From the beginning of his term in office, Governor Neves egy) that justified before society the initial adjustments of set out to implement the Management Shock Project, in the reduction and rationalization of spending, administra- a context “very similar to that of most other states in the tive reforms and increase in tax revenues. (Guimarães and federation, namely a very serious fiscal situation, including Almeida 2006, 22) 1 M&E SYSTEMS AND M&E TOOLS Another important initial aspect of the management FIGURE 1: shock was the creation in 2004 of the GERAES Project M&E System in the Public Policy Cycle Office, attached to the Minas Gerais State Planning and Policy Management Secretariat (SEPLAG). The GERAES project Defining Policy Policy monitoring priorities formulation implementation and office began its monitoring activities by preparing 31 stra- evaluation tegic projects (4 percent of budget expenditures for 2004). What does the M&E Currently, the portfolio has 57 projects (11.5 percent of system consist of? budget expenditures for 2008). Managers of strategic proj- Indicators ects meet each month to present a status report. The re- Feedback Evaluations port describes the physical and financial situation of these What is this system for? projects. In her summary of what was accomplished during the Source: Adapted by the author from Barkenbus (1998). first generation of shock management, Renata Vilhena (2006) highlighted two dimensions of the policy: on the is used to focus on public management. The creation of an one hand, a fiscal equilibrium that ensured the budget’s ef- M&E system must not be understood as an end in itself. fectiveness as a management tool; on the other, moderniza- Rather, it must be linked to the public policy planning and tion and innovation in state public management by means management process. To originate public value, therefore, of instruments that assisted in alignment and personal de- the M&E system must be capable of bringing forth infor- velopment of staff and thus of institutions. Looking for- mation that buttresses the decision-making process leading ward, Vilhena also stresses the embryonic idea behind the to the attainment of government objectives. It is essential, second generation of shock management: of course, that this information be used in practice by de- cision makers. The great challenge we now face is to consolidate and institutionalize the process of transformation, It is pertinent, then, to ask “What does the M&E system in order to ensure it becomes irreversible. This consist of?” Put otherwise, “what are the system’s main must be done by going beyond the idea of fiscal components?” The structure of the system leads to the responsibility and including social and managerial second question, namely, “What is this system for?” The responsibility. In the ultimate analysis, this means answer is generic: to obtain feedback from the public a commitment to development results and ac- policy cycle. countability vis-à-vis society. (Vilhena 2006, 354) Planning and setting priorities The foundation was thus laid to build a management The planning process in Minas Gerais can be differentiat- model geared toward the achievement of outcomes that ed from other Brazilian states. According to Article XX of benefit society, known as the State for Results Program the State Constitution, the Integrated Minas Development (Programa Estado para Resultados). Such a concept needed Plan (Plano Mineiro de Desenvolvimento Integrado, PM- a strong M&E system, capable of generating reliable infor- DI) is a mandatory document crucial to strategic planning. mation. The capacity and importance that a particular administra- tion ascribes to government’s planning function stand be- Monitoring and evaluation system tween the formal obligation and the concept of real strate- Figure 1 illustrates the system’s logic and the two questions gic planning. Starting in 2003, this long-term development that an M&E system should ask. An M&E system consists plan took on a real role in demarcating government action of a set of government performance indicators that are ver- and, in 2007, with the development plan review having ified by a series of evaluations of the government’s public taken place, the document’s function as a goal-setting tool policies. Based on these inputs, results-based management that reflects the results the government strives to attain was is made possible—meaning that the information collected strengthened. 2 S ession 1 : C hallenges to E vidence - B ased D ecision M aking The 2007–23 development plan includes the results ar- entire procedure of creating and determining the strategy, eas, or “target areas,” on which the government’s best ef- discussions take place and morally binding agreements are forts and resources are focused as they work to achieve the entered into with sectoral agencies, thus lending legitimacy desired transformations and results. This introduces the to this implicitly causal model. important idea of final results in public administration. A Once top management at sectoral agencies agreed to the total of 13 such results areas were created (see Figure 2). priorities upon which to act, an important meeting took �Aligned to the development plan, the Multiyear Gov- place in February 2007, which resulted in a series of mor- ernment Action Plan (Plano Plurianual de Ação Governa- ally binding agreements. The governor presided over the mental, PPAG) is bolstered by the linkage between plan- event, accompanied by top-level officials of the state exec- ning and budgeting. It is presented as a document that utive branch. “Commitment Notebooks” were distributed determines the portfolio of structuring projects as strate- and agreements were reached by each secretariat to attain gic government initiatives. Taken together, these two doc- certain goals and projects. uments signal the strategic objectives to be pursued (the These initial commitments were fundamental to align- development plan) and how it is thought these might be ing shared objectives. Once these objectives had been deter- achieved (the action plan). Put another way, an implic- mined, it followed a process of design and detailing of stra- it causal chain underlies each of the results areas. A fun- tegic projects that made up the new strategic portfolio (57 damental point in this process is that it is not led auton- projects) under the leadership of the GERAES Project Of- omously by SEPLAG. Quite to the contrary, during the fice along with the respective project managers. FIGURE 2: Strategic Map, Minas Gerais State Government MINAS GERAIS: THE BEST STATE IN WHICH TO LIVE MINAS GERAIS INTEGRATED DEVELOPMENT PLAN 2007–2023 Comprehensive Investment Competitive Network of Equity and Environmental perspective on and business territorial cities well-being sustainability human resources integration QUALITY AND INNOVATIVE PUBLIC MANAGEMENT STATE FOR RESULTS PROGRAM RESULTS AREAS Quality of Investment and value Development of northern Environmental education added to production Minas Gerais, quality FISCAL QUALITY Jequitinhonha, Mucuri and Rio Doce Youth Technological Social leadership innovation and defense quality Development of northern Logistics of Minas Gerais, Healthy Jequitinhonha, Mucuri Network of living integration and cities and services development and Rio Doce END USERS OF PUBLIC POLICIES Educated, healthy Dynamic Youth Safe and Equity among and qualified and innovative leaders well-kept cities persons and regions persons companies Source: Integrated Minas Development Plan 2007–23. 3 M&E SYSTEMS AND M&E TOOLS With the guidelines and policy design defined, under- balance. The Health Secretariat spends 30 percent on final girded by this implicit causal chain, implementation of results, 30 percent on structuring projects, 30 percent on these public policies was ready to begin. A basic premise the sectoral agenda, and 10 percent on expenditure stream- of the M&E system in Minas Gerais is that of supervised lining indicators. autonomy, in which the planning unit signs the objectives A major positive incentive links the logic underlying the and supports the sectoral agencies in laying down the path results agreements. The outcome of the evaluations (first by which to reach the desired goals. However, it is up to and second tier) determines who receives the productivi- each one of the sectors to find the best possible solutions for ty prize of up to one month’s wages. The prize is linked to implementing the projects. Thus, implementation in the the proper implementation of works and the attainment of M&E system devolves on incentives for implementation. results by the state secretariats and their work teams. This These are detailed in the next section. instrument, created in 2004, became more relevant to the M&E system starting in 2007, when it was linked to the Incentives for implementation: Results strategy indicated by the budgetary process and the four agreements and results committees themes. Based on the premise that the best road to results-based A fundamental role of the State for Results Program management is to generate an adequate set of incentives to (created in 2007 in close partnership with SEPLAG) is to implement the strategy, two instruments stand out: results align the partnership contracts with the strategy, and more agreements and results committees. Results agreements are important, to carry out evaluations that harmonize and- management partnership contracts entered into by the gov- balancethe different sectors. Taking as a point of departure ernor and the state secretariats (first-tier agreements), and the principle that some areas are more strategic than oth- between state secretariats and their work teams (second- ers, and the fact that there are different levels of difficulty tier agreements). Keeping in mind the strategic vision and in achieving the desired results, the State for Results Pro- desire to strengthen the importance of targeted actions, as gram endows the evaluation system in these contracts with well as the notion of supervised autonomy, the M&E sys- simple and transparent rules that lead to a solid and last- tem spends more energy on the first tier, and leaves the ing system. respective sectoral agencies in charge of second-tier agree- Two rules drawn up by the State for Results Program to- ments. gether with SEPLAG illustrate this spirit: This instrument, in turn, is an important tool in the ef- fort to align the institutions with the government strategy, • Create a bonus for those secretariats with the four insofar as it brings together the main results and deliver- largest budgets and a discount for the six secretari- ies to be pursued by the different sectors under one con- ats with the smallest budgets (Decree 44.873/08). tract. These documents reflect the final results sought (in- This makes it easier to balance these differing de- dicators for outputs and impacts with goals), the strategic grees of difficulty, given that more time is spent and projects (100 percent implementation of frameworks and greater audacity is called for when setting goals for goals planned), the sectoral agenda (set of outputs to be large systems (transport, education, health, and so- achieved), and expenditure streamlining indicators (savings cial defense) compared with the smaller systems goals with costing of activities and means of implementa- (culture, sports, advocacy, audits, agrarian reform, tion, as well as quality indicators in budget execution at the and tourism). agencies). • Draw up permanent yet flexible rules to award bo- Each secretariat has a contract with a different corre- nuses for meeting the goals and adjust goals that lation of weights among these four themes, depending on turn out to be unreachable. This is necessary be- the features of each system. The Education Secretariat, for cause of cases in which goals are agreed on for example, directs 70 percent of the contract to reaching the year t + 1, without evaluating the results for year final results, with the remaining 30 percent going to the t. This flexibility strengthens these contracts (De- other themes. Other secretariats, however, show greater cree 45.191/09). 4 S ession 1 : C hallenges to E vidence - B ased D ecision M aking In addition to the alignment generated by means of model (to be evaluated annually through the results agree- the positive incentive contained in the results agreement ments). After two rounds of evaluation of the results agree- and the subsequent awarding of a prize for productivity, ments (2007 and 2008), and having reached consensus on the results committee is another important instrument. a set of 104 indicators for the 2009 agreements, this M&E The State for Results Program’s committee meetings, pre- tool is fully implemented. The 2009 Indicators Notebook sided over by the vicegovernor, deal mainly with strate- was a benchmark for this process (Guimarães and Cam- gic projects by results area. Therefore, they are attended pos 2009). by those in charge of such projects (managers and top-lev- The evaluation logic verifies final results and represents el administrators at the responsible agencies). The objec- an important benchmark in the system’s construction. This tive is to systematize the vicegovernor’s administration of approach goes hand in hand with the premise that sector government activities. At each meeting, an operational- agencies must be aligned with the stated objectives and seek strategic plan of action with a focus on policy implemen- the best possible path to reach their goals. tation is drawn up. These meetings set the pace and keep One successful example of this logic is the final result the government’s strategic planning going, thus mitigat- “citizen’s fear of falling victim to violence”. After the Secre- ing the risk that operational routines end up dominating tariat of Social Defense and the Secretariat of Government the agenda. (charged with communications) agreed on this indicator, These committees, then, are concerned with imple- the related goals (such as a reduction in the proportion of menting the strategy, insofar as “concrete experience pro- the population that feared falling victim to an act of vio- vides clear evidence that, to be effective, it is not enough lence) generated activities to disseminate information on that the government’s strategy be well-formulated. It is also preventive actions and the struggle against violence already important to watch after its introduction and implementa- underway. This joint plan of action is the public manage- tion with the same care and even greater intensity” (Porto ment model, in which all actors coordinate activitiesto at- and Silveira 2006, 67). tainthe agreed-on results. These meetings involve a significant moral incentive, The idea of creating a harmonious M&E system in- given expectations and the vicegovernor’s accountabili- cludes a variety of approaches and results that coexist in ty regarding the implementation of the planned strategy. equilibrium. In addition, the notion of a model in which Furthermore, such forums permit participants to analyze strategic goals are correlated to more operational execu- difficulties, risks, and opportunities when implementing tion methods is strengthened (strategic projects and sec- projects, as well as generating support for sector agencies toral agendas). This allows for an evaluation that not only undertaking certain actions. verifies the effort made by sector agencies, but also strives In brief, in the sphere of implementation, two impor- to be fair, taking into account the externalities assumed tant instruments are intended to encourage and in some for some goals (a reduction in the homicide rate, for ex- measure monitor and evaluate the execution of public pol- ample). icy. The next section describes two important tools of the Although monitoring of final results was implement- M&E system: the final results (a theme selected in the re- ed, the challenge of improving the maturity and quality sults agreement) and the development of evaluation of of these indicators persists. In some results areas—qual- these public policies. ity education, for example—a set of results exists that is rather solid in terms of measures taken; in others, such Results-based management and M&E as the network of cities and services, the indicators nec- One of the main innovations introduced to the 2007–2023 essary for gauging some of the wished-for dimensions do PMDI deals with the creation of a set of final results with not yet exist. short-, medium-, and long-term goals in the 13 results ar- One innovation still underway concerns the creation of eas. During 2007 and 2008, the State for Results Program benchmarks for these indicators, as well as the development launched an effort to strengthen the M&E system and es- of methods capable of suggesting goals that, while challeng- tablish a framework for the results-based management ing, are at the same time feasible to attain. 5 M&E SYSTEMS AND M&E TOOLS Another tool in the Minas Gerais M&E system is the meeting, the Results Committee in the Quality Education impact evaluation of strategic projects. In 2007, when the Area brought up for discussion the results of the 2008 pro- new project portfolio was conceived, it was taken into ac- ficiency exam in Math and Portuguese taken by students in count that it would be necessary to develop evaluations ca- public schools. The evolution of pertinent indicators was pable of testing the implicit causal models. Thus, the state described, along with the performance of the indicators in government’s strategic partners (the Minas Gerais Federal relation to short- and medium-term goals. University and the João Pinheiro Foundation) were asked In brief, a space was created for the State Education Sec- to evaluate nine social projects, with the aim of learning if, retariat to explain the efforts being made in the early years in fact, the actions implemented were achieving the intend- of schooling, and to signal problems faced in middle school ed impacts. This project evaluation culture is still embryon- that explain why the indicators have stagnated. The Educa- ic in Minas Gerais, and the first results of these evaluations tion Secretariat stated that, in addition to measures already will be available in late 2009 and mid-2010. underway, it will take corrective actions, such as reviewing The third wave in the consolidation of this M&E sys- the curriculum, that are thought to overburden students at tem will introduce public policy evaluation tools in their this stage of the learning process. various stages (targeting, efficiency, process, impact, and This example hopefully will elucidate what is expect- so on). One part will be introduced toward the end of the ed in terms of results-based management and making sec- current administration’s period (2007–10), by means of toral agencies responsible for developing new strategies. “executive evaluations.” These evaluations will look more In this context, the role of the State for Results Program closely at the elements still lacking before project plan- is to monitor and deepen some of the evaluations cur- ning processes become definitively rooted in a culture of rently underway. In this particular case, these refer to the evaluation. strategic project for vocational training at middle schools, As it stands, the evaluations have yet to entirely fulfill whose goal is to improve education among middle school their role of feeding back into the public policy cycle. What students. does exist, however, is a clear understanding of the need to Nevertheless, other activities also are being carried complete this M&E cycle. The final section of this paper out by sectoral agencies as they seek to identify correc- traces some of the paths being taken to strengthen the feed- tive action plans. This is the case, for instance, of the back process. Transport Secretariat. Upon realizing that the proposed final result (“number of accidents per 100,000 cars”) Feedback: The current paths was overly ambitious, a workshop was held in Septem- It is up to the sectoral agencies to find the best ways to ber 2009 at which a Plan of Action was drawn up to help reach the agreed-on objectives. When these have been de- improve results. termined, implementation can be monitored and support- Other examples of even more systematic interventions ed. Thus, the feedback process must come mainly from the include the “accompaniment cell” for the Pedagogical In- sectoral agencies. In the light of indicator performance and tervention Program. This small working group focuses on the outcome of the evaluations, policy design and imple- analyzing the results of the PROALFAexam, which is a mentation may be reviewed, if necessary. reading and math proficiency exam administered to stu- The planning process cannot be rigid but rather should dents in the third year of primary school. Results range remain open to changes in frameworks and goals. As long between 0 and 800 points. A score above 500 indicates as a plausible rationale exists, the attitude toward such ini- adequate literacy skills. The goal of the accompaniment tiatives is open-minded, mainly during the annual plan- cell is to help the management team improve its inter- ning reviews. ventions. Inputs for this process of reflection usually have been Despite the foregoing, feedback as a systematic activity offered at the meetings of the results committees, includ- based on evaluations and final results has not yet been ful- ing, even more important, the delivery of strategic reports ly implemented in the state government. Part of the diffi- for discussion purposes. For example, at the May 4, 2009, culty resides in the fact that indicators for some of the re- 6 S ession 1 : C hallenges to E vidence - B ased D ecision M aking sults areas are still immature; another hurdle is the incipient tonomous entities, and the electoral body for the 2007 and degree of consolidation of an evaluation culture in Minas 2008 terms. Gerais. In 2005, the index created a special methodology for Notwithstanding these challenges, some good exam- evaluating special state entities, such as state-owned indus- ples already demonstrate how the introduction of a culture trial and commercial companies and public-private enti- of results-based management has had a positive effect on ties. In 2003, the Departmental Transparency Index began the identification of good practices. It can be observed how to evaluate government departments and agencies (Minis- the building of a well-balanced, harmonious M&E system, tries of Finance, Health, Education, Public Works and In- bolstered with appropriate incentives, can lead to a gradu- frastructure, Traffic and Transportation, Comptrollerships, al change of culture in the public sector as it moves toward and Assemblies of the 32 departments). Lotteries and li- an effective results-based management. quor stores also were monitored. The last measurement, of 2005 and 2006, suspended the evaluation of the De- Tadeu Barreto Guimarães partmental Assemblies because the evaluation tool did not Eder Campos show other kinds of risks that do affect these entities. Im- portantly, in the case of national- and departmental-level indexes, Transparencia por Colombia defined the coverage Transparency Index of Public Entities: of the indexes and took into account the characteristics of Experience in Evaluating Risks of Corruption entities in terms of their functional importance and bud- in Government Administration at National, get-handling levels. Departmental, and Municipal Levels In 2005, the Colombian Federation of Municipalities, the National Planning Council, and the Corporación Trans- Background parencia por Colombia formed an alliance with citizens to Since 2002, Corporación Transparenciapor Colombia has led jointly launch the Municipal Transparency Index. The al- the Government Transparency Index (formerly the Gov- liance called on the municipalities, with the exception of ernment Integrity Index), which is a regular, systematic large cities, to participate in this measuring exercise and study that aims to strengthen public sector institutions and obtained answers from 146 municipalities that voluntari- encourage measures to control the risk of corruption in the ly joined the process. In addition to measuring transparen- public sector. It is the first exercise to quantify risks of cor- cy and institutionalism, this index pays particular attention ruption in state institutions conducted by an independent to accountability to citizens and promoting civic participa- organization that monitors the risks of corruption from a tion in municipal administration. The first measurement civilian standpoint, and strengthens the definition of pub- of firms was conducted in 2007, and evaluated the 2005 lic policies in combating corruption. and 2006 periods. The risk of corruption of 252 munici- With financial support from the Inter-American De- palities was evaluated with regard to centralized administra- velopment Bank, the Dutch Embassy, the German coop- tive management. eration agency GTZ (CERCAPAZ Program), the British Embassy, U.S. Agency for International Development, the The fight against corruption as a starting point European Community, and its own resources, Transparen- Like poverty, corruption often is perceived to have no per- cia has produced and published five editions of the Nation- manent solution; its impact on society might possibly be al Transparency Index, four editions of the Departmental “reduced,” but it will never disappear. Many people, how- Transparency Index, and three editions of the Municipal ever, say that not only is it possible to combat corruption, Transparency Index. but also it is necessary. Ethical, moral, and political forc- Initially, the Corporation focused its efforts on produc- es demand fundamental changes in public administration ing the index of national public entities. The first edition and management processes. It is of the utmost priority for evaluated 88 entities in 2002. The most recent edition eval- government and citizens to take action against corruption uated 158 bodies of the three branches of government, au- because in a country with high poverty levels, the people 7 M&E SYSTEMS AND M&E TOOLS should have access to public goods and services—to which institutions (and the practices of the individuals who man- they have full right. age those institutions—civil servants) to identify the con- ditions and practices that can create a risk of corruption in Risks of corruption, and transparency of government. The analytical category of risk of corruption government entities is understood by the index program as the possible occur- One of the core values of modern government manage- rence of acts of corruption in public institutions because of ment is democracy as a basis for exercising power. Democ- the existence of certain institutional conditions and certain racy as the genesis of power and its rules constitute the po- practices of employees that the evaluations, developed by litical and ethical guide that organizes the relations between Transparencia por Colombia in 2002, have shown to be sus- civil society and state. The rules of democracy to remember ceptible to corruption.1 include consensus, controlled power, accountability, legali- Risk is a concept under construction that stresses the ex- ty, and access to information, among others. All these rules istence of an intermediate state between security and de- are aimed at generating a space of trust in the relationship struction. Pinpointing this state alerts people to the “per- of social and political actors, trust that guarantees the exis- ception of risk” and defines their thinking and actions. tence of society as a whole, although not in the hypotheti- Risks of corruption in public institutions do not exist until cal idyllic picture of harmony, but in a context of diversity an inquiry, investigation, or evaluation brings them to the and pluralism in which tolerance and respect make coex- attention of public servants and citizens and, most impor- istence possible. Trust in diversity is based on the assump- tant, when this new information generates actions to coun- tion that all the actors who live in society develop their role ter the risk. Our primary purpose in evaluating risk levels of within the framework of law, while the state as the lead ac- the entities is to promote institutional changes as a result of tor represents the institutional framework that ensures the changes in the ideas and attitudes of public servants, specif- integrated management of public affairs, where collective ically in relation to management of the public. Obviously interest prevails over private interests. the index is a civilian contribution. It is one among several Viewed in this light, transparency appears as a medium government-modernization and democratization initiatives that allows the different members of society to know what to emerge from the government, from teaching institutions, the government is doing. Its every action is of public inter- other civilian organizations, and international cooperation. est, which is why the government has the utmost respon- The Transparency Index evaluation is based on the as- sibility to provide information about its actions, and why sumption that corruption risks are reduced when man- it must also be willing to submit such information to pub- agement is transparent and abides by the established rules lic scrutiny. Such scrutiny may result in society’s apprecia- and standards, and when local authorities promote civ- tion or sanction, depending on how well the power holders ic participation and social control. From this perspective respond to citizens’ demands. Thus, transparency in gover- and as a starting point for the evaluation, the following are nance is one of the favorable conditions (coupled with in- risk scenarios: stitution building, political skill to manage social demands, a high level of trust, and the existence of social, political, • Monopoly of information and decisions, which af- and economic dialogue scenarios) that allow for public de- fect the population in general liberations. These conditions are crucial for governance. • A high degree of discretionary behavior by civil ser- Corruption is one of the most serious problems affect- vants ing the democratic exercise of power. It deforms democra- • Poor development of institutional processes and cy, especially if it becomes embedded in the social, politi- procedures cal, and economic fabric of society. Not only is corruption a • Absence of or weakness in checks and balances and phenomenon that speaks for the actions of individuals, but controls of power, and institutional controls—in- it also constitutes a nefarious feature of the “architecture” of ternal and external—and control by citizens government in all periods. The Government Transparency • Zero or near-zero transparency of civil servants’ ac- Index focuses on the architectural characteristics of public tions 8 S ession 1 : C hallenges to E vidence - B ased D ecision M aking In these risk scenarios, public affairs stops being public from those offices in allegations against staff is valued as an and becomes private due to the existence of management important internal control mechanism. processes and governance that are shrouded in secrecy. For this reason, the evaluation focuses first and foremost on the Institutionalization search for greater transparency in the activities of govern- The indicators that assess compliance with legal standards, ment personnel. That is to say, government action should guidelines and policies, and procedures established to be visible, governed by clearly established rules and proce- achieve the institutional mission are based on the premise dures, and subject to public scrutiny. The formula is sim- that compliance with such standards, guidelines, and pro- ple: the greater the transparency of public entities’ process- cedures limits the discretion of officials, politicians, and in- es, the lower the risk of corruption and therefore the lower dividuals in the exercise of public functions, and, thereby, the occurrence of acts of corruption.2 reduces the risk of corruption. In addition to these three factors, the National Trans- Measurement structure parency Index evaluates the corporate governance factor for Index results are not indicative of the level of corruption in government-owned industrial and commercial enterprises an institution or sector, but rather of the corruption risks and for mixed economy partnerships. Here, an evaluation that entity faces based on its administrative management. is made of the existence of formal principles and manage- Consequently, these results provide an input that enables ment processes based on transparency, corporate responsi- entities to identify the areas and determine prompt actions bility, and respect for the rights of shareholders and stake- to counter the corruption risks identified. The index iden- holders. tifies corruption risks in different core factors to be mea- The Municipal Index incorporates the civic participa- sured, each of which is composed of multiple variables and tion factor, whereby an evaluation is made of all govern- indicators. The factors, in the case of national and depart- ment initiatives that promote citizens’ involvement in pub- mental indexes are transparency, punishment, and institu- lic affairs to influence decisions (on services, definition of tionalization. priorities, spending, and so on) and exercise control over the performance of government authorities. Transparency The indicators under this factor evaluate permanent expo- Purpose and scope of measurement sure of information about administrative management of In terms of knowledge production, the index has estab- the entity. They evaluate the quality of the information and lished itself as the first independent instrument of civil so- ease of access. Thus, they show whether management ren- ciety that periodically measures the performance of public ders the procedures by which it operates transparently, and entities in the prevention and control of their risks of cor- whether its results are submitted to outside scrutiny by the ruption, and it enjoys great credibility in public opinion general public or private stakeholders. and in the institutions. The design of this tool has made some progress at the Punishment national and departmental level, and has made notable The indicators included in this factor examine judgments progress at the municipal level. Today, the Transparency In- and sanctions—disciplinary and fiscal—related to behav- dex’s objective indicators cover current issues in the anticor- ior considered corrupt by our legislation. Going by these ruption agenda. They include openness of information on indicators, judgments against officers of a particular entity, government activities and decisions, accountability to cit- issued by the supervisory bodies in cases of fiscal or disci- izens by governments, and publication of recruitment and plinary liability, are seen as a warning that the entity’s inter- selection and evaluation processes for public employment, nal processes are not sufficiently shielded or protected from among others. significant risks of corruption. The indicators also note ad- The frequency of measurements has converted the index vance action by internal disciplinary control offices, but into a benchmark for institutions to intensify their institu- from a different perspective, in this case a greater response tional analysis, while allowing them to eventually compare 9 M&E SYSTEMS AND M&E TOOLS BOX 1: Basic guidelines for collection, review, and appraisal of evaluation data • The index uses objective information obtained directly from the monitored entities and secondary government sources and control bodies. The entities appoint within their organization an official spokesperson in the data col- lection process, to centralize requests for information from the entity, and delivery responses. • Information from the entities is obtained through two processes. In the first, the entities complete a questionnaire designed by Transparencia. In the second, they review information published on-line and in citizen service lines. • Transparencia obtains information directly from other public sector sources, which is required to construct the in- dicators that form the index. • The information is consolidated in a single database and processed to calculate the indicators and the index for each monitored entity, with formulas defined in the methodological design. • Data validity filters are applied. The calculations made and checked within Transparencia are sent out to the enti- ties as preliminary results for their information prior to publication. • All of the entities’ doubts about processing and calculation are clarified, and any observations about possible mis- interpretation of any data will be heard to incorporate the corresponding adjustments. • The results are published. Source: Transparencia, www.transparenciacolombia.org.co. themselves with the rest of the world and with their own • The issue of transparency to fight corruption has progress over time. The index is a yardstick for public opin- been incorporated into the agenda of a large number ion about the evolution of institutions in controlling cor- of entities. The proof is that 70 percent of assessed ruption risks. As a tool that promotes improvement pro- entities deliver information in a timely manner. cesses in the management of public entities, the index seeks • The research team has met with more than 180 en- to capture their interest in identifying corruption risks and tities nationwide and more than 1,200 employees, to encourage the design of improvement strategies. Over becoming a tool of consultation and management the years, the index’s results have become a reference tool for public officials. Entities have received warnings for institutions to improve each indicator, raise perfor- about their institutional weaknesses and have de- mance in the graded factors, and grow more self-confident veloped improvement plans guided by the results because they are monitoring their corruption risks and the of the index. effectiveness of the institutional steps to minimize them. • At the national level, and in some organizations at Below are some of the elements that indicate this: the departmental and municipal levels, the index has been connected to the strategic plans of the en- • The measurements made in 2005, 2006, and 2007 tities. It has been integrated as an indicator of qual- highlight a trend of improved scores for most of the ity management systems and institutional develop- entities monitored at the national and departmen- ment in relation to transparency. tal level, particularly in aspects related to the “pub- • Nationally speaking, the index has marked im- lic exposé” of information concerning the manage- provement processes in such areas as information ment of the institutions and the increased public published on Web sites, announcement of contrac- hearings of accountability to citizens. In the 2007 tual processes, and accountability to citizens. measurement, 75 percent of the entities held a pub- • The entities that are sources of information have lic hearing of accountability. contributed significantly to the process of strength- 10 S ession 1 : C hallenges to E vidence - B ased D ecision M aking ening the index. The management teams of the Gen- coordination work with Transparencia por Colombia, the eral Comptrollership of the Republic, the Attorney Vice Presidency, the National Department of Planning, General’s Office, and the Accountant General’s Of- and the National Bureau of Statistics have shown inter- fice have shown a timely response in providing their est in facilitating and enhancing the availability of public information. Likewise, the Attorney General’s Of- information so that it will strengthen the index and oth- fice and General Comptrollership have been work- er gauges of governance, and boost their use in defining ing toward improving their information systems and public policies. fulfilling their monitoring and control functions. Some evidence shows certain interdependency between • The National Transparency Index has become one the index results and measures taken by the government to of the targets of the Information System for Moni- obtain stronger institutions and transparency. Thus, based toring Government Action (ISMGA). The Nation- on the results of the index and the possible adjustments al Vice Presidency, headed by Dr. Francisco Santos, and improvements in public institutions that they trigger, has proposed for the next measurement that nation- the government and the vice president have publicly com- al entities should aim to obtain at least 90 points mitted to concrete measures to entities that have ranked in in the index. The analyses and recommendations the ranges of high and very high risk of corruption. In addi- of the national and municipal indexes, delivered in tion, some departmental comptrollers have used the results December in two publications, have resulted in an obtained by some departmental entities to define strategies interinstitutional meeting and a meeting with civil- and policies to support their monitoring function. ian stakeholders on issues of participation and con- In terms of public opinion, the index offers a reference solidation of reports from local authorities. Both to the evolution of public institutions and promotes in- meetings were coordinated by the Vice Presidency volvement by other stakeholders in the field. To date, pub- and the Presidential Program to Fight Corruption. lications of the index have been well received because of the • In response to the index’s poor results, the Presiden- independent view it represents. The results of its applica- tial Efficiency, Modernization, and Anti-corrup- tion have been disseminated widely in the media, thereby tion Program proposed that corporations subscribe increasing both its visibility and public pressure on institu- to transparency pacts, seeking to engage these enti- tions to participate in the measurements and to implement ties via voluntary agreements on issues like recruit- improvements. ment, meritocracy, and information systems to en- sure their integrity. It will also take the results of the The challenges of measuring Municipal Transparency Index and include them as All these advancements need to be consolidated and pro- a substantial part of the Transparency Pacts, which jected to new dimensions that increase their potential im- this entity endorses with the municipalities. pact. Reducing corruption and corruption risks is not an end in itself but a necessary step toward more effective, fair, In public policy, the index provides a benchmark to an- responsible, and reliable government. Corruption is a ma- alyze the impact of public policy to combat corruption in jor concern because of the effect it has on society and in- the management of the entities and to generate recommen- stitutions. dations for public policies aimed at improving institution- The effects of corruption can spread across all sectors al control and transparency. The variables covered by the in- and areas of public and private life. Corruption is a threat dex are related mostly to the policies pursued and the current to democratic institutions and democracy itself. For this regulations aimed at reducing the risks of corruption, so that reason, it is crucial to increase levels of transparency and to their periodic measurement reveals, at least partially, that the prevent and combat corruption; and, in this way, to help changes produced in the management of the entities are ex- increase effectiveness and accountability and to consolidate plained by the attention paid to such policies and standards. and increase public confidence in democratic institutions The Transparency Index enjoys credibility with pub- and governance. In this vein, the Transparency Index will lic entities and the government. In fact, following advance continue working in the following areas: 11 M&E SYSTEMS AND M&E TOOLS • Continue the work of measuring the three levels acute poverty and exclusion of large sectors of the popula- of government using a tool to detect the transfor- tion, coupled with the presence of an armed conflict, more mation of the corruption phenomenon and institu- transparent, efficient, capable, democratic, and reliable local tional development of organizations. governments are required that can ensure better governance. • Strengthen the analysis of the results obtained From this perspective, the Municipal Transparency In- through the inclusion in the program of lines of dex seeks to make Colombian towns a cornerstone of the qualitative analysis. state structure. And it is betting on improving the insti- • Increase opportunities for discussion with evaluat- tutional conditions of transparency as a formula for se- ed entities in such a way as to make in-depth insti- curing the welfare and development of communities and tutional adjustments in those entities. achieving better local governance. The systematic exposure • Increase opportunities for promotion and dialogue of information to teams, government, and public opinion with representatives of social, political and finan- about corruption risks associated with institutional condi- cial organizations, and with international cooper- tions and practices within the municipalities will motivate ation on the results of analysis (both quantitative them to improve, as has happened with national entities. In and qualitative) to provide information and create controlling risks of corruption, we expect to break one of public awareness on the importance of transparen- the strongest barriers that prevent local governments from cy in government management. achieving greater levels of efficiency in addressing citizens’ • Develop steps to promote adjustments and im- demands. And by analyzing a significant number of munic- provements in government policies to guide the ipalities of varying size (population and fiscal) and in dif- modernization of the state and the fight against ferent regions, it will be possible to deduce concrete policy corruption. recommendations for public transparency and the demo- cratic and institutional development of Colombian mu- Conclusion nicipalities. To encourage these changes, a medium-term In general, Colombian municipalities are on the frontlines intervention is needed to keep up the level of continuous in the fight against corruption. Their institutional capaci- analyses and propaganda, institutional improvement, and ty, however, is weaker than the central government’s and in- favorable public policies to form a virtuous circle of trans- adequate to deal with the scale of the problems and citizen parency, effectiveness, and confidence. This is the purpose demands they must address. This situation proves their low of the Municipal Transparency Index. degree of effectiveness and legitimacy. Recent analyses also The road to the consolidation of the state as honest, ca- warn about a rise in corruption at the municipal level, adding pable, and legitimate is long. Measurement of corruption that this phenomenon has become a serious obstacle to in- risks implemented since 2002 can achieve the greatest gains creasing equity, prosperity, and local democracy. Given these in the promotion of internal institutional adjustments and circumstances, some voices even argue for the need to reverse advocacy on issues of transparency and risk control of cor- the process of decentralization, which is just two decades old. ruption. Measurement also has received the attention of The decentralization process is not responsible for cor- government agents and public authorities. ruption, although it has highlighted this phenomenon. To In this sense, the gains and improved overall perfor- the extent that decentralization has brought power and re- mance of the index should not divert attention from areas sources to municipalities, it has created an attractive oppor- in which adjustments are still necessary. Decisive action is tunity for corruption, which has become a barrier to effec- needed in the formulation of public policies against corrup- tive attention to the social demands of citizens who expect tion and against those that compromise administrative pro- them to be solved. Undesirable or perverse effects of de- cesses. Although much progress has been made in a number centralization are explained largely by the degree of devel- of national institutions, much remains to be done regard- opment of local political culture and institutional designs ing the management of recruitment and human resourc- and practices in the municipalities. The truth is that with- es and on issues such as transparency of management pro- in the complex context of the municipalities, in terms of cesses, and adjustments to planning, evaluation, and control 12 S ession 1 : C hallenges to E vidence - B ased D ecision M aking processes. Institutional arrangements regarding meritocracy, tions in Chile. The process of resolving those challenges has the wage gap, assessment of human resources, and training identified key features of good M&E design, implementa- are still weak. Moreover, the continuing existence of institu- tion, and use of information. The paper focuses on the in- tions with a high risk of corruption is mainly a result of un- strument of program evaluations and institutions, based on due influence in the selection processes and appointment of the important role the latter have played in supporting de- officials. All this has a negative impact on the efficiency of cision-making processes. The paper provides a brief descrip- these institutions and citizens’ confidence in them. tion of the definitions adopted to provide data on the per- We need to learn more and more about the institution- formance of programs and public institutions. It presents al conditions and practices of public servants that explain the alternative methodologies and their main features, and con- management of entities and account for society’s high or low centrates on the functions that the evaluated institutions de- confidence in government action. Corruption can occur in velop. It also describes the type of external assessment and different ways while blending into the management processes the process through which evaluations are conducted and to the point of becoming invisible. For this reason, Transpar- used, and presents the differences compared with the com- encia por Colombia’s work over the last seven years has gener- mencement period of 1997–99. The paper further describes ated important information, methods, and tools, which must ways in which the data yielded by the evaluations are used be collected and used in the fight against corruption. in different decision-making processes, and the people in- Given its independent nature, thoroughness, and ob- volved. Finally, the paper discusses considerations and les- jectivity, the index has proved to be a valuable input to sons learned when building a program evaluation system promote institutional change, guide public policy, and and ensuring its function and development over time. mobilize citizens. Certainly, progress has been made in un- derstanding the phenomenon of corruption from the re- Definitions adopted to acquire information on the sults of the index, but this is a complex and dynamic phe- performance of public programs and institutions nomenon, which changes constantly and imposes new Chile has been no exception to the desire in many coun- challenges. Among these, an important ongoing challenge tries to improve the quality of public spending. In the ear- is to explore innovative ways to detect new expressions and ly 1990s, the Finance Ministry began developing Manage- manifestations of corruption and to encourage institutions ment Control Systems and Budgets by Results, a key part and governments to act on them. of M&E of public programs and institutions. The M&E Additionally, although the media has widely publicized developed in Chile has involved a group of definitions on information produced by the index, it has yet to exploit a the performance of programs and institutions, and has wider stage with a view to raising public awareness of the yielded the following characteristics. problem and ensuring that social organizations are armed with data to support their social vigilance of public man- Existence of different instruments agement. Chile’s M&E consists of instruments that measure perfor- mance and evaluation through different instruments or Elisabeth Ungar Bleier methodologies. This use of different instruments is based Executive Director, Corporación Transparencia por Colombia on experience; a single instrument or a single methodolo- gy proved to be insufficient to obtain the required informa- tion and explain the functioning and results of the many Challenges for Evidence-Based Decision Making: public actions. The Experience of Chile and Chief Features of Its Design, Implementation, and Use of Information Combination of instruments The instruments are based on methodologies and formats Introduction of different degrees of complexity, thematic scope, and re- This paper presents some of the challenges faced by the de- quirements. This takes different technical skills, generates velopment of monitoring and evaluation (M&E) in opera- different kinds of performance information, with different 13 M&E SYSTEMS AND M&E TOOLS costs and application times, and enables different uses. The M&E is performed by institutional players with roles and M&E installation has used the budget process as both a ba- responsibilities. This takes the form of assignment of peo- sis and an incentive. Its development takes into consider- ple in charge in the institutions or public services covered ation budget control, mainly derived from its annual cycle, by the instruments. The existence of an interministerial where well-defined, regulated, and repetitive stages allow in- committee responsible for making decisions for key aspects struments and processes to be integrated and tied to specific regarding the application of evaluations, formed by the work targets, and generate cumulative learning experiences. Ministry of Planning, General Presidential Ministry, and Finance Ministry’s Budget Office. This includes the defini- Performance data tion of technical, operative, and financial handling by the The use of performance data in decision-making processes Finance Ministry’s Budget Office. for resource allocation also considers other data categories, such as political-programmatic priorities and constraints Supply of the M&E instrument: The importance imposed by fiscal policy. That is, the link between perfor- of evaluation mance data and funds-allocation decisions ensures that the The Chilean M&E is composed of performance-indicator information is integrated into a decision-making process instruments and evaluations. The indicators are measuring that is both technical and political. tools that use formulas, which must contain the necessary variables. For example, an indicator covering a program for Political will school meals must use data regarding student population Political will refers to the institutions leading the process. served by the program and information about the popula- Although it is true that the initiative to develop tools for tion eligible to receive the benefit, that is, the potential ben- M&E emerged in the 1990s, it has received basic support eficiary population.5 The ratio between both variables will from the budgetary and fiscal authority throughout its de- measure the program’s coverage. velopment and operation. It was promoted by technical ar- Despite the importance of the measurement indicators, eas of the Budget Office of the Ministry of Finance—at a they do not explain the reasons for whether or not a certain second level of authority, not from central bodies such as performance level is obtained. This limits their use in deci- the Presidency or Congress as in other experiences. sion-making processes, making it necessary to complement the information with other data of a partial, qualitative Public information nature, or taken from other performance data-collecting Since the first tools were installed, it was established that the tools. In the latter case, evaluations have proved valuable in information they generated would be public. This principle Chile to complement the shortcomings detected, generat- has been observed mainly by forwarding data reports to Con- ing a mutual support between the two instruments. gress and publishing them in the Budget Office’s Web site.4 In 1997, the government introduced various program evaluation lines, in effect, to obtain more complete infor- Gradual development mation about the performance levels experienced by public The M&E instruments were introduced gradually, start- actions and to better understand the reasons or causes for ing in 1993. Despite the fact that their processes and ef- those levels.� Initially the evaluation of public programs was fects in the first stage evolved slowly, and in some aspects incorporated in a line of flash or desktop evaluations, called even receded, the gradual incorporation of methodologies government program evaluations. Later the impact evalua- has enabled the persons concerned to learn, correct, per- tion line was introduced, which was followed by the insti- fect, socialize, and generate a critical mass base for a greater tutional evaluation called comprehensive spending evalua- sustainability of the system. tions and, more recently, new programs evaluation design. With the application of these different evaluation lines, Design of institutions this tool has become a key piece of M&E. In effect, giv- Design of institutions means operatives and leadership en the different types of information it generates, it has al- from the Budget Office of the Ministry of Finance. The lowed for a more intensive use in the decision-making pro- 14 S ession 1 : C hallenges to E vidence - B ased D ecision M aking cesses; it has facilitated the job of informing outside players TABLE 1: and has produced a base for following up any changes that Logical Framework Matrix the programs and institutions have had to introduce as a Description of Indicators/ Means of Objective Information Verification result of the findings and recommendations of evaluations. Additionally, the evaluation process has been a learning End: space for the organizational units evaluated and the Budget Purpose: Office. This has helped develop skills. Below are brief de- Components: scriptions of the different lines of evaluation. Activities: Flash or desktop evaluations Government program evaluations are based on the basic Organization and management concepts of the logical framework matrix, supplemented Evaluation of organization and management consists of the with the review of a set of topics. The evaluations use avail- analysis of key institutional aspects within which the pro- able information (mainly simple polls), existing or back- gram operates, key work processes, coordination mecha- ground data that can be quickly systematized and trans- nisms within the institution and between institutions, if formed into relevant data, interviews with key players, and, applicable, and instruments or mechanisms available for in some cases, the application of other simple qualitative operations and reaching objectives.6 methods. They are conducted by a panel of three experts. The panel includes an expert in evaluation, an expert in Results the programmatic sector or area to be evaluated, and an The results or performance evaluation focuses primarily on expert in management and administration of public pro- assessing the effectiveness, efficiency, and economy of the grams. The evaluations last between four and six months. program, observing its processes, products, and intermedi- The evaluation begins with the development of a logical ate and final results, as appropriate and feasible, depending framework matrix (mainly simple polls), which includes on the available information. The evaluation of the results different levels of program objectives through identifica- or performance is made by applying the horizontal logic of tion of its end and purpose (overall objective), the specific the matrix. objectives of each one of its components, its main activities, and their corresponding performance indicators. Table 1 is Sustainability and continuity an example of a logical framework matrix. Based on analysis of the design, organization and manage- The evaluation can begin to be developed after the ma- ment, and outcome or program performance, the evalua- trix has been constructed, updated, or perfected, and used tion incorporates judgments about the program’s sustain- to gather data on the following topics. ability in terms of institutional capacities and whether or not it should continue considering the evolution of initial Justification diagnosis and results achieved. Justification is the stage in which an analysis is made of the Finally, the evaluation must present conclusions and diagnosis or problem that triggered the program and that recommendations, identifying weaknesses and shortcom- the program hopes to solve (or help solve). ings that affect program performance, while consistently suggesting actions to improve performance, including in- Design stitutional and organizational, programmatic, or manage- The evaluation of the design consists of analyzing the or- rial (operational). ganization and consistency of the program, and the com- ponents and activities. Design evaluation considers the Impact assessments program’s initial justification and how these diagnostic el- The flash or desktop assessment methodology makes use ements could have evolved. It uses the logical framework mainly of available information. As a result, the informa- matrix, specifically the vertical logic analysis. tion delivered in some cases is incomplete. In most cases, 15 M&E SYSTEMS AND M&E TOOLS it does not specify how it meets the needs of the program. observed results. In effect, depending on the needs for in- This prompted the introduction in 2001 of the impact as- formation, the following topics, described above, are add- sessments line. The line of impact assessment methodol- ed: justification, design, organization and management, re- ogy is designed to measure the final results or net effects sults, and sustainability and continuity. Finally, as in the of a program on the beneficiaries, that is, the efficiency in line of flash assessments, the evaluation findings and rec- the final stage of the value chain. It uses differences regard- ommendations must be submitted. ing baselines (initial diagnosis), compares these differences with control groups to isolate external effects, and also us- Comprehensive spending evaluation es other quantitative techniques to obtain precise measure- The comprehensive spending evaluation line began in 2002 ments. It requires high technical skills and fieldwork. It is and is an evaluation of an entire organization or agency. It more expensive, lasts between 12 and 18 months, and is mainly uses available information, existing background da- conducted by consultants or universities. ta that can be systematized during the evaluation period Although the gist of impact assessments is to measure and transformed into relevant information, and interviews the effects of the programs, in the Chilean experience, this with key actors and application of simple qualitative stud- focus has been supplemented with the review and evalua- ies. It aims to evaluate all the policies, programs, and pro- tion of specific complementary topics. These topics provide cedures arranged into the following topics or categories of other categories of information that help understand the information (see Table 2).7 TABLE 2: Working Design for Evaluation of Programs and Institutions (ex post), 2000–09 Stage Actors 1. Selection of programs or institutions. Selection is made with the participation of the National The evaluation process begins with the selection of programs Congress, based on a proposal submitted by the Finance or organizations (or agencies), depending on whether the Ministry’s Budget Office. The Ministry of Planning and the evaluation will focus on programs or include expense. Presidential Secretariat participate in preparing the proposal. 2. Selection of consultants or consulting firms. Consultants or consulting firms are selected through public After the selection of programs and institutions comes the se- tenders held by the Budget Office, supported by selection lection of consultants or consulting firms (including universi- committees that include representatives from the Planning ties) through public bids. Ministry and Presidential Secretariat. 3. Preparation of basic information. The information is prepared by the organizational units that At the same time that the evaluators are selected, the com- are to be evaluated and coordinated by the Budget Office. mencement of the evaluation process is announced to the ministers and organizational units to be evaluated, and they are asked to prepare the basic information for conducting the evaluation. 4. Mention of critical aspects of the unit to be evaluated. One or more meetings are held, which are attended by the Remarks are made to the evaluators about those critical areas evaluators, the persons responsible for the program, or the of programs or institutions, so that they can be reviewed institution being evaluated and the Budget Office. Meetings carefully in the evaluation process. may also be held with the Congress for the same purpose. 5. Training and preparation of frameworks. Training is given by the Budget Office, and initial prepara- In the case of flash or desktop evaluations, before their tion of the logical frameworks is done by the evaluators commencement, both the evaluators and the professionals along with the evaluated unit, under the coordination of the who work in the programs’ responsible units are trained in Budget Office. evaluation methodology and logical framework basics. For Impact Assessments and Comprehensive Spending, the basic methodological elements to be used by the evaluations are socialized, and in the latter instance, the preliminary evalua- tion framework is prepared. (continued on next page) 16 S ession 1 : C hallenges to E vidence - B ased D ecision M aking TABLE 2: Working Design for Evaluation of Programs and Institutions (ex post), 2000–09 Stage Actors 6. Evaluation. The evaluation is made by outside evaluators. The technical Evaluation begins with available background data as a base. and administrative counterpart is the Budget Office. The The execution stage of the evaluation requires progress organizational units evaluated participate by submitting reports and a closing report to be submitted. These are information, attending meetings, and checking reports. commented on by the institution under evaluation and the Budget Office. 7. Analysis of evaluations. The analyses are made on the one hand by the evaluated As soon as the evaluation is finished, the evaluated organi- organizational units, and on the other, by the Budget Of- zation analyzes the final report and issues an Institutional fice. Participants in the Budget Office’s analysis are budget Response to the evaluation. experts, experts in charge of evaluations, and the budget The report also is reviewed in the Budget Office, which director. pinpoints the changes needed and the possible effects they would have on the budget. 8. Forwarding to Congress and presentation on Web site. The budget director sends the evaluation reports to Con- The final evaluations report, an executive report prepared by gress, and the Budget Office posts the data on the Web site. the Budget Office and based on the final report, along with final comments from the evaluated institution (institutional response), are officially sent to the National Congress. 9. Forwarding to Congress with draft budget. The budget director sends the Public Finances Reports to Incorporation of the data added in the public finances report Congress. and executive summaries in attachments to the draft budget. 10. Formalization of commitments. Commitments are established between the Budget Office Based on the evaluators’ recommendations, commitments are and the evaluated institution (ministry). defined between the evaluated institution and the Budget Office. 11. Follow-up. The Budget Office follows up on improvement commit- A follow-up of institutional commitments in keeping with ments, based on the data sent by the evaluated organiza- recommendations is made every six months. tions. 12. Report of fulfillment of commitments and analysis of follow- The Budget Office analyzes the follow-up data. up data. The Budget Office analyzes data tracking commitments in the Evaluation of Budget Execution stage. 13. Forwarding report on fulfillment of institutional commit- The budget director sends the government financial man- ments to Congress. agement evaluation reports to Congress. This is sent twice a year, as part of the Public Finances Report and Report on Evaluation of Financial Management by the Government, which also updates information about com- pleted evaluations. Source: Author. Institutional design Institutional management Institutional design considers evaluating consistency The aspects evaluated in institutional management relate to among government priorities, mission, strategic goals, stra- institutional capacity (professional, technological, organi- tegic products, and their beneficiaries. Similarly, based on zational) and management mechanisms or procedures ap- the foregoing, it evaluates the coherence of institution- plicable and relevant to the organization, such as (1) mech- al structure and division of responsibilities between work anisms for coordination and allocation of responsibilities; units that make up the body or public agency. (2) allocation mechanisms, funds transfer, payment proce- dures, and audits; and (3) accountability and transparency 17 M&E SYSTEMS AND M&E TOOLS in the use of resources, activities, M&E tools, and targeting consultants organized into panels of experts, universities, criteria, or selection of beneficiaries. or private consultants. In all cases, the selection is made through public tenders. Performance and use of resources This process does not mean that the institutions eval- Performance and use of resources consists of evaluating the uated do not participate but rather that the form of this performance of products and strategic subproducts,8 and participation should allow all facts and data available to linking this information to the information about the re- be gathered, while safeguarding the objectivity of the pro- sources assigned to providing them. Finally, as in the line of cess. In other words, they provide information, partici- flash evaluations, the evaluation findings and recommen- pate in technical meetings, and comment on reports, but dations must be submitted. Comprehensive spending eval- they do not play the role of technical or administrative uations last about 14 months. Considering the various ac- counterpart. tivities that need to be undertaken, it was decided that they should be carried out by consulting firms or universities se- Operational process and actors lected through a public bidding process. The evaluations are conducted according to a working de- sign that orders the different work stages of the evaluation Evaluation of new programs cycle, and incorporates the use of information. Table 3 The new program evaluation line was introduced in 2009 shows these different stages, in sequence, and the actors in- and is developing the design to include evaluation of new volved in each one. public programs from the beginning to ensure the avail- The first phase of the Chilean experience did not ability of the basic elements and information that will en- consider all the stages that characterize the current pro- able the effects to be measured with the utmost rigor once cess, nor were they developed with sufficient technical the program is up and running. The evaluation design con- quality. This jeopardized the quality of information de- siders the baseline identification with regard to the vari- livered and its use. The lessons learned, however, from ables that will be affected by intervention and construction this first period (1993–2002) allowed for a number of of the control group and uses the best possible methodol- changes to be introduced. Moreover, during the last 10 ogies to apply to each program. An external agency expe- years, the operative process has undergone a continuous rienced in quantitative techniques and data collection and improvement.10 management designs the ex ante evaluations. The devel- opment of this evaluation line has the technical support Process for selecting programs or institutions for of an international advisory panel established for this pur- evaluation pose and composed of academics with extensive experience The decision of “what to evaluate” should consider the in the area of impact assessments. This panel meets annu- demand for information. In Chile, as in most countries ally in Chile and reviews the progress of the designs con- in the region, the demand for appraisals to learn more cluding with a detailed report of comments and recom- precisely about government actions and performance ex- mendations.9 ceeds the resources available. As a result, every year on- ly a limited number of assessments are made, and it is Principal features of operational design necessary to select those programs and institutions to which each of the available methodologies will be ap- Outside aspect of the evaluations plied. In 2009, for example, 20 quick or desktop evalu- Chile opted for an external evaluation process for the or- ations, eight impact evaluations, five comprehensive ex- ganizational unit responsible for the program or institu- penditure evaluations, and five new program evaluations tion being evaluated, and that also is external to the min- were conducted. istry to which the program or institution belongs. Indeed, The choice of what to evaluate arises from a proposal evaluations are coordinated from the Ministry of Finance prepared by the Budget Office and presented to Congress through the Budget Office, and are executed by external for its final definition within the context of discussions 18 S ession 1 : C hallenges to E vidence - B ased D ecision M aking TABLE 3: Uses and Actors in Ex Post Evaluations Uses Actors Review of evaluation results in the Finance Ministry’s Budget Budget Office: Office defining guidelines to establish pledges for improve- • Management Control Division’s Evaluation Unit ment and identifying budget impact. • Head of Management Control Division • Head and analysts of budgetary sector to which the evalu- Period: ated program or institution belongs • GPE evaluations before the start of the budget formula- • Budget director tion • Evaluations on impact and comprehensive spending as soon as each is completed Incorporation of data from the evaluations into meetings of Budget Office: the Technical Commissions as information on which to base • Head of budget sector decisions. • Sector analysts Ministry or institution: Period: Draft budget formulation process • Undersecretary • Services chiefs • Other institutional headquarters and analysts Summary report of the evaluations and compliance with com- Budget Office: mitments for better dissemination of the public finance report • Management Control Division and its submission to Congress. • Director Period: Presentation of Draft Budget Law to Congress Presentation of evaluations data to Congress in a document Congress: Mixed Budget Commissions complementary to the Draft Budget Law. Budget Office: • Assistant director Period: Discussion of Draft Budget Law in Congress • Head of budget sector Ministry or institution: • Undersecretary • Services chiefs • Other headquarters and institutional analysts Preparation of pledges to improve evaluated programs and Budget Office: institutions. • Management Control Division • Evaluated ministry or institution Period: As soon as the evaluations are completed and their results have been analyzed Overview of evaluation results and compliance with commit- Budget Office: ments published in the “Evaluation of Government Financial • Management Control Division Management” and submitted to Congress. • Budget director Period: May to June each year, before commencement of bud- get formulation Source: Author. about draft budget laws. The final selection is established and evaluation indicators, the current management in an agreement protocol signed every year by the Minis- control system includes instruments such as the try of Finance and members of the Special Mixed Budget management improvement programs and the in- Commission after the Budget Law has been passed.11 That tegrated management inventory, which also gath- proposal, which represents the relationship between supply er information). of and demand for performance information, takes into ac- • Status of programs or institutions as to wheth- count the following aspects: er they have been evaluated in the context of the evaluation program run by the budget office, or by • Preliminary performance data of programs or in- other evaluation processes, and the year when that stitutions that the Budget Office holds in keep- happened. ing with its functions, both regarding budget and • Time the program has been in existence, and its management control (in addition to performance priority. 19 M&E SYSTEMS AND M&E TOOLS • Suggestions from the General Presidential Min- Pledges to improve programs and institutions istry-Secretariat and the Planning Ministry, both The evaluations also can be used for the commitments members of the Interministerial Evaluation Pro- made to improve the programs and institutions evaluated. gram Committee. These commitments are made on the basis of an analysis • Suggestions put forward by the special mixed bud- of the assessment’s findings. These commitments promise gets commission and congressional budget sub- that actions will be undertaken in the coming semesters, commissions during the processing of the draft and they are formalized by the Budget Office. Subsequent- budget law in Congress. ly, this office monitors compliance with promised actions and specific aspects of the changes introduced. The com- Ways to use the information: Incorporation into mitments concentrate mainly on improving design and the budget cycle �� management, and imply that the evaluations will used for In terms of resource allocation, the use of performance da- purposes other than those directly related to resource allo- ta that emerge from the evaluations has not translated in- cation. Table 3 shows the different uses of evaluations and to an immediate mechanical step to allocate more or fewer the actors involved. resources depending on the results of the evaluation dur- ing the period, but rather the data have been integrated in- Main challenges to more comprehensive analysis processes. One way to use The development of evaluations as part of the M&E and the evaluations is to integrate the data at different stages of the growth of this line over time has allowed important in- the budget cycle. A review of evaluation results in the Bud- sights that, introduced in a timely manner, have granted get Office implied the following: sustainability. New challenges and risks, however, are even more complex than those faced in the early stages. Some of • Guidelines are defined for making commitments to these are described below: improve evaluated programs or institutions, classi- fying programs into categories according to the ef- • Optimize the M&E cycle for the purposes of eval- fects or implications from the evaluations. uation. Advance the link between evaluation, mon- • Budget impacts of component adjustments are itoring incorporation of changes or improvements identified if they correspond to findings of the eval- and ex post measurement of performance or effects. uation, or extensions if warranted by those find- The evaluation aims to improve the quality of spend- ings, if the financial resources are available and if ing, which means actual achievements in the purpos- they represent a high priority (high relative impor- es of the programs and their contribution to policy tance in terms of established policies). goals. The commitment to improve arising from eval- • Data from the evaluations are incorporated into the uations and compliance therewith is essential to im- technical commissions’ meetings during the budget proving program results. Achieving better results is formulation process, to analyze and lay the ground- the intended effect of the implementation of evalu- work for decisions among the Budget Office, the ations. So how does one know whether these effects ministers, and their institutions.12 have occurred? A first step to get an approximation • Evaluation-derived data, together with the draft of these effects is to observe measurements by com- Budget Law and also the data summarized in the paring performance indicators over time. This line of public finances report, are presented to Congress. work, seemingly simple, is a major challenge for the • The evaluation data in the Budget Office’s evalu- links between evaluations, indexes, and reevaluations. ation of budget execution are presented for anal- • Integrate the evaluation function. Move toward ysis along with financial performance data, and more coordination with other elements of the eval- summarized data are incorporated into the annu- uation function, such as investment appraisal, eval- al publication “Evaluation of Government Finan- uation of government policies or goals, and so on. cial Management.”13 This raises a challenge to harmonizing technical, 20 S ession 1 : C hallenges to E vidence - B ased D ecision M aking operational, and institutional facets, particularly the evaluation consider definitions on three key in- because these areas have different levels of devel- terrelated aspects: (1) the purpose of the evaluation, opment, have different features, and are located in (2) the methodology to be used, and (3) the focuses different institutions. It entails risks as well since on which each evaluation will concentrate. the integration of functions and institutional relo- • The object of an evaluation may be a program, a cations may affect current achievements, especially component of a program, a set of programs with a regarding the use of evaluations. common goal, or even an organization. In this re- • Consider the growth in number and complexity of gard, the Chilean case shows that it is possible to the evaluations and care about their quality, timeli- initiate an evaluation system of between five and ness, and use. How many more evaluations can we eight individual programs within two years, gradu- make without affecting their quality or the time- ally increasing the number, and always prioritizing liness of the information available (availability of the country’s greatest needs on the question of per- evaluation results to use for decision making)? In formance data. Chile, the number of evaluations has increased in • Having identified the object of evaluation, it is the last three years. This makes it necessary to re- then a question of deciding what methodology view the ratio between growth and the quality- will be used to perform the evaluation. In gener- timeliness-use trilogy, before continuing to increase al, this selection must consider the resources avail- evaluations. able to make the evaluations; organizational, finan- • Disseminate the information. Despite all the pro- cial, and professional skills; the runtime required paganda efforts made, in the form of publica- for each program; former evaluations available, if tions and information available on the Web site, any; and the date when the information is required. press conferences, and other tried-and-true outlets, Chile began with flash or desktop evaluations us- greater transmission of the evaluation results and ing a simple methodology that allowed the devel- their effects still is needed. This information is need opment of skills that later made it possible to im- for both the actors involved in the execution and plement more complex methodologies. the beneficiaries, with the latter requiring a differ- • After selecting the object and the methodology, ent form and language. the most important subjects to be addressed by the • Conduct self-evaluation, goal assessment, and con- evaluation can be specified. To this end, it is ad- tinuous improvement. Considering the progress visable to gather the opinions of people responsi- achieved and the size of Chile’s Evaluation Program, ble for the programs and the actors who would be it is a challenge to maintain this regular evaluation, using the information (technical professionals, au- either in the form of self-evaluation of process and thorities, parliamentarians). uses, or external evaluations in the form of meta- • Consider progression. Evaluations should be devel- evaluation as carried out by World Bank in 2004–05. oped gradually, as far as selection of the object of evaluation, methodology, and focus are concerned. Lessons and recommendations They should be accompanied by the correspond- The key lessons learned and recommendations regarding ing operation processes. It is possible to start pro- the development of evaluations arising from the Chilean gram evaluations using a simple methodology, and experience, the experience of other countries in which the subsequently, depending on progress, move toward author has been involved, the review of documentation, more complex methodologies. The number of eval- and participation in workshops and international seminars uations to be performed each year may be increased in the region are the following: in keeping with the resources and skills available. • Consider the purpose of the evaluation, method- This pattern requires progress in constructing internal ology, and subject covered. It is recommended that task forces to ensure strong partners, while at the same time 21 M&E SYSTEMS AND M&E TOOLS constructing an evaluators’ market. In this area, a team of ed. Additionally, these improvement commitments between five and seven people should be trained to perform must be monitored every six months to one year the technical leadership role. These people then would to verify the incorporation of agreed-on measures. train other professionals or supervisors (cascade training), • Consider the institutional and basic operating prepare technical guides, train evaluators to act as the tech- model. For evaluations to fulfill their purpose of nical counterpart of the evaluations, and learn from the ex- providing feedback for decision-making processes, perience to perfect the system in the following cycles. basics such as institutionalization and implementa- tion management should be considered. In fact, the • Consider the availability of information and lim- development of evaluations and their use do not itations of the evaluation process. The availability happen automatically. They require new practices of information is central to evaluation processes. and skills that involve a variety of technical and po- Whatever the methodology, it must use informa- litical actors, as well as a large number of technical tion. Failure to do so will limit the evaluation pro- and administrative processes. In view of these fac- cess. The true scope of the evaluation in terms of tors, it is recommended that an institutional orga- implementation of methodological elements there- nization be created to perform evaluations and use fore depends on available data, in addition to the any information that may be obtained from them. time allotted for the evaluation process and techni- In this regard we suggest considering institutions cal competence of the evaluators and counterpart. and organizations that play a part in defining, as- Consequently, it is recommended that special at- signing, and reporting on the various functions and tention be paid to the generation and collection of responsibilities; and in designing the stages, proce- information as part of a regular work practice. dures, and annual timetable of the operation pro- • In many instances the evaluation will face a situa- cess, including the phases of analysis of information tion in which the units responsible for the program for integration into the decision-making processes, will not have complete or reliable information on as well as for preparation and monitoring of com- crucial aspects of performance. That could hinder mitments. the identification of the program’s baseline and pre- vent any changes from being measured, or it could Marcela Guzmán S. prevent core performance-gauging indicators from International Consultant, former being measured. In these cases the focus should be Chief of Budgeting Division Office, Chile on evaluating other aspects, such as program de- sign, basic management processes, and the efficien- cy level of products (outputs). Session 1 Notes 1 • Develop a mechanism for following up the recom- The institutional conditions are the rules, functions, skills, ac- mendations. The design of an evaluation system tors, and systems that structure administrative and govern- ment action. The practices of authorities and public agents are should not end with the delivery of the final re- the behaviors with regard to institutional conditions. port, or after completing the analysis of its findings. 2 For more information on the variables and indices, see Along this line, a phase should be incorporated to www.transparenciacolombia.org.co. facilitate, promote, and encourage compliance with 3 See the Web site, www.dipres.cl/control. 4 the objective of “improving” after an evaluation, in- The measurement unit of the indicator is expressed as a per- troducing changes into the “object evaluated” based centage, according to the following formula: No. of students attending the program/No. of students eligible to receive the on the findings and recommendations that the benefit * 100. evaluations indicate. Indeed, the recommendations 5 A program consists of a number of activities arranged in- should be seen as an opportunity to discuss and to components or lines of work to supply goods or services, commit to changes by the institutions or organi- whose purpose is a common objective. zational units responsible for the program evaluat- 22 S ession 1 : C hallenges to E vidence - B ased D ecision M aking 6 11 Characteristics of the program‘s responsible organization- Document available at the Web site, www.dipres.cl. 12 al unit, mainly considering its functions, its location in the Technical commissions are meetings of the Budget Office of structure of the organization, its own structure of subunits in the Ministry of Finance and ministries and institutions whose the central and subnational levels, and allocation of responsi- budgets form part of the Budget Law, and take place each year bilities to each, including staffing. when the Budget Law bill is drafted. Each ministry explains 7 Legal information, strategic and programmatic definitions, in- the priorities and technical aspects on which the proposed stitutional memories and balances, prior studies, official docu- budget is based. 13 ments, financial data, indexes, data, and so on. The Budget Office conducts an evaluation of financial perfor- 8 These may refer to programs or sets of programs or actions mance and management as an internal task. The budget direc- with a common unit or purpose. tor and the Sub-Department of Budgets and the Management 9 This report is presented to Congress in certificate form. See Control Division analyze background information on finan- Budget Office (2009a), 155-161. cial management and the performance. This analysis serves 10 This improvement has been helped by self-evaluations con- as the basis for defining exploratory budgets and budgetary ducted periodically by the Budget Office, and by the Im- frameworks for the next stage of drafting the Budget Law Bill. pact Assessment made by the World Bank in 2004–2005 (see World Bank 2005). 23 Session 2: Institutional Arrangements for Monitoring and Evaluation Systems: International Experiences Monitoring and Evaluation in the Government Over the past 30-plus years, however, many changes of Canada: Lessons Learned from 30 Years of have occurred with policy, practices, and standards of eval- Monitoring and Evaluation Development uation. The policy, for example, has been updated on three occasions: 1991, 2001, and 2009. These changes occurred Introduction for a variety of reasons. They included (1) a recognition This paper provides an overview of the Canadian govern- that the need for evaluation or performance measurement ment’s monitoring and evaluation (M&E) model and points has evolved or become clearer, (2) M&E practices have ma- to how the concepts have evolved and key lessons learned tured, or (3) the demands of the system through public from more than 30 years of public sector experience. sector reform put emphasis on new areas—for example, the widespread introduction of results-based management The evolution of M&E in Canada (RBM) in the 1990s and 2000s. In Canada, the concept of M&E recognizes that evalua- The 1990s saw an increased move to performance mon- tion has a distinct identity from monitoring. In other parts itoring and high-level reporting in an attempt to make per- of the world, M&E may ignore or minimize evaluation. formance information more accessible and useful to par- Evaluation in public sector management precedes by many liamentarians and parliamentary committees. A formal years the notion of “outcome monitoring.” It goes back requirement for each government department to submit an to 1969, with the initiation of formalized and centralized annual departmental performance report (DPR) to parlia- evaluation practices. That initial approach to centrally led ment placed new emphasis on program managers (and not evaluation was replaced in 1977 by the first government- simply evaluators) to be more accountable for measuring wide evaluation policy that established the model on which and reporting on the performance of their programs. Asso- the practice of evaluation still functions in the Canadian ciated with this was a new emphasis on the measurement government. of “outputs” and “outcomes” and not simply “inputs” (re- The focus on evaluation was inspired by the notion sources expended) when reporting on “performance.” As of “letting the managers manage,” that is, allowing depu- with the evaluation function, the Treasury Board Secretar- ty ministers of federal government departments to assume iat (TBS) has played an active role in providing guidance greater responsibility of their departments and programs, and oversight over performance reporting and the produc- and be accountable for the performance of those programs tion of DPRs across the system. and the prudent use of public funds. The model is based on Unlike some of the other member countries of the Or- a strong central management board that oversees and holds ganisation for Economic Co-operation and Development deputies accountable. One of the mechanisms to achieve (OECD), which significantly reduced the role of evalua- this is performance evaluation. tion during their reform of public sector management, 25 M&E SYSTEMS AND M&E TOOLS Canada held the view that the two tools were complemen- available to the deputy head, and provides important in- tary, and not competing, within the performance measure- puts to strategic reviews that may be needed to make man- ment tool kit. agement decisions regarding program priorities and possi- The 2000s have seen two attempts to revitalize the eval- ble changes. uation function, with updated policies in 2001 and in At a governmentwide level, the TBS is an important 2009. In line with the government’s modern management player in the production of M&E information and in its agenda, as articulated in Results for Canadians, evaluation use in informing funding decisions about certain programs and performance monitoring (and internal audit) are seen (grants and contribution programs subject to funding re- as key tools to ensure a results focus, responsible spending, newal) and governmentwide expenditure management. and greater transparency and accountability across govern- Finally, in a legislative context, M&E information about ment. Table 4 highlights key elements in the evolution of government programs and operations is reported directly to M&E in Canada. parliament on an annual basis through DPRs and an over- view performance report, Canada’s Performance. The intent An overview of the Canadian M&E system is to enhance the transparency and accountability of gov- ernment operations with parliamentarians and Canadian Many uses and users of M&E information citizens. More detailed discussion about the performance of The intent of M&E in the Canadian system is to provide government programs or operations may occur in the par- results information that will serve a variety of needs and liamentary committees that have the authority to seek ad- users at different levels throughout the system. At an op- ditional information or clarification directly from the rele- erational, departmental level, M&E would be expected to vant departments. serve as learning tools to assist in program improvements and sound management practices. At the level of an in- Roles and responsibilities in Canada’s dividual department, with the deputy head accountable M&E system for good governance and performance of the department, Within the Canadian M&E system, the delivery and use of M&E represents key management and accountability tools M&E information has two key focal points: the individu- TABLE 4: The Evolution of M&E in Canada Key Features Canadian Experience Initiation of formalized evaluation 1969 practices Governmentwide Evaluation Policy 1977 introduced Evaluation Policy revised 1991, 2001, 2009 Drivers of ongoing performance moni- • Departmental Performance Reports, DPR (1995) toring and reporting (date introduced) • Results-based Management and Accountability Frameworks, RMAF(2000) • Management Accountability Framework, MAF (2003) • Management Resources and Results Structure, MRRS (2006) Key drivers of change in emphasis on • Public sector reform governance tools • Fiscal restraint • Government Management Agenda: Results for Canadians Current emphasis on evaluation • Formalized role within each department capacity • Central management (guidance and oversight) of the function (Centre of Excel- lence for Evaluation, CEE) Current emphasis on governance tools • Evaluation (program and policy) • Performance monitoring • Internal audit • Strategic reviews Source: Author. 26 S ession 2 : I nstitutional A rrangements for M onitoring and E valuation S ystems al government departments and the TBS. The TBS plays a ongoing performance monitoring will provide much of the strong and proactive role in the practice and performance needed data for the eventual evaluation of a program; in monitoring of evaluation within departments. The evalu- practice, however, this does not always happen. ation policy center, the Centre of Excellence for Evalua- The national audit office, the auditor general (AG) of tion (CEE) resides within the TBS. This unit plays a variety Canada plays an important role in providing oversight of of roles in support of evaluation capacity building and its the effectiveness of evaluation and performance report- use, and provides operational oversight regarding standards ing—that is, by examining the use of the function across and quality of evaluation practices and products. With the the system and the “quality” of performance reporting. As 2009 policy, TBS can now direct the conduct of govern- an independent auditor and reporting directly to parlia- mentwide evaluation studies. TBS also provides guidance, ment, the reports of the AG raise the profile of M&E and proactive support, and oversight for performance monitor- its use in the public sector. The reports generally receive ing that takes place in departments. Finally, TBS links eval- public attention and provide constructive advice to indi- uation to inform program funding decisions and the broad- vidual departments as well as to the central agency. His- er expenditure management system. torically, the AG has been a strong supporter of the central For individual government departments and agencies, agency role as a champion and facilitator of departmental deputy heads are required to provide dedicated resources to evaluation. establish an evaluation capacity appropriate to their organi- The other key group implicated by M&E is represent- zations. Although the resourcing of the evaluation function ed by elected officials. Parliamentarians may be exposed to is flexible, the TBS/CEE monitors this and ensures that M&E information in various ways: each department establishes the required evaluation infra- structure—that is, a senior-level evaluation committee, an- • through the annual tabling of departmental perfor- nual and multiyear planning for evaluation, a departmental mance reports and the strategic-level report, Cana- evaluation policy reflective of the government’s policy, and da’s Performance; the mechanisms needed for follow-through on delivery of • through more detailed discussions that may take credible evaluation products. place in the context of a parliamentary committee; Part of the evaluation infrastructure in any department or is the evaluation unit, led by the head of evaluation. This • through discussion among TBS ministers on pro- position ensures that the evaluation function reflects the grams seeking funding renewal or broader-based government’s policy requirements and the deputy head’s expenditure review exercises. priorities in the department’s work. The position general- ly reports to the deputy head or, at least, to ensure the in- Often, though, more exposure is given (through the me- dependence of the departmental evaluation function, has dia) to results emanating from internal and external audits direct access to the most senior official in the department. of government operations. Clearly, though, the public sec- Deputy heads of departments are accountable for es- tor reform agenda of the government (including fiscal re- tablishing a corporate performance framework, called straint) has over the past 30-plus years had a major impact the management resources and results structure (MRRS), on the direction of evaluation and performance monitor- which links all programs of the department to the expect- ing. Figure 3 provides an overview of the Canadian M&E ed outcomes. This is the basis for performance monitor- system. ing and reporting, and its development is monitored close- ly by TBS to ensure adherence to the government’s MRRS Some key features of the M&E system policy. Performance monitoring is an ongoing responsibili- The Canadian M&E system has invested heavily in both ty of individual program managers, although development evaluation and performance monitoring as key tools to of the underlying program logic model and identification support accountability and results-based management. of appropriate performance indicators often comes about Performance monitoring provides managers with an on- with the assistance of the evaluation specialists. In theory, going assessment of how programs are performing, while 27 M&E SYSTEMS AND M&E TOOLS FIGURE 3: An Overview of the Canadian M&E system Parliament of Canada Parliamentary Committees Auditor General of Canada • Independent audit of government operations • Direct report to Parliament • Oversight promotes good practices Treasury Board & accountability • Government’s management board Treasury Board Secretariat • Policy centre for Evaluation, performance monitoring & reporting, results based management (RBM), Internal Audit • Centre of Excellence for Evaluation (CEE) • Capacity Building, guidelines, standards and norms • Oversight of M&E products from departments & agencies • Prepare national performance report. Canada’s Performance, tabled in Parliament • Can direct government-wide evaluation studies • M&E to inform Expenditure Management System (EMS) Federal Department & Agencies • Deputy Head accountable for performance of the organization & good governance • Uses Evaluation function and performance monitoring & reporting tools • Head of Evaluation leads departmental Evaluation function, as per TBS policy & standards • Departmental Evaluation Committee senior-level committee determines priorities for evaluation, monitors use and follow-up to evaluation studies • Department required to establish a corporate performance framework (Management Resources and Results Structure (MRRS) • Program managers required to develop & implement ongoing performance monitoring of their programs • Department/Deputy Head assessed annually against a comprehensive Management Accountability Framework (MAF) that includes quality of M&E information • M&E information used for both internal accountability & management and external accountability and reporting to TBS and Parliament Source: Author. evaluation provides a deeper understanding of performance The introduction of a number of centrally driven ad- and whether adjustments might be needed. Both advise ministrative policies that have been key drivers for M&E deputy heads on how well their departments are function- throughout the 1990s and 2000s generally have been ing and inform parliamentarians on the performance of phased in (that is, pilot tested in select departments) to al- government operations. low for a period for learning and adjustment before their The responsibility for M&E in a government depart- systemwide rollout. ment falls on both technical staff evaluators and nontech- The broad goal in investing in an M&E system has nical officials (program and senior managers) in the depart- been to generate and use results information that supports ment. The intent is that M&E becomes integrated into the the government’s management agenda from the perspec- decision making of the department and is a key driver in tive of learning and accountability in the design and de- moving the organization toward a “results” culture. livery of government policies, programs, and services and The current state of the M&E system has evolved over the use of public funds. In this way, performance report- time, as the central designers have recognized that the de- ing generally aims to“tell the performance story,” rath- velopment and implementation of M&E is long term and er than simply reporting on a limited set of indicators in iterative. This puts emphasis on the “process” of implemen- the context of an accountability scorecard. Furthermore, tation as an important mechanism in itself in developing an evaluation informs decision making, rather than expect- “evaluation culture” or “results culture” in an organization ing that decisions will rest solely on the results of an eval- and across the entire system. uation study. 28 S ession 2 : I nstitutional A rrangements for M onitoring and E valuation S ystems FIGURE 4: Organization and Implementation of M&E in a Government Department Departmental Deputy Head Evaluation Committee (DM and Executive Team) Head of Evaluation Departmental Evaluation Plan (5-year risk-based plan) Evaluation Unit Evaluation Studies • Manage & undertakes evaluation studies • Apply TBS standards • Advise on performance measurement Program Managers Management Response & Action Plan • Participate in evaluation studies Performance Measurement Frameworks • Facilitates performance measurement The model for evaluation in the Canadian system re- reinforce the credibility and quality of the practice of mon- lies on a strong central presence in setting the policy, stan- itoring and evaluation.�� dards, and guidelines for the conduct of evaluation. Most evaluation studies are carried out by evaluation units that Some lessons learned from 30 years of M&E are internal to departments, although the new 2009 poli- development in Canada cy allows for centrally driven and governmentwide evalua- Many lessons can be drawn from the 30-plus years of Can- tions. To reinforce the independence of evaluators, so that ada’s experience with M&E. To help in drawing the broad they may “speak truth to power,” a number of elements in implications, a set of lessons learned have been organized the model ensure objectivity and neutrality of the evalua- under three key headings: (1) drivers for M&E; (2) imple- tion process and the reporting on results. Table 5 identifies menting the M&E system; and (3) key elements of capac- some of the key features of the Canadian M&E system that ity building. TABLE 5: Drivers for M&E Building and using M&E capacity requires more than resources and technical skills—it requires a political will and sustained com- mitment—central leadership and a plan are important. Think in terms of years, not months, to getting to a mature M&E system. M&E information is not an end in itself; it needs to be linked to particular management and decision-making roles, particularly in the context of public sector reforms or government agendas. Clarify the distinction between the “M” and the “E” and what each contributes to results-based management (RBM), and what each requires regarding capacity building. To be effective, build a capacity to do evaluation (and gather performance information), and the capacity to use M&E informa- tion within organizations and across the system. A supply of good evaluations is not enough to maintain a robust evaluation function. There also should be a realistic demand for evaluation. The capacity to use M&E information relies on the nature of the incentives in the system for managers to demand such informa- tion and actually use it as part of their normal operations. Incentives can take many forms, both sanctions for not complying, as well as rewards for meeting requirements. It is important to manage the expectations for evaluation and not create unrealistic expectations about its role. While evaluation can and should inform decision making, it is generally one of many sources of relevant information. Questions about the per- formance of government programs generally do not have simple yes or no answers. Evaluations can and should provide useful information to inform debates about programs, their performance, and future direction. 29 M&E SYSTEMS AND M&E TOOLS Drivers for M&E what it can actually deliver on need to be tempered with re- Developing M&E in an organization or a country requires ality, and generally remain modest. Table 6 provides some a significant investment in time, resources (human and fi- lessons relating to the drivers for M&E. nancial), and energy. To be successful, there ought to be a real need for M&E information. In Canada, much of this Implementing the M&E system need has been driven by various public sector reform ex- Implementation of M&E is long term and iterative, and not ercises and, most recently, the government’s management costless. As such, senior-level commitment and champions agenda, Results for Canadians. at both senior and operational levels are important elements At times, the tendency is to adopt unrealistic expec- to ensure sustainability through the long period of develop- tations in considering what evaluation and performance ment and implementation. Over time, as M&E fatigue ap- monitoring can deliver on, as might be the case in satis- pears, reminder participants why M&E is important. Even- fying, for example, a political need. While M&E should tually, the goal is to move M&E beyond the point of being serve an important place in public sector management, the a special project to one in which it is a normal part of doing expectations about how and when M&E can be used and business and the management practices of the organization. TABLE 6: Key Features That Reinforce Credibility and Quality of the M&E System Element Description Where is the emphasis • Balanced emphasis and active support for both ongoing monitoring and ad hoc evaluation. placed: on the M, on • Program managers accountable for performance monitoring; evaluation specialists within de- the E, or on both? partments accountable for more probing evaluation studies. • 2009 policy gives TBS authority to conduct governmentwide evaluations. • Deputy head of a department or agency accountable for high organizational performance and delivering on expected results—uses both the M and the E (as well as internal audit) to support this—some flexibility in the resourcing of toolsto ensure that they are appropriate to the size and needs of the organization. How formal is the basis • M&E requirements are based on administrative policies (allows more flexibility than legislation to for the M&E require- modify and improve the policy as experience is gained over time). ments? • Identifies roles and responsibilities for all the key players and expectations for the system. • Current basis relies on several centrally administered policies and procedures: evaluation policy; annual requirement to report to parliament via the departmental performance report (DPR); management resources and results structure (MRRS) policy requiring each department to develop a corporate performance framework and reporting structure; policy on transfer payments (re- vised 2008); program and strategic-level performance frameworks (RMAFs); annual departmental assessment by TBS through the management accountability framework (MAF). What capacity-building • Proactive M&E capacity-building efforts from the central agency. efforts exist? • Expectation that deputy heads will resource M&E to the level appropriate to their department/ agency—central agency provides both guidance as well as monitoring and oversight. • For evaluation, creation of the Centre of Excellence for Evaluation (CEE) has provided a system- wide champion to lead various capacity-building efforts (formalized training and workshops; a network/forum for heads of evaluation; and so on). • Professional association (Canadian Evaluation Society, CES) actively leads training and develop- ment and, recently, an initiative to identify competencies for evaluators. • For performance monitoring and reporting, TBS has issued and updated guidelines over the years, initially working with pilot departments and phasing in the introduction of reporting requirements for departments—intent was to implement in pilots, learn, adjust, and reissue guidelines on a broader basis. Who conducts • Internal evaluation units within government departments and agencies. evaluation studies? • New evaluation policy (2009) recognizes resource constraints for smaller departments and agen- cies—some special considerations, although evaluation is still a requirement. • Central agency (TBS) may dictate a department to undertake a specific evaluation or initiate a governmentwide evaluation—a new requirement in 2009. • Conduct of individual studies may be through use of internal evaluators, private sector consul- tants, or a combination. (continued on next page) 30 S ession 2 : I nstitutional A rrangements for M onitoring and E valuation S ystems TABLE 6: Key Features That Reinforce Credibility and Quality of the M&E System Element Description What quality control and • The TBS CEE monitors the planning and conduct of evaluation in individual departments, includ- oversight mechanisms ing evaluation coverage and quality of evaluations. exist? • TBS reviews annually the incoming performance reports (DPR) from each department and agency before their tabling in parliament • Through the MAF process, TBS assesses each department/deputy head annually against a number of criteria (including the use of results and performance information). • The Auditor General of Canada (AG) conducts periodic audits of how well the evaluation policy is being implemented across government, evaluates the state of results-based performance report- ing, and issues public reports to parliament on the findings and recommendations. • Formal requirement (2009) to conduct an evaluation of the government’s evaluation policy every five years. How is the evaluator’s • Evaluation function has “direct and unencumbered access” to the deputy head of the individual independence of department or agency. maintained? • Neutrality of both the evaluation function and the evaluator (“impartiality in behavior and pro- cess”) is stressed in the evaluation policy and by the government’s central policy center (TBS/CEE). • Each department has a senior-level Departmental Evaluation Committee in place that plays a variety of roles regarding evaluation planning, conduct, and follow-up. • Departments may make use of external review panels (independent experts, for example) for the planning and conduct of individual evaluation studies • Ongoing monitoring of the evaluation function and individual studies by the TBS/CEE—an operational-level oversight. • Oversight role played by the AG, an independent body reporting directly to parliament; public disclosure of information that reinforces both “independence” of the function and “transpar- ency.” How transparent is the • Evaluation policy stresses the importance of “transparency.” Formal requirement to post all information? evaluation reports on the department Web site. • Annual Departmental Performance Reports (DPR) are public documents. • Annual reports of the AG are public and generally receive wide attention in the media. • Access to Information legislation requires compliance with all external requests (general public, media, and so on) for public disclosure of government documents (some exceptions). • Over time, the media has made increasing use of this access to publicize certain findings of par- ticular audit or evaluation studies. Table 7 offers some lessons on implementing the M&E sults-based management (RBM) practices. These networks system from the Canadian experience. provide part of the enabling environment for professional development, but it is equally important that time and re- Building M&E capacity sources are set aside to allow for the necessary training and A critical element associated with the sustainability of an development. M&E system relates to the adequacy of human resources For its part, the TBS CEE has a capacity-building com- with the needed skill sets. Human resources capacity de- ponent as part of its mandate. This has resulted in the de- velopment has and continues to be an ongoing issue. Ca- velopment of tools, guidelines, and an evergreen Web site. pacity building is aided by the existence of well-developed Additionally, the CEE has sponsored capacity-building professional networks. Canada have two evaluation net- workshops, conducted local and national learning events, works. The first is within government, where heads of eval- and spearheaded national training programs for new eval- uation in departments meet on a regular basis. The sec- uators. ond is across the professional spectrum, with the Canadian The CES recently completed an exercise of identify- Evaluation Society (CES). CES is an active professional as- ing competencies for evaluators, as a first step in address- sociation representing evaluators from the public, private, ing the issue of ensuring a supply of trained and credi- and nonprofit sectors, including academics. A third asso- ble professional evaluators. Table 8 identifies a number of ciation, the Performance and Planning Exchange (PPX) lessons from the Canadian experience that relate to deal- brings evaluators and managers together on issues of re- ing with building M&E capacity. In considering training 31 M&E SYSTEMS AND M&E TOOLS needs, it is important to reflect on the training and orien- needs to address an often-ignored area—that is, data devel- tation to M&E required for nontechnical officials (the us- opment and credible databases. ers of M&E information). Additionally, building capacity TABLE 7: Implementing the M&E System Sufficient communication (and a forum for information sharing) is needed across organizations about the role of M&E and how it can help management to link the demand for and supply of M&E information—that is, to ensure that what gets produced is what is needed and delivered in a timely way. A formal policy document is a useful basis for clarifying roles, responsibilities, and accountabilities of key players—deputy heads, evaluation specialists, program managers, and central agency officials. The Centre of Excellence for Evaluation (CEE), the central agency champion for the evaluation function in government, has played a key role in the M&E system. It is the policy center for evaluation; it provides guidance; it leads and promotes capacity development; and it provides oversight to ensure quality control. The TBS has phased in a number of requirements under the general philosophy of “try, adapt, and learn.” This lengthens the period of building the M&E system, but allows for a period of learning and an ease of adjustment when needed without a major investment in reorienting the whole system. An acceptance that it is long-term and iterative manages expectations. In establishing internal evaluation units in departments and agencies, allowance for some flexibility is important to account for the unique circumstances associated with each organization. Deputy heads are given some flexibility in implementing the gov- ernment’s evaluation policy, although all are equally accountable for the performance of their organizations. Oversight by the national audit office is important in giving broad and public exposure of how well the M&E system is being implemented and whether adjustments are needed. TABLE 8: Building M&E Capacity Building an adequate supply of human resource capacity is critical for the sustainability of the M&E system and generally is an ongoing issue. It needs to be recognized that “growing” evaluators requires far more technically oriented M&E training and development than can usually be obtained with one or two workshops. Both formal training and on-the-job experience are important in developing evaluators. Two key competencies for evaluators are cognitive capacity and communication skills. Program and senior managers are important audiences for less technical training on M&E and RBM. They need to have enough understanding to trust and use M&E information. This type of broad training/orientation is critically important in building a results culture within organizations. There are no quick fixes in building an M&E system—investment in training and systems development is long term. Various op- tions for training and development opportunities include the public sector, the private sector, universities, professional associa- tions, job assignment, and mentoring programs. In introducing an M&E system, champions and advocates are needed to sustain the commitment needed over the long term. Identifying good practices and learning from others can help avoid the fatigue that typically accompanies any change process, as enthusiasm starts to wane over time. Evaluation professionals possess the necessary skill set to play a key role in providing functional advice and guidance to depart- mental/agency managers about the design and development of appropriate results-based performance monitoring systems. While managers should be responsible for performance measurement and monitoring per se, a recognized role for evaluators should be to provide such assistance and oversight on results measurement and monitoring. Ongoing performance monitoring and the conduct of ad hoc evaluation studies should be positioned to be complementary func- tions that support one another. Within organizations, this would mean that the measurement plans and strategies to put in place ongoing monitoring systems would be implemented so that sufficient performance information is available to effectively support the evaluation of programs. In so doing, organizations would make best use of the limited resources typically available for their measurement and accountability tools. Data quality is critical for the credibility of an M&E system, but an area often not adequately dealt with. A data development strategy (at an organizational level or a national level) should be considered as part of the long-term strategy of building a cred- ible M&E system. The national statistics office can be an important player in assisting data development. Performance reporting is often required at different levels—that is, providing information about a particular program, a sector, a department, or and agency, or at a national level. It is important to recognize the various audiences for performance reporting and their different needs and to tailor the level and breadth of reporting appropriate to those different levels. Source: Author. 32 S ession 2 : I nstitutional A rrangements for M onitoring and E valuation S ystems Conclusion comes and impacts. It promotes the concept of “account- “Getting it right” is the goal of every country implement- ability for results.” ing an M&E system. While each country has its unique cir- The Sri Lankan government fully recognizes the grow- cumstances, the lessons from any country can be instruc- ing international consensus that MfDR is an essential as- tive in helping identify best practices. In this regard, two pect of good governance to improve development effi- broad conclusions emerge from the experience of Canada ciency and effectiveness, transparency, accountability, and with M&E. The first is that the road to getting it right is informed decision making. In the recent past, monitor- long. The second is that the journey is important in itself, ing and evaluation (M&E) expanded globally and diver- for the learning that is gained along the way. sified in many contexts with many uses, such as decision making, organizational learning, knowledge base, program Robert Lahey improvement, policy development, impact and outcome President, REL Solutions Inc, Canada assessment, improved service delivery, accountability, per- formance audit, empowerment, and even transformation. Ambitious government systems with multiple stakeholders The Monitoring and Evaluation System in needs tend to achieve most of these desired uses. A good Sri Lanka: Experiences, Challenges, M&E system should go beyond institutional boundaries to and the Way Forward cover national, sectoral, program, and project levels to en- sure results orientation in government. Background Furthermore, Sri Lanka, as a signatory to the March Governments are challenged to respond to the urgency 2005 Paris Declaration on Aid Effectiveness, is commit- of citizens’ needs and to be more accountable to them. ted to institutionalizing MfDR throughout the govern- All over the world, the public is putting governments un- ment. The MfDR process typically involves several phases der growing pressure to show that they are providing good such as articulating and agreeing on objectives, selecting value for money and results. In the past three to four de- Key Performance Indicators (KPIs), setting targets, mon- cades, significant budgetary resources and efforts have itoring performance, analyzing and reporting on results been deployed to accelerate development, reduce poverty, against targets, and facilitating management to take time- ensure equality, and improve social living standards and ly corrective actions. Different countries will approach quality of life of the people. With fast tracking of public MfDR differently, as it is not a one-size-fits-all model. investment programs, Sri Lanka has been able to main- Ultimately, it should lead to sustainable improvement in tain its average annual economic growth rate at 5 percent. the country’s development outcomes. The government of However, the country experienced challenges in translat- Sri Lanka is committed to promoting the application of ing this economic growth momentum into poverty reduc- MfDR principles at various levels, namely, the national, tion. The level of poverty remained at 26 percent to 22 sectoral, agency, and project level and at the planning, percent over the last two decades. This situation led to implementation, and postimplementation stages. At the the need to strengthen the government’s planning, mon- planning stage, the results-oriented country-owned Na- itoring, and evaluation system to focus on delivering of tional Development Plan and sectoral plans are being outcomes and impacts beyond the traditional output fo- aligned with the Medium-Term Expenditure Framework cus. The concept of Managing for Development Results (MTEF), and the use of performance budgeting instead (MfDR), otherwise known as results-based management of line-item budgeting has increased. Line ministries are (RDM), has been adopted gradually since the mid-1990s. required to justify their budgets with well-defined out- MfDR is about public sector management, and the con- put and outcome indicators. Today, what counts is not so cept strongly emphasizes that shared vision, clear goals, much how many clinics have been built, but whether cit- and measurement of results would lead to a better future. izens’ health has improved; not how many schools have MfDR is a change management process that emphasizes a been constructed, but how many girls and boys are bet- shift in focus from inputs, activities, and outputs to out- ter educated. 33 M&E SYSTEMS AND M&E TOOLS Sectoral plans use outcome-based KPIs in setting tar- • Deliver what they planned in a manner consistent gets and directions. The Department of National Budget with public service ethics, values, and standards and the Treasury have revised their budget circulars to fo- while meeting standards such as timeliness, quali- cus on “results” to institutionalize performance budgeting ty, quantity, and within cost systems. The government’s three-year MTEF incorporates • Track their progress by monitoring, measuring, outcome-based key performance indicators to justify pub- and evaluating, learn from success and failures, and lic expenditures, and this helps achieve results. make continuous improvements MfDR in Sri Lanka: A whole-of-government The MfDR approach adopted in Sri Lanka is more of a approach whole-of-government approach (see Figure 6) covering the The institutionalization of MfDR is seen as a major shift in national, sectoral, institutional, and project level. A results focus in the M&E arrangements in Sri Lanka. MfDR in- focus is being built into the National Development Strate- cludes a management cycle of setting directions, planning, gy (NDS), sectoral plans, ministries, and projects. implementation, and delivering and reviewing the results, which then feed back into the cycle to improve planning Enabling environment and to making ongoing improvement. Figure 5 represents In Sri Lanka, the M&E of projects, programs, and develop- the localized version of the MfDR process in Sri Lanka. ment initiatives of the government is not a new phenom- In the Sri Lankan context, the organizations that are enon. Efforts to improve planning and project implemen- managing for results must tation have been a feature of development efforts since the early 1970s. One of the special features in Sri Lanka is the • Have a clear vision of what they want to achieve functioning of a separate Ministry of Plan Implementation • Plan their work, keeping in mind the clear vision (MPI) charged “as a National Focal Point for Monitoring and mission and Evaluation of all government development projects and programs to ensure achievement of results and devel- opment effectiveness.” The MPI is headed by a senior cab- FIGURE 5: FIGURE 6: MfDR in Sri Lanka Whole-of-Government Approach Direction Setting What do we intend to achieve over next 3–5 years? NDS Identify Vision, Mission, Thrust area, Goals National Sector and KPIs Audits Plan Review Planning What impact have our What is the best way interventions had? Continuous to achieve Assess Efficiency, Results Identify strategies Effectiveness, Impact Performance and Actions MTEF/ and Sustainability Monitoring & Budget Implementation Evaluation Are we implementing as planned and managing risks Projects Ministries & Deliver interventions & Departments and Monitor Progress Programmes (Service Delivery) Source: Author. Note: KPIs = Key Performance Indicators; MfDR = Managing for Devel- Source: Author. opment Results. Note: NDS = National Development Strategy. 34 S ession 2 : I nstitutional A rrangements for M onitoring and E valuation S ystems inet minister and is mandated with the responsibility of and impact evaluations, the mainstreaming of the project monitoring and evaluating all government policies, pro- performance management system, results-based manage- grams, projects, and institutions. The Department of For- ment (RBM), performance budgeting and performance au- eign Aid and Budget Monitoring (DFABM) is the func- diting, localization of the Millennium Development Goals tional arm of the MPI to provide technical support and (MDGs), and results reporting through country-based undertake the M&E functions with skills and technical M&E systems—have been moving forward on a gradual know-how. This dedicated institutional arrangement plac- basis since early 1990s. A strong link is evident among the es M&E more strongly in the institutional setting of the MPI, the Ministry of Finance and Planning, and the Pres- government of Sri Lanka. ident’s Office: they work closely with each other on M&E One of the key features in Sri Lanka is the top-lev- strategy. The line ministries, agencies, project management el commitment for M&E with a focus on MfDR. In the units, provincial councils, and district/divisional secretar- 1990s, with the technical support of the Asian Develop- iats work closely with the MPI and provide progress re- ment Bank (ADB), the postevaluation system was strength- ports in specified formats on a periodic basis. His excellen- ened in the MPI. As a result, the MPI conducted a number cy the president chairs the progress review meetings, which of post evaluations of projects and programs. In late 1990s, are held on a regular basis to assess progress of develop- the United Nations Development Programme (UNDP) ment projects, programs, sectors, ministries, and institu- provided significant technical support way to strength- tions. This high-level forum is a guiding and troubleshoot- en the results-based M&E system in Sri Lanka. This en- ing forum with top-level political commitment. abled the government officials at the national and subna- The Cabinet of Ministers made a formal decision and tional level to understand and recognize the importance of directed the MPI to monitor all development projects over results-focused monitoring. Also many positive factors con- Sri Lankan rupees (SL Rs) 50 million executed by the line tributed to a positive enabling environment, such as (1) ministries and submit quarterly progress reports to the political will, (2) overarching policy, (3) coordination to Cabinet of Ministers. The MPI has developed an online, collect information, (4) flow of information from line min- real time Web-based electronic monitoring and reporting istries and projects to the MPI/DFABM, (5) strengthen- system to collect, analyze, and report progress of all devel- ing of the electronic information management system in opment projects over SL Rs 50 million. Progress review the National Operations Room (NOR) of the MPI, and meetings are held by the MPI with project management (6) demand for information for decision making. However, units and officials of line ministries on projects that are be- capacity in government agencies, a large number of minis- hind schedule to address issues and ensure efficient and ef- tries, and the resultant coordination issues are some of the fective implementation. The MfDR principles increasingly challenges that need to be addressed. are being applied in the national M&E systems and relat- The MPI has introduced MfDR in government to track ed areas. Some of the major achievements and lessons are development results of the line ministries and their pro- discussed below. grams. Logical Framework Analysis (LFA) and Results Frameworks (RF) increasingly are used in planning and Governmentwide performance measurement M&E arrangements within government. All these devel- system and scorecards opments clearly indicate that the concept of “Managing for The Sri Lankan government has introduced MfDR with Results” set out in the Paris Declaration is being institu- the technical support of the UNDP to track performance tionalized and gradually moving forward in Sri Lanka. of ministries and institutions using output and outcome in- The MPI and its DFABM championed the institution- dicators. This new management style holds ministries and alization of MfDR within government with the belief that departments and public officials accountable for results the process has potential to ensure results through a shift in rather than efforts. At a macro perspective, ministries and focus of public sector management from inputs and activi- departments receive budget appropriations from parlia- ties to outcomes and results. Many initiatives—such as the ment to carry out specific mandates. These mandates have institutionalization of postevaluations, process evaluations, to be translated into detailed management expectations. 35 M&E SYSTEMS AND M&E TOOLS The results framework of the ministry or department sets the technical support of the UNDP and theADM. Cur- out the breath, depth, and meaning of management expec- rently, MfDR has been expanded and is operational in 35 tations. By measuring performance against management line ministries. An MfDR core group has been established expectations, set out in the results framework, the institu- to drive this initiative with the leadership of MPI secretary. tion will be in a better position to objectively assess the re- The experiences of the U.S. RBM models operated in Or- sults achievements. MfDR translates the vision and mission egon (Oregon Benchmarks), Minnesota (Minnesota Mile- of public sector agencies into the practicalities of managing stones), and Virginia, as well as the Canadian models were the organization better at every level. MfDR translates the taken into consideration in developing a localized model. vision and mission of public sector agencies into the prac- The performance tracking system in Sri Lanka will be ex- ticalities of managing the organization better at every lev- panded to all ministries. el: it defines expected results, monitors and evaluates prog- The MPI has established a Web-based MfDR platform ress toward the achievement of expected results, integrates in its information arm, the NOR, to maintain the agen- lessons into management decisions, and reports on per- cy’s results framework and scorecards, or report cards, for formance. MfDR when in operation ensures line-of-sight, each line ministry. The agency results framework (ARF) which implies everyone in the organization understands and scorecard are the centerpieces of the government’s Mf- the strategic vision and mission irrespective of their level DR (see Figure 7). The ARF sets out the ministry’s mission and position in the organization. and its core business (thrust areas, goals, key performance A comprehensive performance measurement system was indicators) with baselines and medium-term targets. The piloted in 2006–07 with four key line ministries (Educa- customized scorecards and report cards will appear on the tion, Health, Agriculture, Highways). The selection of pi- computer screen to enable the assessment of the line min- lots was initially based on the interest shown in the reforms istries’ performance. The scorecard employs a red-yellow- and the commitment of the leadership of the ministries to green grading system to track performance of government the change management initiatives. A range of activities institutions. This “dashboard” is an early warning signal to such as awareness programs, advocacy and sensitization to alert when the achievement of targeted outcomes expressed policy makers and training programs were conducted with by the KPIs is slipping. FIGURE 7: Snapshot of the Agency Results Framework Thrust area 1 – Current and preventive health services Strategic Objective – Provision of comprehensive Health service delivery and health actions Targets Key Performance No. Goal Indicators Base line 2006 2007 2008 2009 2010 1 Reduction of infant Infant mortality rate 11.2/1000LB (2002) 11 10.8 10.6 10.4 mortality 2 Reduction of under five Under five mortality rate 4.4/1000 Under five 4 3.8 3.6 3.4 mortality population 1997 (AHB 2003) 3 Reduction of maternal Maternal mortality rate 38/1000000 36 35 33 31 mortality 4 Reduction of % of underweight 29.4% (2000) DHS 25% 24.5% 24% 23% prevalence of children (W/Age) under underweight five years (malnutrition) children under five years 5 Reduction of incidence % low birth weight 17.6 (20005) AHB 16.5 16 15.5 15 of low birth weight babies babies Source: Author. 36 S ession 2 : I nstitutional A rrangements for M onitoring and E valuation S ystems A snapshot of the Ministry of Health’s ARF is present- As in many developing countries, the availability of ed in Figure 7. Under the thrust area of curative and pre- trained M&E personnel is a key constraint in Sri Lan- ventive health services, the goals and KPIs are infant mor- ka. Also, incentives need to be in place to reward success. tality rate, under-five mortality rate, maternal mortality MfDR, when fully operational, will help the government rate, percentage of underweight children, and percentage strengthen performance accountability and improve a con- of low-birth-weight babies. The medium-term KPI target tinuous learning culture. in the ARF enables a performance assessment of the Min- istry of Health in each relevant thrust area. The framework Readiness assessment is not carved in stone. It will continue to evolve as minis- With the support of the ADB, a readiness assessment on tries and departments strengthen their management prac- MfDR was conducted in key ministries and agencies to as- tices. The MfDR is to be used to improve results reporting sess the capacity gaps. Based on the readiness assessment to the cabinet and to the parliament. The scorecard raises a findings, a capacity development plan has been developed “red flag” when something is wrong. The Sri Lanka MfDR to prioritize capacity-building initiatives in the selected in- model follows the North American performance measure- stitutions. The capacity development plan helps to address ment framework models and is further articulated through capacity gaps at the institutional level to enable the imple- a plan-do-check-adjust cycle, as depicted in figure 8. mentation of MfDR. The capacity development plan in- cludes initiatives such as strengthening operating environ- • The plan stage defines the desired results. Perfor- ment, clarifying the organizational results, aligning results mance expectations capture key dimensions of per- to systems and processes, and upgrading management in- formance of interest to important stakeholders. formation systems to improve data collection, analysis, and Agreement of expectations implies a shared vision reporting. of what is to be accomplished. • At the do stage, the ministries’ strategies and actions Implementation of MfDR in government are linked to the desired results. The minister of plan implementation placed MfDR in • At the check stage, KPIs are used to evaluate per- the policy arena by initiating a Note to Cabinet on Mf- formance. DR, in which he outlined the importance and the steps • At the adjust stage, resources are adjusted and necessary to institutionalize MfDR on a government-wide streamlined with the strategies and action to scale. The responsibilities for the institutionalization pro- achieve desired results. cess were assigned to the MPI. The technical backstopping �� of the DFABM was complemented by the advisory tech- nical support of the UNDP and the ADB. The ministry FIGURE 8: took the following steps to institutionalize MfDR govern- Agency Performance Measurement Framework mentwide. Define Desired Results: strategic direction and outcome oriented goals Launching of the process PLAN The launching of the process involved many key sequen- Adjust to Link Actions tial initiatives, including the establishment of a core group Improve: to Results: adjust business to drive the initiative, endorsement of government policy priorities, move plans and resources, ADJUST DO performance on MfDR, formulation of a governmentwide strategy, for- eliminate or targets streamline mulation of a “change agent” concept to establish owner- actions ship and leadership in line ministries and agencies to take CHECK the initiative forward, and support by advocacy and capac- Evaluate Performance: KPMs and ity development to sustain the initiative. The approach ad- other tools that inform on how you are performing opted by MPI in launching the institutionalization of the Source: Author. MfDR is discussed below. 37 M&E SYSTEMS AND M&E TOOLS Champion the initiative Change agent concept: Leadership at all levels of The MPI as an apex ministry responsible for M&E, cham- government pioned the institutionalization of MfDR government-wide. The MPI provided the technical support to introduce the A core group was headed by the secretary, MPI, and includ- MfDR methodology and approach. Each ministry was en- ed representatives of the Ministry of Finance and Planning, couraged to establish a steering committee of senior officers Department of National Planning, Department of Nation- who were expected to function as “change agents” to take al Budget, the AG’s Department, the President’s Office, the this change management initiative forward. The change Prime Minister’s Office, Department of Census and Statis- agents functioned as drivers of change and helped to facil- tics, and selected secretaries of the line ministries. The core itate the MfDR initiative at the ministry level. The change group was to provide guidance, direction, and leadership agent concept clearly shows the importance of leadership at to the initiative to mainstream the MfDR in government all levels of government to take this effort forward. and to function as a think tank to formulate strategy and an action plan. The concept of core group helped to estab- Adoption of a process approach: lish governmentwide commitment and ownership of the Consensus building process. The core group formulated the strategy and action In each case, the ARF of each ministry was developed by plan to institutionalize MfDR with the technical inputs of that ministry through a participatory process with the the DFABM of the MPI. MPI’s technical guidance. Each ministry assumed owner- ship and the process approach helped to create this own- MfDR as a government policy ership versus a product approach. Workshops were held The Cabinet of Ministers approved the MfDR initiative in the line ministries by involving all key players, and the and empowered the MPI as a lead organization to lead consultations gave a sound basis for ensuring wider own- this initiative governmentwide. The top-level political ership. They helped to tap the expertise, skills, and expe- support, which comes through the endorsement by the rience on one side and the trust, support, and network on Cabinet of Ministers, indicates the political will and pol- the other side. icy commitment to take this initiative forward. The MPI initiated a planned program of advocacy and sensitization Buy-in at policy level. Dr. Ray Rist, advisor to the World Bank, The process approach and the involvement of senior gov- was invited to conduct an initial orientation and sensiti- ernment officials and other relevant stakeholders created zation program, which helped to share international expe- wider ownership, which contributed as a positive factor for rience and raised the importance of RBM/MfDR at the the buy-in ensuring sustainability of the initiative. policy level in government. At the policy level, a number The ministry-level MfDR formulation workshops in- of programs were conducted to sensitize the policy mak- volved active participation of the top-, middle-, and low- ers and senior government officials on the importance of er-level staff of the institutions, and this enabled the op- the MfDR initiative. erationalization of the concept of line of sight. Moreover, other stakeholders also were consulted in this process. MfDR strategy and action plan The ARF of the line ministry will be closely integrated The strategy provided phased approaches with a pilot into the national budget to make it a performance budget, phase involving four line ministries and the MPI. Based and thus the initiative has built in factors of sustainabil- on the lessons of the pilot phase, the MfDR institutional- ity. Moreover, the link with the AG to measure perfor- ization was expanded to cover 35 line ministries. A com- mance through performance audit arrangements further prehensive strategy and action plan was developed, which strengthens the MfDR initiative as a national process. Ex- included capacity development, staff training, advocacy, pected results are mutually defined and agreed on through strengthening information systems, methodology, and re- a consensus-building process involving all major stake- porting arrangements. holders. This enhances the stakeholder’s sense of owner- ship. 38 S ession 2 : I nstitutional A rrangements for M onitoring and E valuation S ystems Country-level community of practice to facilitate • Setting targets peer-to-peer dialogues Set explicit medium-term targets to provide medium term- The core officials of the ministries to which MfDR was in- directions and to judge performance. This is the main as- troduced formed into a community of practice and shared pect of the process for each indicator. their experiences and expertise through a face-to-face and peer dialogue. This helped to establish connections among • Monitoring performance the “islands of best practices” and helped participants learn Monitor performance to regularly collect data on actual from successes and challenges. Moreover, a quarterly news- results. letter on “Results Focus” helped to complement and sup- plement the face-to-face community of practice dialogue. • Analyzing and Reporting Review, analyze, and report actual results as against the Capacity building and readiness assessment targets. Capacity building is fundamental to institutionalizing Mf- DR in government. The readiness assessment tool devel- • Integrating evaluation oped by the ADB has been used to identify capacity gaps Evaluate to provide in-depth analysis and to provide com- in line ministries and to formulate a capacity development plementary performance information not readily available plan. The UNDP and ADB support helped formulate a from performance monitoring system. capacity development plan as a complementary activity to institutionalize MfDR. Awareness creation, staff capacity • Performance reporting and feedback building, and on-the-job training on MfDR methodolo- Use performance information for internal management ac- gies, tools, techniques, and practices were emphasized in countability, learning and decision-making process and for the implementation of capacity development plan. performance reporting to stakeholders. The first three steps generally relate to a results-orient- Strategy to action ed planning approach, sometimes referred to as “strategic The MfDR approach encourages ministries and depart- planning.” The first five steps together are usually includ- ments to understand the results chain and establish logical ed in the concept of performance measurement. All steps links between planned inputs, expected activities/outputs, combined are essential to an effective results-based man- and envisaged outcomes based on the theory of change. agement system. The MPI took the following key steps to operationalize The NOR is the information platform of the MPI to MfDR in line ministries. support evidence-based decisions. The MfDR Web-plat- form was established in the NOR to facilitate and track re- Articulating and agreeing on objectives sults and performance of line ministries. Identify clear and measurable objectives (results) aided by In the past, the progressive monitoring of ministries fo- the results chain-logic model. This basically involves prepa- cused heavily on financial progress (budget utilization) and ration of a vision statement, mission statement, thrust areas, physical progress (activity monitoring) with little emphasis and goals. The government’s 10-year national development on achievement of outcomes, results, and policy objectives. framework, sector plans, and localized MDGs and mandate However, implementation issues and interagency coordi- of the ministries are considered to be the fundamental basis nation issues were addressed through the high-level mon- to articulate and agree on line ministry objectives. itoring meetings held by the president. The MPI submits, on a quarterly basis, to the Cabinet of Ministers, a prog- • Selecting indicators ress report on the capital budget of all line ministries and Identify output and outcome-based KPIs to measure prog- development projects over SL Rs 50 million. In the recent ress toward each objective. KPIs for each expectation in the past, the emphasis was more on synthesis-type reporting framework are meant to convey the breadth and meaning rather than on the traditional way of producing large vol- of the expectation. umes of reports that resulted in information overload. The 39 M&E SYSTEMS AND M&E TOOLS institutionalization initiative of MfDR is a major shift in to the Cabinet of Ministers on a quarterly basis to fulfill the focus of the government’s M&E system. reporting requirements. The MfDR initiative is to be complemented with per- The ePMS captures feedback from beneficiaries and cit- formance agreements to ensure ministries and institu- izens. Notable features of the system are as follows: tions manage priorities by establishing strong links be- tween agency plans and employee performance. While • Keeping track of the development project progress the MPI is introducing MfDR to ministries and depart- electronically covering financial and physical prog- ments, the Department of Public Enterprises of the Min- ress. istry of Finance and Planning has introduced MfDR as • Harmonizing results reporting through results part of its corporate plan initiative to state-owned enter- frameworks. National ePMS is a homegrown, prises (SOEs) and statutory bodies. Advocacy and aware- country-owned, country-driven system that pro- ness-building efforts on MfDR are in progress. Spending vides the public sector with paperless monitoring money is no longer sufficient evidence for results. Minis- capability. tries and agencies are asked to demonstrate how their pro- • Facilitating pictorial proof to demonstrate progress grams and strategies are effective in producing the intend- and results at ground level. ed results. According to Osborne and Gaebler (1992), • Capturing data at the source, reducing transaction “Perfectly executed process is a waste of time and mon- costs. ey if it fails to achieve the outcomes desired.” Performance • Providing alerts and early warnings on shortfalls management redirects our efforts away from “busyness” and gaps, including time and overruns; providing towards “effectiveness.” Being busy is not the same as pro- red alerts on problem projects and projects behind ducing results. schedule. • Advising the higher authorities of the problems, is- Electronic project monitoring system: sues, and constraints faced in the implementation A distinctive feature in Sri Lanka of development projects and facilitating trouble- One of the noteworthy and significant aspects is that a shooting of problem projects and projects behind homegrown, user-friendly, national, Web-based online schedule. electronic project monitoring system (ePMS) has been es- tablished in the DFABM to track the implementation A recently completed donor evaluation mission rat- progress (financial/physical) and results of all development ed the ePMS a success story in terms of its comprehensive projects and programs. National ePMS is a proactive, Web- coverage, periodical updating, and use of information for based project monitoring system that tracks whether devel- troubleshooting. However, the current low levels of use of opment projects and programs are implemented according the system by sector ministries indicate an unexploited op- to the agreed work plan in a results-oriented manner. The portunity. The Ministry of Finance and Planning is using system provides access to project information on a donor, the system as part of its efforts to strengthen project man- sector, and ministry basis.14 agement capacities. The MPI uses the system for trouble- The system uses early warning (traffic lights) signals shooting. Also, it is necessary to enhance “data capture at that enables “problem projects” to be separated from oth- source.” The system allows for better overall coordination ers. The system helps to identify bottlenecks, delays, issues, and helps to address execution problems. and constraints in the implementation of projects and any The national ePMS was based on the Malaysian Na- additional needs of the executing agency. It includes re- tional Operations Room Model used by Prime Minister sults monitoring using logical framework analysis, moni- Dr. Mahathir Mohamed to track implementation prog- toring compliance of loan covenants, tracking cash flow, re- ress of all development programs. The Sri Lankan ePMS imbursable foreign aid claims, procurement progress, and was a homegrown model with a user-friendly facility to major issues with pictorial proof. Flash reports on problem track development project progress. It is not intended to projects help in troubleshooting exercise and are submitted be used as a “policing function” but as a system that iden- 40 S ession 2 : I nstitutional A rrangements for M onitoring and E valuation S ystems tifies projects on schedule or behind schedule or prob- Procurement monitoring lem projects. The system depicts these projects using a Project management mainly involves managing con- traffic light color code by sector and ministry classifica- tracts. Hence, procurement management is an important tion of projects and facilitates troubleshooting. Current- aspect. Procurement monitoring keeps track of the pro- ly, the national ePMS includes more than 120 large and curement progress, which is a core area in all develop- mega projects. ment projects. Key elements Monitoring major issues The national ePMS captures many key elements that are The system keeps track of the major issues affecting the fundamental and essential to track the progress of devel- implementation of projects and action taken on such is- opment projects. The key elements are the project pro- sues. This facilitates troubleshooting. The nine knowledge file, financial progress, activity monitoring, compliant management areas identified by the International Project of loan covenants, procurement monitoring, and ma- Management Institute are (1) scope management, (2) time jor issues. management, (3) cost management, (4) quality manage- ment, (5) human resources management, (6) procurement Project profile management, (7) risks management, (8) communication The profile provides the basic information of the proj- management, and (9) integration management. ect, its name, source of funding, implementing arrange- The current national ePMS of the MPI integrates all ments, objectives, purpose and outputs, location, cost, nine knowledge management areas in the system. To sup- timelines, and officer responsible, including other basic port the governance functions, the major functions of information. ePMS are as follows: Financial progress • Serve as an executive information center for central This element tracks the cumulative financial progress as agencies such as the President’s Office, the MPI, the well as the monthly financial progress against total funds Ministry of Finance and Planning, and other line available and the annual budgetary targets. It monitors ac- ministries. tual disbursements against targeted disbursements on a • Provide information to project managers to man- projectwide basis. age projects effectively. Hence, the ePMS supports managerial functions. Activity monitoring • Ensures access to information to citizens and create The projects outputs are broken down into components, transparency and accountability. subcomponents, activities, and subactivities using work breakdown structures. All activities have planned time- Figure 9 depicts the flow of capturing data from its lines and targets and progress is monitored against such source to its end use. The project information is captured timelines. at source from project office and stored in the Web serv- ers and finally shared with potential users at various levels. Logical framework analysis Hence the system is a decision support system and execu- The results framework of the projects shows the “program tive information system. theory” with a results chain and measurement system, in- The ePMS has many salient features. It helps to create cluding KPIs and risks. The system helps to track results online information any where, any time and enables infor- and major risks. mation to be projected on to a large screen and used for discussions at meetings. The potential users of the system Monitoring compliance of loan covenants include apex agencies such as the President’s Office, the This tracks the compliance of loan covenants of all donor- Presidential Secretariat, the MPI, and the Ministry of Fi- funded projects. nance and Planning, line ministries, project directors, and 41 M&E SYSTEMS AND M&E TOOLS FIGURE 9: Bank of Sri Lanka has developed the prosperity index to Data Capture at Source capture wider aspects of development. Data Captured at Source: Project Office feeding Information to ePMS Poverty monitoring and tracking of MDGs The MPI also monitors the progress of national devel- 1. Data Entry National opment goals. The MDGs are the best known and most at source Agencies ambitious global commitments to results-based develop- 3. Shared data ment management. The agreed MDGs, targets, and in- available at P P various levels dicators provide the basis for measuring progress. It col- M M (DSS) S S lects performance indicators to measure achievement of localized MDGs and poverty with a focus on the “big • Project Office Project picture,” with the support of national statistics and sur- Directors 2. Data stored in the veys (such as Household Income and Expenditure Sur- Web Server Citizens vey and Demographic Health Survey) from the Depart- (Restricted) ment of Census and Statistics. The government prepares Source: Author. an annual MDG country report to identify the gaps, pov- erty pockets, and regional disparities. This M&E infor- citizens (restricted access). The system’s special report gen- mation helps governments to allocate budgetary resourc- erating facilities include the following: es for needy areas. • Flash reports (higher level reports to senior policy National evaluation arrangements: ongoing, makers on overall financial progress, physical prog- ex post, and impact evaluation ress, status of projects, major issues, and action tak- The DFABM undertakes ongoing, ex post, and impact en in the implementation of projects) evaluation of selected mega projects and disseminates eval- • Summary reports (reports on aid utilization and uation findings to concerned stakeholders. The evaluation budget progress, accessed by ministry, sector, do- function of the MPI was strengthened through the techni- nor, and project by various monitoring authorities) cal assistance support of the ADB. The Organisation for • Tailor-made reports (using the crystal report fa- Economic Co-operation and Development’s Development cility) Assistance Committee (OECD/DAC) evaluation criteria, such as the strategy’s relevance, efficiency of implementa- Outcome-based national economic performance tion, effectiveness, impact, and sustainability, are given due Under the traditional national economic performance re- consideration in undertaking the evaluation. The OECD/ porting system, high economic growth, low inflation, and DAC evaluation quality standards are considered to main- unemployment were regarded as indicators for healthy eco- tain the quality of evaluations. On invitation by donors, nomic climate and believed to result in prosperity for cit- the DFABM participates in joint evaluations with donors izens. However, citizens are increasingly concerned about to create national ownership and build local capacity in their quality of life measured in terms of quality of edu- evaluation. The department also undertakes diagnostic rap- cation, health care, safety from crimes, clean environment id assessment through field visits of problem projects and (safe drinking water and sanitation), and the like. Hence, submits flash reports to the ministry’s secretary to trouble- traditional economic-based measurement systems failed to shoot projects that are behind schedule. Given the human address such quality-of-life issues and outcomes. The Sri resource constraints in the ministry, it was decided to out- Lankan government has established a results reporting sys- source the evaluation of priority mega projects. The minis- tem for the localized MDG. The Department of Census try, in consultation with the relevant line ministry, decides and Statistics supports the collection of the data, which are areas to be covered in the terms of reference for evaluation. widely used within the government. Moreover, the Central Also, the ministry identifies the information needs of the 42 S ession 2 : I nstitutional A rrangements for M onitoring and E valuation S ystems president, the cabinet, and other stakeholders in designing tal to improving “aid effectiveness” and development effec- the terms of reference. The findings of such evaluations are tiveness (see Figure 10). disseminated to the line ministries and project offices for �Hence, with UNDP’s support, the MPI undertook an necessary follow-up action. independent evaluation to assess the implementation prog- A key problem in many countries has been the inabil- ress and results of the Paris Declaration commitments. An ity to access the evaluation information available on vari- evaluation reference group was formed to advise, and a ous development projects and program either completed or management group coordinated the evaluation. This eval- currently being implemented. Such evaluation lessons and uation identified the impediments to implementation, and findings are important and useful to improve the quality of helped the Sri Lankan government take suitable action to new projects and programs, especially because it helps to ensure effective implementation of the declaration’s prin- avoid past mistakes and build on best practices in the for- ciples. Moreover, the evaluation findings have fed into the mulation and designs of new projects. global evaluation process, enabling the OECD/DAC De- velopment Evaluation Network to undertake a synthesis of Evaluation information system the declaration’s evaluation in eight countries and 11 do- Having recognized the importance of a systematic use of nors. These findings were useful in the formulation of Ac- evaluation and feedback arrangements, the MPI has tak- cra Agenda for Action at the High Level Forum on Aid Ef- en action to establish a Web-based evaluation information fectiveness held in September 2008 in Ghana. system (EIS) to ensure effective dissemination of evalua- The Accra High Level Forum has endorsed the need for tion findings, lessons learned, and synthesis of such find- aid predictability, use of country systems by donors, un- ings. Such evaluation information will provide sectorwide tying of aid, and mutually agreed conditionalities instead synthesis to ensure more effective feedback and assist in in- of imposed conditionalities as fundamental action required tegrating evaluation findings into the planning, budgeting, to accelerate aid effectiveness. This implies that aid effec- and policy-making process. Also public availability of the tiveness requires radical change in behaviors of donors and evaluation reports through the EIS is expected to improve development partners. Implementation of the Paris Dec- public accountability and transparency. laration Principles and the Accra Agenda for Action is Evaluation asks what works, what does not work, why, and in what context. The responses are important for plan- ning and programming and contribute to development ef- FIGURE 10: fectiveness. The EIS enables development practitioners to MfDR and the Paris Declaration access evaluation information anywhere, anytime—thus Managing for Development Results empowering them to make evidence-based decisions in de- velopment. The establishment of an EIS is considered a Priority Action: Ownership critical milestone in the MfDR initiative as it is expected • Medium Term Predictability for not only to improve aid effectiveness but also to promote Multi Year Planning Partners and Budgeting a learning culture. The support toward the development of set the agenda EIS was extended by the UNDP. • Use of Country Systems Alignment Align- Relying • United Aid to improve Mu Evaluation of the implementation of the Paris ing with on value for money tua partner’s partner’s Declaration on aid effectiveness l Ac • Mutually Agreed agenda systems Conditionalities cou The MPI and the DFABM undertook an evaluation of the Harmonisation nta implementation of the Paris Declaration on Aid Effective- Establish- Simplifying Sharing bili • Information ing procedures information t ness. The government of Sri Lanka strongly believes that Sharing y common the five principles of the Paris Declaration, such as national arrangements ownership, alignment, harmonization, managing for devel- opment results, and mutual accountability, are fundamen- Source: Author. 43 M&E SYSTEMS AND M&E TOOLS fundamental to the change management process and im- • Ensure an appropriate balance between “monitor- proved development effectiveness. ing” and “evaluation.” Preferably separate evalua- tion from monitoring to ensure balanced resource Sri Lanka Evaluation Association allocation for evaluation. The Sri Lanka Evaluation Association (SLEVA), as a civ- il society, plays a catalyst role in advocacy, awareness cre- Sound institutional arrangement ation, training, and developing standards, ethics, method- • Strengthen institutional arrangements to place ologies, and best practices to improve evaluation culture. M&E and MfDR in a strategic context. Its membership includes academia, researchers, private sec- • Establish links between evaluation exercise and tor consultants, government officials, and nongovernmen- performance audit exercise by encouraging part- tal organization (NGO) representatives who are interested nerships between evaluation institutions and per- in evaluation. SLEVA works closely with the MPI in build- formance audit institutions (that is, the AG’s De- ing M&E capacity and culture in such areas as evaluation partment) with regard to accountability-oriented training, sharing evaluation, best practices, organization of evaluations. evaluation forums to discuss evaluation topics, internation- al and national conferences on evaluation, and promoting Standards, ethics, and guidelines community of evaluation practitioners. (quality of evaluations) • Develop evaluation standards, guidelines, and eth- Strategies to strengthen the national evaluation ics to ensure quality evaluations. Ensure scop- system ing sessions are conducted to clarify the evaluative The following strategies have been identified to strengthen questions and to ensure that the needs of the poten- the MPI’s National M&E system. Currently, the UNDP tial users are taken into consideration and timing of technical assistance project has provided assistance to sup- evaluation is appropriate. port these strategies. • Encourage national evaluation association to ac- tively promote evaluation culture and MfDR con- Policy commitment and support cepts. • Advocate and sensitize at political and policy level • Develop standards and criteria for good evaluation on the importance of results-based M&E and en- in collaboration with civil society such as SLEVA sure its acceptance and placement of M&E in key and undertake meta-evaluations to ensure quality decision-making centers of the government to cre- evaluations. ate local demand for M&E. • Ensure that M&E institutions are linked to the Strengthen evaluation guidelines and systems planning, budgeting, resource allocation, and pol- • Strengthen localized guidelines for systematic eval- icy functions of the government and MfDR con- uations. cepts are integrated in all areas of the development cycle. Strengthen methodologies and practices • Make evaluation a process within the development Legal and budgetary support policy and project cycle. Expand evaluation to cov- • Develop a legal foundation to make M&E and er projects, programs, sectors, policies, and insti- MfDR mandatory. Use law, decree, cabinet deci- tutions. Encourage synthesis of project evaluations sion, or other high level pronouncement to legiti- to provide sectorwide learning. Promote cost-effec- mize MfDR concepts and results-based M&E sys- tive rapid assessment methods under time, budget, tems. and resource constraints. Also, consistent, localized • Provide sufficient financial allocation for MfDR evaluation methodology and terminology are es- and M&E strengthening in the line ministries. sential. 44 S ession 2 : I nstitutional A rrangements for M onitoring and E valuation S ystems • Reexamine the approaches and tools for evaluating Success factors the multiple dimensions of development. Encour- The ultimate success of evaluation depends on how well the age the use of diverse or multiple methods, as well planners and decisions makers utilize the valuable M&E as participatory methods for lessons learning (utili- findings and lessons to improve future programs, projects, zation) oriented evaluations. policies, and institutions. • Encourage more joint evaluations instead of donor- Sri Lanka’s MfDR and M&E systems and practices have driven evaluations. been internationally recognized as best practice approach- es worthy of scaling up. Senior government officials from Evaluation capacity development Afghanistan, India, the Republic of Yemen, and Uganda, • Strengthen the professional evaluation capacity as well as representatives from the Asia Pacific Community within the government through continuous staff of Practice on Managing for Development Results and In- training. ternational Program for Development Evaluation Training • Promote in-country evaluation faculty develop- (sponsored by the World Bank in partnership with Carl- ment programs in Sri Lanka Institute of Devel- ton University of Canada) visited Sri Lanka to study the opment Administration, the government arm for emerging good practices of MfDR and M&E initiatives. training, and in other universities at the graduate The OECD Sourcebook (2008) on emerging good practice and postgraduate level. in MfDR highlights the Sri Lankan case. The Asia Pacif- • Strengthen the documentation center on evalua- ic Community of Practice on MfDR has identified the fol- tions, and promote the exchange of experiences and lowing factors for successful institutionalization of MfDR access to best practices and sharing of databases. in Sri Lanka. Strengthen the feedback arrangements • Strong support from the top and strong and sus- • Improve disseminations of evaluation reports tained leadership for results oriented reforms and through in-house workshops and seminars, cus- buy-in. Government policy on MfDR endorsed by tomized reports, evaluation summary reports, press Cabinet of Ministers and champions at national briefings, and post EIS. level and change agents at different levels of govern- • Establish strong feedback arrangements among ment to drive the initiative have been instrumental evaluation, planning, decisionmaking, policy for- in taking this change process forward. mulation, project appraisal, program management, • The MfDR approach adopted in Sri Lanka is a and budgeting and resource allocation functions. whole-of-government approach covering nation- • Ensure action is taken on the recommendations in al, sectoral, institutional, and project level. It is not evaluation reports. Wider dissemination of evalua- just viewed as a technical tool but as a comprehen- tion information preferably should include the Par- sive way of thinking to achieve outcomes and im- liamentary Public Accounts Committee, Parliament pacts. library and media. User-friendly evaluation synthe- • Cascading approach, which combines a mix of top- sis or summary reports should be widely circulated. down and bottom-up approaches and sequenced • Stimulate the evaluation issues in the country’s de- approach beginning with pilots and then expanded velopment dialogue and sector program assistance. and mainstreamed in government, is a more prag- M&E units must have active involvement in the matic strategy. planning of new programs. • The MfDR reforms as complementing the exist- • Incorporate evaluation lessons into the new project ing initiatives, strategies, and general reform agen- concept documents or project submission formats da and is part of the country systems. But “perfect” so that past mistakes are not repeated. Revise proj- should not become the enemy of the “good” in the ect submission formats to incorporate evaluations MfDR area, as most developed countries are still lessons of past projects. struggling to achieve a satisfactory solution. 45 M&E SYSTEMS AND M&E TOOLS Issues and challenges lines are not given. Setting achievable targets is not possible In many countries, including Sri Lanka, the wider dissem- in the absence of baseline. The greatest problem associated ination of M&E findings remains a problem. M&E insti- with performance management is unrealistic expectations. tutions and the planning institutions seem to function in In addition, information overload can be a problem. isolation and do not have an effective formalized feedback Policy makers were unable to absorb the vast amount of in- arrangement to integrate lessons into the planning and de- formation provided to them. Outputs are manageable for sign of new projects. These institutional gaps defeat the institutions, but outcomes are the results of collaborative very purpose of M&E. Therefore, it is necessary to estab- efforts of other institutions. Agencies can more easily be lish strong links between M&E and policy formulation, re- held accountable to output targets but not necessarily for forms, planning, budgeting, and resource allocation func- outcome targets. tions. The Sri Lankan government has identified the need In some countries, MfDR systems have not fully to establish strong feedback mechanism. Feedback is the achieved the goals. Performance levels slipped partly due to weakest link in the project cycle. This issue is now being se- a lack of resources and unrealistic expectations. Also, weak riously addressed. links between “agency performance” and “individual per- In many cases, donors and partner countries contin- formance” are a concern. To ensure successful operation of ue to be disbursement oriented. Moreover, donors tend to RBM systems, incentives are needed for achievements as be using their own donor systems rather than country sys- well as some form of penalty for slippage. tems to maintain visibility and attribution. The lack of de- Because MfDR systems often create information over- mand for MfDR, shortage of professionals, multiple results load, it is also necessary to understand who needs what in- frameworks, too many indicators, lack of aid predictabili- formation, for what purpose, and when. It should not be ty, and weak statistical capacity have been identified as con- overly supply-driven information. Moreover, in some cas- straints in many developing countries. Also, MfDR, as a es, MfDR creates fear of being held accountable for perfor- change management process, takes time. mance when cooperation and assistance from outside the Recognizing the demand side of the equation for cre- organization are necessary for success. ating local demand for evaluation with a utilization focus, It is necessary to look at the balance between learning and the supply side of the equation, which includes skills, pro- accountability. Although independent evaluation is impor- cedures, methodology, data systems, and manuals, also has tant for ensuring objectivity, too much emphasis on account- to be addressed. The need to focus on national evaluation ability-focused donor-driven independent evaluation can capacity development is equally important. Making M&E be a potential constraint for lessons learning and feedback. information available does not necessarily mean effective Hence, the importance of lessons learning and ownership has utilization. It is hard to justify the existence of an M&E sys- been recognized in the government evaluation system. tem that is not effectively utilized. The Sri Lankan govern- Many countries have not adequately responded to the ment was able to address some of these issues with the tech- observation that ex post evaluations are more often viewed nical assistance support of UNDP and the ADB. as postmortem exercises that do not contribute much to Attribution is a challenge for measuring performance strategic decision making. It is necessary to recognize the in ministries and departments. In selecting the KPIs, care importance of lessons learning and performance account- should be given to the attribution issues. Unlike the private ability. Increasingly, concurrent evaluations are encouraged sector, the public sector does not have a bottom line. Mul- for midcourse corrections. Also, the government encourag- tiple stakeholders and divergent interests characterize many es donors to undertake more joint evaluations to ensure na- areas of government activities. Target setting is a challeng- tional ownership, lessons learning, and capacity building. ing process. It is difficult to get consensus on goals and val- There has been a generally tend to monitor rather than ues and difficult to agree on KPIs. to evaluate. It is necessary to give equal importance to eval- Specifying and agreeing on expected results is not easy. uations by finding the appropriate balance between the two The results chain is not always logical as expected. Indica- activities. Despite the enormous methodological and tech- tors are missing for some results areas. Targets and base- nical challenges, institutionalization of evaluation is the 46 S ession 2 : I nstitutional A rrangements for M onitoring and E valuation S ystems way to ensure results orientation in development work. fers to the work involved in the evaluation of public policies Moreover, development policy and aid tend to shift from and programs in response to political reaction to the impact projects and programs to a sectorwide approach and, as of economic and financial crisis on the whole population. Fi- such, an M&E approach needs to cover policies, sectors, nally, the fifth section concludes with some ideas on the insti- and thematic areas on a countrywide basis. Policy evalua- tutionalization of evaluation policies and programs. tions, sector evaluations, and thematic evaluations are be- coming equally important, and the Sri Lankan government Institutionalized evaluation of government policies has given much emphasis to such evaluations. The govern- in Spain ment of Sri Lanka is mindful of these aspects and views The Government Agency for Evaluation of Public Policies M&E from a wider country-based context. and Service Quality is the result of the commitment included in the electoral program the Spanish Socialist Workers’ Par- Conclusion ty presented in the 2004 general elections. In the program’s Institutionalization of MfDR has been instrumental in cre- chapter on public administration guaranteeing essential pub- ating a shift in focus in M&E systems and practices by ex- lic services, “the creation of a Government Evaluation Agen- panding the focus of “results” in development. The ambi- cy” was identified as a priority project. This commitment was tious institutionalization of MfDR in Sri Lanka has laid renewed in the program in 2008, with the idea of strengthen- the foundation for the emergence of a thriving results-fo- ing the Agency for Evaluation and Quality. cused M&E community in the Sri Lankan government. Al- The creation of the new institution was part of a reform though it has a long way to go before MfDR will be as- program that included a set of far-reaching measures in the similated into the public sector culture, the initiatives have service of innovation and modernization of public activity, achieved significant progress toward a government-wide among which are the State Agencies Act, the Basic Statute shift to results-based M&E practices. for Government Employees and the law on citizens’ access to e-government. VelayuthanSivagnanasothy The institutionalization of evaluation through the cre- Director General, Department of Foreign Aid and Budget ation of the agency was preceded by a profound and exten- Monitoring, Ministry of Plan Implementation, Sri Lanka sive analysis conducted by a Commission of Experts, which was given the task of making the methodological and insti- tutional proposals most recommended for starting up the Institutional Arrangements for International agency after a review of international experiences and a di- Monitoring and Evaluation Systems agnosis of the status of evaluation in Spain.16 Political action and the work of governments as guarantors The diagnosis showed the lack of a culture of evalua- of public interest and citizens’ welfare have gained renewed tion in the administrative practice of public institutions strength in addressing the new financial, economic, and so- and, therefore, the availability of the necessary tools to car- cial reality. Evaluation, in particular, as a mechanism for ry out evaluations of public policies. As a result, a signifi- feedback on public action and accountability to citizens, cant shortfall was found of what might be called the de- will be high on the political agenda for the transformation sign of, or “evaluator approach” to, public policy. It was of public administration. evident that evaluations of public policies were not being This presentation is divided into five sections. The first carried out in an institutionalized or systematic manner. deals with the commitment of the government of Spain to The lesson drawn was that accountability and transparen- institutionalize evaluation and, consequently, to create the cy of public resources were not visible to the public, or, in Government Agency for Evaluation of Public Policy and Ser- other words, that citizens did not have access to this instru- vice Quality. The second section discusses the results of the ment to measure accountability in the management of pub- agency’s action and the basic principles that support it. The lic resources and assets.17 third section reflects on the Spanish reality of a decentralized The consensus established between the political com- state and its relationship to evaluation. The fourth section re- mitment expressed in the government program and 47 M&E SYSTEMS AND M&E TOOLS academic guidelines from the Committee of Experts was macy of the work of the institution. Thus, the agency must reinforced by the legislature with the addition, on July 18, evaluate the following: of a provision to Law 28/2006 on state agencies to improve management of public services.� The additional provision • Programs and public policies annually commis- authorizes the government to establish the Agency for the sioned by the Council of Ministers Evaluation of Public Policies and Service Quality. • The degree of implementation and success of the Consequently, the new agency was the result of a triple main measures of the national reform program endorsement—political, academic, and legislative—and • The Spanish Strategy of Sustainable Development explains the government’s wish to place the administration in collaboration with the Sustainability Monitoring at the service of citizens by making it more transparent, in Spain(Observatorio de la Sostenibilidad en España) with full and fair accountability on the part of incumbents and civil servants, to enable an effective participation in The agency also must do the following: public affairs by informed citizens. Government resolve should be in tune with the principles of efficiency, partici- • Submit an annual report to the Congress of Depu- pation, and accountability that underlie good government. ties on the activity of state agencies and their com- Evaluations of public policies and programs of the Gen- mitments to improve the quality of services provid- eral State Administration and the drive to improve the ed to citizens quality of management of public organizations are the two • Conduct an annual report on the Quality of Pub- pillars of the agency’s activity. Improving the quality of gov- lic Services Department (Departamento de Calidad ernment performance will in turn improve the service it de los Servicios), including citizen perception of the provides to its citizens. Thus the action of the agency serves quality of those services a dual purpose: political and socioeconomic. The political goal refers to accountability, transparency, and democratic Since its inception, the agency has produced a total of quality, whereas the socioeconomic objective refers to the 15 evaluations, which, in most cases, correspond to the pursuit of efficiency, the rationalization of resources, de- government’s commitment to review the policies of the cision making based on evidence, and evaluation of policy National Reform Program (NRP). Also being evaluated in outcomes in society. 2009 are eight policies, including five for the NRP. The The agency is assigned at present to the first vice pres- evaluations have focused on various issues. However, policy ident and minister of the presidency. Consequently, it oc- evaluations include social issues (increases and reductions cupies a strategic position in the organization of the gov- in social security contributions, education grants, and the ernment, inasmuch as it is within the ministry that handles participation of the general state administration in the sys- executive and legislative affairs jointly. This location of the tem for autonomy and care of the unit), and the promotion agency in the executive branch confirms how important of economic activity (rationalization of drug expenditures, evaluation has become, as well as the government’s aware- administrative procedures for startups or lines of finance ness of its dual usefulness. The government recognizes the for entrepreneurial activity). Certain environmental poli- value that evaluation provides both at the policy formula- cies have been evaluated (national registry of allowances of tion stage, that is, on a purely legislative side, and at the greenhouse gases, the National Parks Network). Also, eval- policy design and implementation stage, which is an exec- uation of the innovation and technology policy has risen in utive job. importance. For example, programs that promote research, development, and innovation have been evaluated. These The evaluation of public policy: Scope and include the Ingenio 2010 plan to improve the technology practice transfer system in line with the government’s concerns. The evaluation and promotion of quality developed by the The guiding principles that underpin the agency’s per- agency are largely determined by specific parliamentary and formance are (1) independence of criteria (opinion and government mandates. This feature strengthens the legiti- judgment in carrying out work based on values of public ac- 48 S ession 2 : I nstitutional A rrangements for M onitoring and E valuation S ystems countability and professional competence), (2) transparen- The value of evaluation in public action cy (accountable to citizens), (3) autonomy in management The market-based economic system is incapable of re- and responsibility in the achievement of results, (4) quali- sponding to social needs derived from the principles of re- ty and continuous improvement to deliver services in an in- distribution, equity, and social justice. This inability has be- novative manner, and (5) professional ethics and public ac- come evident in different historical moments and has been countability, understood as the commitment of the agency exacerbated in crises that clearly show that the market alone staff, especially their directors, to abide by the values con- cannot respond to emergencies and that government inter- tained in the agency’s code of professional ethics and in the vention is required to bring order and regulation. The need rules of conduct applicable to all government employees. for public intervention is, therefore, one more sign of the Strict compliance with the principles confers maximum imperfections of the market. Hence the need once again for stringency, systematization, and transparency on the activi- the role of politics and government action to guarantee the ties of the evaluation of public policies. Values are formulat- general interest and come up with a political strategy to fit ed under these principles. the new financial, economic, and social reality. The dissemination of evaluation results, the publica- In Spain’s case, that role has materialized in recent times tion of the activity reports, approval of the ethics code, the in the Spanish government’s commitment to renewing the programming of training agency staff, or development of economic growth model, with the strategic goal of achiev- methodology are indicators of the habitual exercising of ing sustainability in three key dimensions: these principles. • Economically sustainable in the medium and long The evaluation of public policies in a decentralized term. The idea is to develop a growth model based state: Cooperation with other levels of on the known economy and innovation that will government increase the levels of competitiveness of the Spanish The evaluation of public policies, as mentioned, is a poli- economy and successfully confront the phenome- cy and management tool that allows the government to re- non of globalization. view the designs of policies and programs implemented by • Socially sustainable. This objective promotes stabil- the administration at any time, if not the actual social need ity in employment, equal opportunities, and social for these policies. This check is even more necessary in a gov- union. ernment as decentralized as Spain’s, where policies sometimes • Environmentally sustainable. This objective ad- are designed by one political entity and implemented by an- dresses the European targets on energy and climate other, or, they are designed with the help of different enti- change.18 ties and implemented by one of them or by a third party—all with complex financing arrangements. The agency’s activities At the same time, a management model must be devel- should be configured as a staple of a cooperative government oped in keeping with this political strategy, focused on revi- evaluation system, in which the agency’s added value becomes talizing the government’s spaces of action so that they play a complement, and in which it has its own specific space. a proactive role in this design and effectively encourage ac- Also, it should cooperate bilaterally with other levels tivity and competitiveness of production and, furthermore, of government—autonomous and local. In that sense, the play the role of public administrators and service managers agency statute not only offers the possibility of technical to the public’s satisfaction. cooperation and joint effort with the autonomous com- This new model should gravitate toward evaluation as munities, but also opens up to the participation of these the hub of the new public administration. Evaluation, as a entities in the institution’s top management through their key concept of the new framework of decentralization and representation on the Governing Council. Therefore, eval- administrative modernization, seeks to cover all areas and uations of the results of public interventions, the quality of reach people, organizational processes, and outcomes of the services, and income from the organizations are key aspects intervention. We can therefore say that the evaluation has in assessing the patterns of good governance. four primary objectives: 49 M&E SYSTEMS AND M&E TOOLS • Accountability among managers of policies, pro- ularly Spanish ones, are a direct result of that intention and grams and public projects a catalyst for further efforts in that direction. • The consideration of the actors involved in politics Second, the evaluation of public policies and programs • The effective and efficient management of public is based on the idea of building a government organization resources that legitimizes its action depending on the performance • Analysis of the impact of public interventions it generates in the provision of tangible results for citizens. Third, the independence of criterion used in the evalu- So, it is possible to identify the instrumental nature of ation process is correlated to strict methodology, quite sep- the evaluation in two ways: policy and management. On arate from the executive or legislative workplace where the the one hand, it enables social and political control, aimed evaluator institution is located. at meeting the proposed objectives (effectiveness), and, sec- Fourth, evaluation is not an end in itself, but acquires ond, it facilitates management improvement through con- value in decision making, to replenish the beginning of tinuous learning (mainly efficiency). the management cycle, the development of budgetary pro- Therefore, the observance of the principles of effective- grams, and the design of public policies. It is therefore im- ness and efficiency becomes more important in the current portant to design procedures to incorporate the improve- context in which the evaluation has an essential role in the ments early in the process and as effectively and efficiently design and implementation of public policies. The contri- as possible. Otherwise, evaluation itself would only increase bution of evaluation in the current context grows and ma- the expenditure incurred by the policies without ensuring terializes on two fronts: its effectiveness and efficiency. Finally, this cannot be achieved without government • Improved efficiency in the delivery of priority ser- commitment to make effective and efficient policies, and vices and in meeting strategic objectives, which we institutional technical support offered by the State Agency know contain measures aimed at families, business- for Evaluation of Public Policies and Service Quality. es, or promoting employment. • Appreciation of compliance with the principle of María Luisa CarcedoRoces efficiency in spending, which will lead to more ra- President of the Government Agency for Evaluation tional distribution. of Public Policies and Service Quality, Spain This requirement must go hand in hand with the principle of transparency and public accountability. We Institutionalizing Monitoring and Evaluation by do not outline these two criteria independently, because the Public Service Commission of South Africa: in the case of evaluation, effectiveness and efficiency are A Continuing Journey not understood separately in that one needs the other to be complete. So that, “only after reaching efficacy can ef- Background fectiveness can be observed, since otherwise you could es- The establishment of monitoring and evaluation (M&E) tablish an economy of means, but without fulfilling the in South Africa was a process that paralleled the herald- goals and objectives proposed in the policy design” (Ba- ing of democracy in 1994. Democracy brought about the ñon 2003). modernization and transformation of the state and gov- ernment. Before 1994, the government lacked significant Sustainability of evaluation systems M&E, as it is currently understood. Issues of transparen- To conclude with a reflection on the sustainability of the cy and accountability and the public engagement on the evaluation systems, the following are some ideas that may performance of government were largely absent. M&E in result from this discussion: South Africa was not only about performance improve- First, the practice of evaluation contributes to transpar- ment, but also a critical strategy and tool for strengthen- ency in public administration. Evaluation agencies, partic- ing democracy. 50 S ession 2 : I nstitutional A rrangements for M onitoring and E valuation S ystems The democratic government’s accountability framework also to engage politically and with various constituencies to has been supported through different pressure points and maximize its influence to be the “custodian of good gov- streams of performance information placed in the public ernance,” its motto. By taking such a path, it has benefit- domain and the administrative and political decision-mak- ed from engaging and supporting those institutions whose ing environment. M&E information has ensured a focus primary purpose is to ensure a form of good governance, on the actual performance against the plans—the promis- whether it is financial (the auditor general), transparen- es versus the reality. In light of the prominence and value cy (Transparency International), and good governance attributed to M&E, M&E institutions exert a particularly initiatives of the continent (African Peer Review Mecha- important influence on South African society. It is against nism), or thrusts from the Commonwealth Association for this background that the institutionalization question is ad- Public Administration (CAPAM), to name but a few. Its dressed. This paper moves from the premise that for a sus- work is supportive of the Millennium Development Goals tained M&E culture, central M&E institutions must en- (MDGs). Finally, it joins the networks around the world gage with all sectors that have a common interest in good seeking to measure and make public the performance of governance, which should include those sectors that may government, and ensure transparency and accountability. be critical of government. Such an approach will provide This paper briefly explains the strategy used by the PSC a broad-based approach to address the myriad of perfor- to institutionalize M&E in the country. As mentioned, it mance issues in a transformation context. has explicitly set about using approaches that are democrat- ically enriching and forming partnerships to support the The Public Service Commission of South Africa thrust for good governance as a key strategy. In using this The original Public Service Commission (PSC), which was embracing approach, it has managed to tap into the po- established in 1912, was fashioned after the British Civil tential of stakeholders and role-players in the country, who Service Commission. By the 1980s it had burgeoned into possess insights and expertise that can be harnessed to help a vast and powerful statutory body. With the new consti- government improve. Importantly, such an approach has tution, the PSC was reestablished as one of the institutions given the institution more credibility and respect, elements supporting constitutional democracy. It had multiple man- that are important when it comes to producing and pre- dates, and between 1994 and 1996, it changed from an senting evaluation findings. executive policy-making and implementing body to fo- cus exclusively on investigating, monitoring, and evaluat- Advocating change to entrench good governance ing public administration. With this new mandate, it has The PSC has utilized its constitutional mandate effective- become the central M&E player in the country and now ly to craft an M&E program that addresses all areas that in- provides support and leadership for the development of fluence good governance and service delivery. It recognized M&E at the regional and continental level. In this way, it that the democratic transition period (1994–2000) was a is different from its counterparts in most parts of the world difficult one, as new political and administrative leadership where the focus remains on administrative issues; the PSC emerged, and new policies had to be developed and imple- of South Africa has embraced and developed its mandate to mented. The PSC’s approach during this period was large- become a key player in M&E. ly one of advocacy and support, producing guidelines and Given the limited resources available to the PSC, against manuals to support the new government. It also raised the an onerous responsibility, the PSC has developed a differ- profile of M&E by engaging with the M&E community at entiated approach to institutionalizing M&E. This involves the country and regional level, the most pronounced be- forging social compacts, partnering with the M&E com- ing in 2004 when it cohosted the highly successful Third munity at country and continental level, engaging the ac- African Evaluation Association (AFREA) conference in ademic and research sector on its products, and general- Cape Town, South Africa. This was an important event ly playing a strong advocacy role for different focal areas to given that it helped to dispel the notion that performance bring about change in the country. It thus has had not only and democratic culture did not exist on the continent, and to develop technical and managerial expertise in M&E, but many keyspeakers spoke about how M&E could change 51 M&E SYSTEMS AND M&E TOOLS the stereotypes of the continent and region, which tended The links that were forged with the academic and re- to be negative. The event also brought together a commu- search sector come about at events when the PSC presents its nity of practice, and the papers delivered across all develop- annual State of the Public Service (SOPS) reports to a cross- ment sectors proved that, indeed, capacity was present on section of society roundtables for discussion. These events the continent, as in the case of South Africa to use M&E have proved useful in that the SOPS offers a critique of to improve government performance and support dem- methodology and perspective, and demonstrates once again ocratic engagement. What was important about the AF- why M&E must include the work of as diverse a grouping REA Cape Town event was the support from many societal of persons and institutions as possible. These efforts will il- and academic sectors for the M&E of governments in Afri- lustrate how a meta-evaluation has been produced and en- ca. Events such as these may be viewed as “institutionaliz- gaged with, and where the results go to bring about the de- ing moments,” in that they help propel the discipline for- sired change.� Other events and strategies that are important ward. Other such events, in which the PSC engaged during for forging social partnerships include the following: this advocacy for M&E phase, were the subsequent AFREA conferences, each of which has helped to consolidate the • Public hearings on disability where representative community of practice that now exists continentally and organizations attend and provide lived-experiences that has increased intra-African dialogue on M&E. of their challenges In South Africa, the PSC has helped form the South • Partnering with organized labor in conferences African Monitoring and Evaluation Association (SAMEA) dealing with key labor relations issues conference, and has since co-hosted two conferences (2007 • The development and holding of the leadership of and 2009). It has signed a memorandum of understand- the Association of African Public Service Commis- ing with SAMEA to ensure a long-term relationship, which sions (AAPSCOMS), an umbrella body for PSC it believes is important to professionalize the discipline in and Civil Service Commissions in Africa the country. • Collaborative research with the formal research The advocacy element of M&E has been necessary and sector, for example, the evaluation of government important, given the historic and contextual factors in the projects on poverty country, which means that the good governance, with its • Participation in training programs at the country, supports and transparency and accountability imperatives, continental, and international level, such as guest cannot be regarded as a given. The very act of M&E some- lectures on M&E to graduate programs, support- times may be perceived as threatening to governments, as ing the Development Evaluation Seminars and M&E talks truth to power. It is for this reason that full re- teaching at the International Programme for De- sponsibility for M&E cannot be accorded to one or two velopment Evaluation Training (IPDET), among centralized institutions; it must be a shared responsibili- others ty across society and its institutions, so that different per- • Presenting papers at relevant seminars, conferences formance perspectives can be provided, and different pres- and special sessions sure points used to improve government performance. Only when M&E is broad-based, will citizens be empow- From the above, it is apparent that the past decade has ered to hold government to account, and democracy deep- been one of charting new ground, and seeking support for ened. The contribution of M&E to citizen empowerment the mandate of M&E from a cross-section of sectors. This has been an important consideration in the design of the has helped to get these sectors to understand the role of PSC’s M&E program, and this is particularly important in the PSC and also has helped the PSC to develop its own a country in which political and social alienation reached thinking and methodologies on how to conduct its work a point at which most citizens disregarded government. To in a more effective manner. Currently, the PSC is high- reverse this situation, a citizen-centered M&E component ly respected across these sectors, and it is seen to be inde- was needed, whereby citizens would be capacitated to pro- pendent, doing its work in a transparent and credible man- duce and apply M&E in a useful manner. ner, and without fear, favor, or prejudice. It is not viewed 52 S ession 2 : I nstitutional A rrangements for M onitoring and E valuation S ystems as an extension of the executive, but as an M&E leader in recommendations made and tracked. The process for im- the country. plementation includes a briefing session with the depart- ments’ management thorough ongoing engagement over M&E from process to products the three-month period of assessment. The draft report is The above also points to a large extent to the processes fol- presented to the management on completion. Thereafter, lowed in securing the support of other key sectors to bolster the department is given 10 days to make written submis- M&E capacity in the country. This section looks at some sions to the PSC, which may be considered in reviewing of the processes that have been followed to ensure that the the draft score. Once the report is finalized, the department products of the PSC result in results. is given six months to implement recommendations, and the implementation thereof is reported on to relevant po- National Anti-Corruption Hotline litical and administrative structures. The system is now well Given the high priority of fighting corruption and graft, the known and has worked by reinforcing other M&E work, as PSC has set up a National Anti-Corruption Hotline (NACH), it also draws on existing evaluations, such as that of the au- a toll-free facility operating 24 hours a day, seven days a week, ditor general. Through a mutually reinforcing process, the in 11 official languages. The hotline receives complaints from question of performance management is entrenched. the public on corruption and service delivery. It has proved At the aggregate level, compliance results have improved popular with more than 7,000 complaints lodged to date slowly. The pattern has been uneven, however, and the up- and directed to departments for investigation. Its accessibility take on the initial set of recommendations has in many cas- means that checks and balances within society can ensure that es been poor, resulting in repeated poor assessments for sev- corruption does not go unreported. The hotline has had chal- eral departments. Given that the system has not changed lenges, the main being the lack of investigative capacity with- significantly over time, it is of concern that departments in departments to follow up on cases referred to them. This continue to score poorly in some of the areas critical to challenge is being addressed through better collaboration with good governance, such as poor public participation and law enforcement agencies, but it remains a key factor as timely inadequate decision-making processes. It may be that the addressing of corruption cases is a key deterrent factor. Thus turnover in leadership at the management level contributes far, several officials have been dismissed because of complaints to the limited improvements that have been seen. lodged, and more than South African rand (R)100 million has been recovered as a result of this facility. Citizen M&E The PSC has, through announced and unannounced in- Investigations spections, tested the extent to which public services are pro- The PSC conducts its own investigations, and upon receipt vided according to the Batho Pele (people first) standards. of a request or complaint. Several complaints have received These are eight standards that stipulate how government intensive media coverage, which helps to restore public departments should operate, and what citizens can expect confidence in the institutions of democracy. Tangible out- of them. Several surveys have been conducted, and specific comes have arisen from the investigations. investigations have resulted in reports presented to the po- litical leadership for action. Last year, the focus was on po- Implementation of the transversal public service lice stations, and this year it is on clinics. These reports pro- M&E system vide a quick and objective assessment of the facilities. By The PSC has designed a good governance M&E system tracking the implementation of recommendations and re- to assess departmental performance across the nine con- turn visits to the facilities, the PSC can see whether this stitutional values and principles for public administration. form of M&E is delivering the expected results. The system, implemented since 2001, has thus far assessed more than 100 departments. For each of the principles and Evaluation of department heads values, indicators are chosen and assessment takes place ac- The PSC manages the secretariat for the evaluation of de- cording to verifiable standards. Scores are awarded, and partment heads. In this, a department head is evaluated by 53 M&E SYSTEMS AND M&E TOOLS a panel composed of key people from the sector relevant to formance information to citizens, and increases the levels of the department, a peer, and others who know and under- transparency and accountability of the democratic state, it stand the department. An advantage of the process is that it also has achieved the following: forces reflection and provides valuable feedback on the per- formance of the department heads and the departments by • Improved the receptiveness of government to being a well-constituted committee to the minister and the de- measured, which means that it has to review how it partment heads. While the process is useful from a transpar- works and as such has instilled a performance ethos ency and accountability perspective, further development is with departments. The guidelines by the National required so that individual performance is aligned to orga- Treasury, the auditor general, and the Department nizational performance. The impact of this evaluation pro- of Public Service and Administration, in particular cess on accelerating and improving the management of per- with regards to planning and reporting, has for the formance within departments is not known. Questions that first time quantified what needs to be done, and with are being addressed currently are the arrangements for fu- what resources. Outputs also have to be reported on. ture evaluations of department heads, which is tied up with • The results-based orientation is greater now than broader issues of performance contracting. was the case previously, where reports such as de- partment annual reports were largely related to ac- Releasing results of reports that have national tivities. The current format of the annual report is importance an M&E document, and it allows the public to en- The PSC annually releases results from some of its systems, gage on actual performance versus plans, through which have national importance. These have becomes key these oversight reports. points for deliberation in the country. The most important • The process used by oversight or M&E agencies of these annual performance releases are as follows: and bodies, which include parliament and its com- mittees, have been informed by M&E reports and • Financial disclosures of the Senior Management information. This allows for a more directed and Service (SMS). The financial disclosures, where- objective engagement when accountability struc- by all 6,000 SMS members are expected to declare tures call departments to account. their financial status and interests, has been a mea- • A key user of M&E in the country has been civil so- sure to prevent conflict of interests, given that de- ciety and the media. This is apparent in their par- cision makers can abuse office by making decisions ticipation in the events of the PSC, which are di- for personal gain. It ties up with the issue of fight- verse and indicate the broad-based interest held in ing corruption and making the public service cor- government performance. ruption resistant • Financial misconduct reports. All departments are Location of the M&E function obliged to report on their cases of financial miscon- The PSC, as a constitutional body, is outside the execu- duct annually. This provides a status of the situa- tive branch. This has been an advantage in that it has had tion and helps to quantify graft. the latitude to engage more broadly and is perceived as in- dependent and impartial. A challenge is that it makes rec- Answering the brief? M&E links to the reform and ommendations and waits for it to be implemented by the modernization process executive office. The rate of implementation has been im- Based on the above, this section attempts to pull together proving, but is not satisfactory. It will soon receive pow- some of the arguments made to answer the brief of this pa- ers to issue directives. One way in which the PSC has ad- per. M&E makes an important contribution to the reform dressed the disjuncture between producing findings and and modernization process of government in South Afri- realizing results is by producing reports in which all its rec- ca. Apart from its important contribution to democracy, ommendations are made public, and the action thereof re- whereby it provides previously unknown government per- ported on. This indentifies where the problems are in mak- 54 S ession 2 : I nstitutional A rrangements for M onitoring and E valuation S ystems ing the translation of findings into results. The current basis. The PSC augments its research with that of the other arrangements are thus appropriate, because they retain an government institutions. Thus, its SOPS reports are heavi- arm’s-length distance, but also keep avenues open for en- ly referenced with work from these institutions. It draws on gaging the executive office, which is willing to receive PSC research undertaken by research units in the country and findings. The PSC has managed to get audience with the abroad, and attempts to avoid duplication of effort. It par- highest offices of the land, and as such, its location has not ticipates in initiatives led by the presidency from the min- been an inhibiting factor. istry for M&E, and works with all those involved in M&E in the country. Through its memorandum of understand- Institutional design and its impact on M&E ing with SAMEA, and as a follow-up to the SAMEA 2009 decision-making process conference, it will engage more directly with the nongov- The PSC is led by a board of commissioners who are ap- ernmental organization and research sector, perhaps com- pointed by the president of the country (nine of whom are ing up with an evaluation program that sees the existing based in the nine provinces) and who may serve a maxi- harmonization of M&E and maximization of evaluation mum of two renewable terms of five years each. The board capacity. This remains a work in progress. There is no overt is supported by an office of public servants who are ap- competition to the work of the PSC from the M&E sectors pointed according to the terms of the Public Service Act. in the country, and this may be due to an established histo- This provides adequate capacity within the organization to ry of cooperation. engage with the various legislatures, as well as adequate ca- pacity and skills to produce the type of networking with Sustainability of systems stakeholders mentioned. In terms of decision making, the The PSC is now moving to support more directly the PSC acts as one body, and although there is space for pro- M&E capacity-building initiatives in the country. This vincially specific research, the decisions of the PSC are that will take the form of producing and presenting its M&E of a unified institution. guides, and in general providing a much-needed support The institutional design has not impeded the speed of for emerging M&E practitioners who have been appoint- work, and provincially based commissioners are able to un- ed in many departments. It seeks to do this in a collabor- block problems at that level. The question of how findings ative program with SAMEA. It believes that holding more are presented, after they have been engaged with and pro- sector- or province-specific events on M&E will be a way duced by the PSC, and acted on is being addressed on an to build M&E capacity in the provinces. M&E currently ongoing basis by customizing approaches that seek to get is concentrated at the national level, which is an anoma- the most influence in the shortest period of time. ly given that most of the departments and population are not. Through the more effective synergy between SAMEA The impact of institutional design and and the PSC offices, whereby communities of practice decision making flourish at the provincial and district level, this necessary The design of the board supported by an office (or secretar- change can occur. iat) has not adversely affected decision making, as over the last decade management steps have been institutionalized Incentives for evaluation? to address the broad scope of work undertaken, in a man- Creating incentives for evaluation is a difficult question, and ner that has not compromised quality. The PSC has com- more needs to be done. Thinking is under way on linking missioned an assessment of its work, and the issues raised departmental results with some form of acknowledgment. therein are being incorporated into the organization as it Currently, the reports of the PSC rate departments against moves forward.20 their peers, but this has not taken the form of awards for good performance, except citations in reports. The ques- Link of PSC M&E system to sector systems tion of annual league tables for performance across a range The link between the PSC and the sector systems takes of performance criteria is being considered to bring in the place more by agreement and convention, than on a formal incentive element more directly. 55 M&E SYSTEMS AND M&E TOOLS Translating process into results Session 2 Notes The efficacy of an M&E system depends on buy-in and un- 14 The ePMS can be accessed through http://www.fabm.gov.lk/. 15 derstanding, and thus the advocacy element and engage- Ministerial Order, APU/1463/2004, of May 19, (BOE May ment are needed. When the PSC brought decision makers 25). 16 For the full report, see http://www.aeval.es/comun/pdf/Informe_ onboard during the evaluation process, it received great- comision_expertos_esp.pdf. er cooperation. What still needs to be achieved more ful- 17 For the full provision, see http://www.aeval.es/comun/pdf/ley_ ly is ownership of evaluation results, which will come about de_agencias.pdf. when there is a perceived benefit to M&E, and those eval- 18 For the complete text, see www.plane.gob.es. 19 uated see the direct benefits for their own operations. The For the full report, see www.psc.gov.za. 20 challenge will remain between doing what is considered to See the report on the PSC‘s Web site, www.psc.gov.za. be necessary from the top, and what is deemed important from the bottom. A flexible M&E systems approach, which caters to all of these needs, is a way to institutionalize M&E. Indran A. Naidoo Deputy-Director General: M&E: Office of the Public Service Commission, South Africa 56 �Session 3: Institutional Arrangements for Monitoring and Evaluation Systems in Latin America Challenges and Opportunities for the National Planning System Institutionalization of Regional Institutional This system is made up of the Planning Ministry’s institu- Agreements for Monitoring and Evaluation tional and sectoral planning units (as appropriate) and the Systems in Latin America: The Experience remaining public institutions and mechanisms of coordina- of Costa Rica tion and consultancy, such as consultancy boards, interinsti- In democratic systems, modernization of the state is not tutional committees, consultative commissions, and others. only a central concern, but also a permanent activity that The ministry works on each administration’s National De- seeks to strengthen public management and guarantee velopment Plan (NDP) through a participatory process in adequate functioning of planning, budgetary, and eval- which sectors, leaders, and new authorities present their stra- uation systems, and to enable the latter to measure the tegic development proposals for each thematic area or sector. impact of public policies designed to improve the quali- In addition, the ministry maintains a current and prospec- ty of life of citizens. In Costa Rica, the Planning Ministry tive diagnosis that permits the evaluation of the internal and (Ministerio de Planificación Nacional y Política Económica) external events that can affect national development, such as is charged with the task of strategic planning, monitor- the various policy proposals that require promotion or ex- ing, and evaluation of public-management strategic ac- ecution at the governmental level. Thus, the NDP orients tions, and to provide consultancy at the highest level for the objectives of the government’s actions to promoting the establishment of the corrective measures necessary to per- country’s development. It establishes policies and objectives mit the various sectors to achieve their proposed goals in at the regional and sectoral level for public entities, minis- each administration. tries, and other government organs. The 2006–10 plan, in This function takes shape in the legal framework effect at present, contains the following goals and activities: through Law 5525 (1974) created by the ministry, which in turn creates the National Planning System, the Decree • Eight national goals: (1) combat corruption in all for the National Evaluation System dated 1994, and Law areas of action of the public sector, (2) reduce pov- 8131 (2001) of the finance administration and public bud- erty and inequality, (3) increase the growth of the gets and its regulations, a norm that—together with the economy and employment, (4) improve the quality creation of institutional planning units, services comptrol- and increase the coverage of the educative system, lers, and internal audits—provide an institutional frame- (5) halt the growth rates of criminality, drug traf- work that strengthens ministry of planning, monitoring, ficking, and drug addiction, and to revert the sensa- and evaluation actions. tion of insecurity on the part of the entire citizenry, (6) strengthen public institutions and to order state priorities, (7) recover and broaden the country’s 57 M&E SYSTEMS AND M&E TOOLS transport infrastructure, and (8) ennoble foreign National Evaluation System policy and restore Costa Rica’s role in the world From its beginnings, the National Evaluation System (NES) • 84 activities in the following areas: social and the is limited to the executive. It was designed as a self-evalua- struggle against poverty, education, health, citizen tion mechanism of the government’s strategic actions, with safety and crime prevention, cultural, financial, the objective of offering timely information to allow the ex- productive, foreign trade, tourism, infrastructure ecutive branch (ministers, governors, executive presidents, and transportation, science and technology, em- managers, and even the president of the republic) to make ployment and social security, financial supervision decisions related to advancement of the NDP. and monetary policy, environment, energy, tele- In addition, the NES establishes an evaluation mod- communications, governmental coordination, and el for other government programs considered strategic foreign policy through the M&E of specific policies, plans, programs, and • 141 strategic actions projects. Incorporation of the monitoring and follow-up • 492 associated goals in the following areas: social system of public investment projects that are linked with and the struggle against poverty, education, health, NDP goals was initiated in 2009. citizen safety, and crime prevention, cultural, finan- Law 8131 on Financial Administration and Public cial, productive, foreign trade, tourism, infrastruc- Budgets clearly indicates certain M&E characteristics that ture, and transportation, science and technology, should be implemented each year by the executive branch employment and social security, financial super- and more concretely, by the Planning Ministry and the vision and monetary policy, environment, energy, Ministry of Finance, which are amplified in their regula- and telecommunications, governmental coordina- tions. In budget formulation, accepted techniques and tion and foreign policy budgetary principles are utilized, based on functional cri- • 95 public institutions with 730 linked goals teria that permit evaluating compliance with annual plans and policies, as well as the incidence and economic-finan- In addition, the NDP includes a Contract with the Cit- cial impact of execution. For this, attention should be paid izens, which constitutes the commitment of the leadership to elements such as provision of services, production of to comply with the established goals. The NDP lies within goods, and general management functions and support of the orienting framework of the institutions’ annual opera- each organ, entity, or institution. tional plans and yearly budgets. Its purpose is to sustain the The Head Office of the National Budget, the Technical president of the republic’s constitutional mandate with re- Secretariat of the Budgetary Authority, and the Ministry of spect to directing, planning, coordinating, and evaluating National Planning and Economic Policy, according to their the strategic objectives, policies, programs, and projects for respective competencies, define the methodology and in- the country’s development. To achieve this, the Planning struments utilized in the evaluation process. For the latter, Ministry designed the Annual Institutional and Sectoral the aforementioned instances will establish the mechanisms Programming, Follow-up and Evaluation Matrix (MAPSE- and communication and coordination that they regard as SI), which, as its name indicates, is the instrument that in- pertinent, to the effect that the requirements for informa- stitutions and sectoral rectorates have at their disposal for tion are the most uniform possible and consistent with the programming and evaluating the advancement of public needs of each, without this implying duplication of func- management in compliance with NDP-established goals. tions. In all cases, the contents of the reports respond to Thus, the state’s strategic planning is only one compo- the technical criteria and guidelines, according to their ar- nent of the mechanism that permits defining goals and the eas of competency, established by the Head Office of the means to achieve these, and this defines institutional be- National Budget, the Budgetary Authority, and the Plan- havior. The remaining basic piece is the monitoring and ning Ministry. evaluation (M&E) of public management, which, in the For the information of the Ministries of Finance and case of Costa Rica, is focused on the strategic actions de- National Planning and Economic Policy, the Head Office fined in the NDP. of the National Budget should prepare a report on the re- 58 S ession 3 : I nstitutional A rrangements for M onitoring and E valuation S ystems in L atin A merica sult of the programs executed during the corresponding “are oriented toward improving the quality of sectoral and economic period. institutional strategic programming, follow-up, and sec- The Ministry of Finance and the Planning Ministry toral evaluation of NDP goals, National Public Investment evaluate institutional management results to guarantee Plan projects, and the institutional budget, generating in- compliance with objectives and goals and the rational use formation of a strategic nature for the evaluation processes of public resource. Evaluation reports are at the disposal of of management, results, and accountability.”21 the comptrollership to measure compliance. At present, a three-party commission made up of the The Planning Ministry remits to the general comptrol- Planning Ministry, the Ministry of Finance, and the gener- ler the final report on the results of execution of the budget, al comptroller, discusses issues such as the guidelines. These compliance with NDP strategic goals, objectives, priorities, discussions have aided in arriving at consensuses and es- and actions, and contribution to the socioeconomic devel- pecially in achieving the placement of evaluation, not as opment of the country. These reports include the explan- a form of auditing or punishment, but rather, as a trans- atory elements that measure the effectiveness of the pro- parency and decision-making tool, which it is necessary to grams, the cost of the services, and the efficiency of the use achieve better public management results. of public resources. A favorable evolution has been achieved in the search As the result of follow-up and evaluation, periodic re- for a public administration that establishes goals and acts ports are issued and presented to the hierarchies of the or- on the basis of the latter, although it is evident that diffi- gans and entities comprising the System of Financial Ad- culties and challenges will persist. One of these is planning ministration of the public sector, with the aim of applying in the longer term, which allows for carrying out evalua- corrective measures to ensure compliance with the pro- tions with greater impact, in addition to creating commit- grammed objectives and goals, and an annual report. ments for those who assume power, not to impose solu- The Planning Ministry’s reports, as established by Law tions, but, conversely, so that those entrusted with power 8131, Articles 52, 55, and 56, are carried out gradually in have goals, with each choosing the pathway to comply with terms of their diverse aspects and evaluation contents. This these goals. gradualism includes two dimensions: coverage areas and evaluation types. Coverage areas of the evaluation processes Advantages and disadvantages of NES are national, regional, sectoral, and institutional. They are Taking the evaluation task established by NES as a refer- incorporated in gradual fashion according to the informa- ence, as well as the law of the Financial Administration and tion available. Gradualness with respect to evaluation types Public Budgets, certain advantages and disadvantages exist are applied in the following order: (1) the evaluation pro- and function from the executive branch. cess of the goals of policies, plans, programs, and institu- tional projects; (2) the evaluation process of the efficacy of The advantages policies, plans, programs, projects, and institutional orga- The M&E process in the hands of the executive branch en- nization; (3) the evaluation process of the efficiency of pol- ables those who implement public policies, and the pro- icies, plans, programs, projects, and institutional organiza- grams and projects contained therein to issue and control tion; (4) the evaluation process of the impact of policies, data that contribute to the system in a firsthand and offi- plans, programs, projects, and institutional organization; cial manner. and (5) the measurement process of the costs of policies, If there is an adequate level of communication, the ad- plans, programs, projects, and institutional organization. vancements and delays reported allow for short-term cor- rections of the original plan at the strategic as well as at the Guidelines operative or programmatic level. To implement each of these aspects, each year the Planning Given the existing set of regulations, the institutions Ministry issues the Technical and Methodological Guide- and the system as a whole are obligated to implement peri- lines for Sectoral and Institutional Strategic Programming odic M&E, which guarantees production of data and fol- and Follow-up, and Sectoral Evaluation. These guidelines low-up. 59 M&E SYSTEMS AND M&E TOOLS The systematic nature of follow-up and evaluation of units that are the most programmatic links of the public the goals of the executive branch could facilitate implemen- evaluation system. tation of evaluations on the long-term impact of policies, with consequent advantages in terms of public policy de- Present and future challenges sign concerning identified demands and design and accord- ing to prior implementation experiences (whether success- Information system ful or not). An information system is needed that will speed up in- formation gathering and exchange between the Planning The disadvantages Ministry public officials and its links, both institutional On specific occasions, modifications to the NDP, even in and sectoral. At present, work is being carried out to create its most programmatic version, could yield better results a platform that the NDP has outlined in such a way that in than expected, collaterally affecting the plan’s strategic the future, it can be accessed with a special user to achieve goals. Political agreements inside the executive power could data incorporation in real time. The Delphos system, select- impede timely implementation of corrective action mea- ed to undertake this task, is “is an administrator of strategic sures or could minimize public information on advance- management indicators, commonly known as the Balanced ment of the NDP. Scorecard, which permits visualization, in very simple fash- The current institutional framework separates the de- ion, of the strategic state of the organization, enormously sign of the plan from that of the budget, and similarly sep- facilitating its follow-up and compliance,” which is com- arates, despite efforts, the system and the focus of the eval- patible with the task performed to date.22 uation of the plan for expense liquidation reports. This has been accompanied by various practical consequences, with Culture change emphasis on the following: Rejection of evaluations is common, because evaluation is perceived as synonymous with punishment and negative • An evident lack of linking between government consequences, and backwardness in thought that requires priorities and the endowment of systematic re- modification. The new culture of evaluation should not sources for these lead to discrediting an institution or a program, but rath- • The impossibility of public implementation from er should perfect and use it to greater advantage. Therefore, a multiyear perspective of goals and budget, which evaluation becomes a required process for the greatest ex- displays a more strategic than programmatic char- ploitation of the state’s resources. acter, despite the design and due to an annual bud- getary focus that impedes a vision of the future Evaluation and the project cycle • A shallow capacity for interinstitutional coordina- Evaluation must be established as a part of the project cycle tion (Finance Ministry and Planning Ministry), and not an optional element, to avoid developing programs which is overcome by the maintenance of public in which no evidence of the results and impacts, negative or administration “ghettos” and which is due to the positive, is ever produced. One practice that has been car- lack of the previous links ried out in countries such as Mexico is that of allocating a percentage of the budget to the evaluation of the budget it- On the other hand, a sectoral focus of government or- self, ensuring that the funds are available to prove whether ganization has been put into practice since 2006, and sec- the direction that has been chosen is appropriate, or wheth- toral links have been established, including those respon- er a change is required. sible for the planning, monitoring, and evaluation of the sector and its institutions. This institutional design, how- Short-, medium-, and long-term planning ever, does not have sufficient accumulated experience to Planning in the short, medium, and long term encourag- date. Thus, it is necessary that it be strengthened organical- es more revealing evaluations that will make it possible to ly and that it be distinguished from institutional planning make changes in the direction the administration has tak- 60 S ession 3 : I nstitutional A rrangements for M onitoring and E valuation S ystems in L atin A merica en at all stages of the projects and policies. Deep reflection U.S. Monitoring and Evaluation Trends: on the needs of the country will draw the government clos- Using Performance Information er to a planning and evaluation orientation and toward in- to Improve Results tegrated development. U.S. President Barack Obama is committed to results and Improved routine planning and evaluation processes is taking steps to improve on the accomplishments of his will lead to a public administration that is better able to re- predecessors. On January 20, 2009, in his Inaugural Ad- spond to goals and objectives and to employ the budget in dress, President Obama said, these terms. This means a change in mentality among man- agers and users, and the utilization of information systems The question we ask today is not whether our gov- that facilitate contact between those interested in the eval- ernment is too big or too small, but whether it uation and the planning establishment at all levels and at works—whether it helps families find jobs at a de- all centers. cent wage, care they can afford, a retirement that is dignified. Where the answer is yes, we intend to System sustainability move forward. Where the answer is no, programs The sustainability of the M&E system is guaranteed mainly will end. And those of us who manage the public’s by strengthening the sectoral planning and evaluation sec- dollars will be held to account—to spend wisely, tor and its link with NES. This agreement toward achiev- reform bad habits, and do our business in the light ing sustainability has to do with the following: of day—because only then can we restore the vi- tal trust between a people and their government. • Maintaining the present decree of sectoralization, ensuring its being in effect in the future Over the past 50 years, the U.S. Congress and previous • Ensuring sectoral decentralization of the automat- administrations have laid important groundwork for gov- ed information system in favor of the Delphos ernmentwide performance improvement, including most M&E system recently the Government Performance and Results Act • Naming sectoral links in an organic manner that is (GPRA) enacted in 1993 and the Program Assessment Rat- distinct from institutional links ing Tool (PART) introduced by President George W. Bush • Strengthening and updating NES M&E tech- in his 2003 budget. niques and practices, orienting these toward re- sults-based management, in which the plan deter- The Government Performance and Results Act mines the budget The GPRA of 1993 requires every federal agency to set stra- tegic and annual goals (dealing with societal outcomes, not Costa Rica is making efforts to shift toward results- just agency activities), measure performance, and report to based management, strengthening the culture of evaluation Congress, the Office of Management and Budget (OMB), and its advantages; however, many efforts are pending. The and the public on progress relative to selected goals. Feder- way to better public management is outlined; what Costa al agencies must identify both long-term and annual goals, Rica, and in general, Latin America, needs is a willingness collect performance data, and justify budget requests based to assume the commitment of proposing quantifiable goals on that data. that permit evaluation, which makes the perfecting of proj- Under GPRA, each federal program was required for ects, policies, and programs possible, and that encourages the first time to identify explicit measures and goals for accountability. judging program performance and to collect performance information annually to determine whether the program Carla Morales-Rojas was meeting those goals. Senator William Roth, sponsor of Vice Minister, Ministry of National Planning and the measure, stated at the time of its enactment that the act Economic Policies, Costa Rica would bring about “accountability by federal agencies for the results they achieve when they spend tax dollars.” 61 M&E SYSTEMS AND M&E TOOLS Unfortunately, the implementation of this law has not cause many believed that the answer to this question would met its authors’ hopes. Agency plans suffer from perfor- vary depending on the respondent’s or the reviewer’s phil- mance measures that are meaningless, vague, too numer- osophical outlook, the question was removed so that the ous, and compiled too often by people who have no di- PART would yield a more objective rating. rect connection with budget decisions. Agencies produce thousands of pages of performance plans every year that are Building on the past: Going beyond GPRA and largely ignored in the budget process. PART OMB’s current Deputy Director for Management Jeff The Program Assessment Rating Tool Zients has stated that both GPRA and PART placed too With the creation of PART within the OMB, President much emphasis on producing performance reports for its Bush attempted to carry out the spirit as well as the letter of own sake and too little attention on analyzing and acting GPRA. The Bush administration created the PART to shift on that information. Zients has stated that the federal gov- the focus of federal goal-setting and measurement from the ernment should use program performance information to agency level to the program level. In the 2003 Budget, the achieve the following: Bush administration rated approximately 130 federal pro- grams on their effectiveness. Within five years virtually ev- • Communicate priorities, progress, and raise issues ery federal program was rated—more than 1,000 individu- • Illuminate what works that should be continued al programs, representing 98 percent of all federal spending. and what does not work that needs attention This attempt to rate the effectiveness of all federal pro- • Motivate the best from our workforce and our ser- grams on an objective and consistent basis was far from vice delivery partners perfect. The criteria used to rate programs were not uni- • Allocate scarce resources wisely form and ratings were based on limited information. Its influence on budget decisions was limited. Nonetheless, Shelley Metzenbaum, OMB’s current associate director with PART, the federal budget process adopted a new tool for performance management, in a study prepared before for rating program performance that in successive years at- the 2008 presidential election concluded that both GPRA tempted to generate more and more extensive information and PART failed to offer a comprehensive way to see how on program performance that was more objective, more the federal government is performing and to identify the credible, and more useful than had been available ever be- agency goals and program targets (Metzenbaum 2009). De- fore. In cases in which a program was rated, the results of spite large amounts of GPRA and PART performance mate- the rating process often influenced program policy and rial, she found that it is difficult to find meaningful govern- management decisions as well as the budget request. ment performance information, for example, performance PART is a multipart questionnaire that was developed levels, performance trends, and even targets, because too lit- by career staff within OMB. The questionnaire was com- tle attention has been paid to communicating targets and pleted by small teams of career professionals who worked trends and too much attention has been paid to communi- within each program being assessed at larger agencies cating the “percentage of targets met” as the primary indica- across the government. Each team completed the survey tor of overall performance. Metzenbaum further asserts that instrument with the assistance and oversight of profession- too little attention was paid in these initiatives to under- al OMB career staff (“program examiners”) whose regular standing the size and characteristics of problems and why responsibilities largely focused on oversight of budget and performance levels changed. She states that few agencies and policy development and execution for the program. programs routinely analyzed their performance and other The goal of the PART initiative was to devise a tool that data, as businesses do, under these initiatives to generate in- was objective and easy to understand. Most important, the sights needed to improve performance. In particular, she ar- findings had to be credible, useful, and ideologically neu- gues that the program review process has been overly subjec- tral. For instance, the first draft PART questioned whether tive, creating unfair inconsistencies and frustrating disputes a particular program served an appropriate federal role. Be- about what constitutes appropriate measures, targets, and 62 S ession 3 : I nstitutional A rrangements for M onitoring and E valuation S ystems in L atin A merica evaluation methods. She asserts that too little attention has mance and other relevant data pertaining to presi- been paid to identifying key audiences for goals and perfor- dential, cross-agency, agency, and program targets. mance data, to determining their performance information • Appoint experienced performance managers to key needs, and to delivering information where and when it is government management positions, and recruit needed in a format the target audiences can understand. As experts to provide counsel to cabinet secretaries. a consequence, key audiences, including Congress, field of- Identify and manage cross-agency targets and mea- fices, delivery partners, and others, have not received the sures where performance improvement needs cross- performance information that they need. Finally, she asserts agency attention and cooperation. that under GPRA and PART, too much attention has been • Adjust accountability expectations. The president given to program review, assessment, and control and too should hold agencies accountable for the persistent little attention to providing expert advice and to stimulating application of evidence, intelligence, and effort to innovation, discovery, cooperation, and assistance achieve continual performance gains. Guiding principles to improve federal performance Recommendations for the OMB management • Direct agencies and programs to communicate tar- Metzenbaum concluded that the key performance manage- gets and performance trends for key indicators. ment challenge is to use—not just produce—performance • Create a federal performance portal with easy-to- goals and measures. She enunciated four guiding principles find performance targets, trends, and other relat- on which to base changes to current federal performance ed information. management efforts: • Encourage outside expertise and multiple perspec- tives on selection of targets, performance measures, • Communicate performance trends and targets, not and strategies to improve performance. target attainment and ratings • Build communities of practice to facilitate cross- • Encourage performance improvement with in- agency learning. creased diagnostic analysis, practical experiments, • Increase training in performance management and knowledge sharing practice and analysis. • Present information in ways that meet the needs of • Revise, but continue PART, shifting the emphasis specific audiences from program rating to performance improvement. • Structure accountability mechanisms to encourage • Remove the centrality of “rating” and “assessment,” and inspire, not embarrass, reprimand, or punish and shift attention to performance improvement and communication, not target attainment for its Metzenbaum then outlined a series of interconnected own sake. recommendations for the president and for OMB to put • Eliminate barriers that complicate audience-fo- these principles into practice. cused delivery of performance reports and informa- tion. Recommendations for the resident • Continue the President’s Management Council as • Clearly identify presidential priority targets. Meet a forum for senior agency deputies to meet regu- at least quarterly with each responsible cabinet larly to discuss progress toward performance and secretary to keep agencies focused on the targets. management priorities and to reduce management Appoint a chief performance officer and create a risks. White House Performance Unit working closely with OMB to advance progress on presidential pri- Recommendations for cabinet secretaries and orities. agency heads • Run goal-focused, data-driven meetings pertaining • Immediately review agency performance trends to the priority targets. Increase analysis of perfor- and update priority targets. Each cabinet secretary 63 M&E SYSTEMS AND M&E TOOLS and agency head should review and refine their or- • Regularly convening agency program management ganization’s strategic and annual targets to reflect personnel to assess and improve program perfor- and communicate the new administration’s pri- mance and efficiency orities, informed by a review of past performance • Assisting the head of the agency in the develop- trends. ment and use within the agency of performance • Run goal-focused, data-driven meetings. Each cab- measures in personnel performance appraisals, par- inet secretary and agency head should run their ticularly those of program managers, to ensure real own goal-focused, data-driven meetings to keep the accountability for greater effectiveness organization focused and continually searching for opportunities for improvement. Metzenbaum recommended that the interagency Per- • Identify information needs of key audiences. Cab- formance Improvement Council inet secretaries and agency heads must ensure that their organizations identify key audiences for feder- • Lead a review of PART. The Performance Improve- al performance information, determine their needs, ment Council should be directed to lead a process and establish priorities among the audience. to propose changes to PART for subsequent action • Improve federal information presentation and dis- by OMB. This should include revising and renam- semination capacity. Each cabinet secretary and ing the process to shift the emphasis from program agency head should pay increased attention to the rating to performance improvement, fixing some presentation, dissemination, and use of perfor- the questions, and adding a few new questions. mance information to communicate more effec- • Consider specific revisions to PART. The Perfor- tively with targeted audiences and inform their pri- mance Improvement Council should consider the ority-setting and performance improving decisions. following recommendations concerning revisions • Create agency Web-based performance portals. to the PART process: Each cabinet secretary and agency head should di- • Align program targets with GPRA goals and allow rect their organizations to add a performance por- agencies to define what constitutes a program tal on their home pages that makes it easy to find • Revise the PART scoring system and eliminate the performance targets, trends, and other related in- ratings formation. • Increase reviewer perspectives • Reorder and revise PART questions to be served Recommendations for the Performance Improvement Council State and local government models On November 13, 2007, President Bush signed an execu- OMB’s performance managers are particularly impressed tive order on improving government performance that es- by successful efforts at the state and local government level tablished an interagency Performance Improvement Coun- to use performance information to monitor, evaluate, and cil composed of performance improvement officers from manage government programs. Local governments—in- every major federal agency who were responsible for co- cluding New York City; Charlotte, North Carolina; Bal- ordinating the performance management activities of the timore, Maryland; King County, Washington; and Austin, agency, including the following: Texas—have demonstrated how performance information collected and analyzed with the aid of management tools • Developing and improving the agency’s strategic like CompStat and CitiStat can be used to improve out- plans, annual performance plans, and annual per- comes, reduce crime, increase housing starts, and drive formance reports, as well as ensuring the use of down costs. With these tools, managers lead frequent goal- such information in agency budget justifications focused, data-driven staff meetings that review statistics • Ensuring program goals are aggressive, realistic, about outcomes, costs, unwanted side effects, patterns of and accurately measured problems in an area, probable causes, apparent effects of 64 S ession 3 : I nstitutional A rrangements for M onitoring and E valuation S ystems in L atin A merica prior actions, and future plans. For example, Baltimore’s ta about what works and then act on evidence that Office of CitiStat is a small performance-based manage- emerges—expanding the approaches that work ment group responsible for continually improving the qual- best, fine-tuning the ones that get mixed results, ity of local government services of all kinds. Staff analysts and shutting down those that are failing.24 conduct studies to identify areas in need of improvement. Beyond investing more in evaluations, we are Department managers participate in regularly scheduled also designing new initiatives with evaluation presentations with the mayor and other top city officials standards built in using a two-tiered approach— designed to maximize accountability.23 providing more money to programs that gener- ate results backed up by strong evidence. That is Planning for the FY 2011 budget and performance the top tier. Then, for programs, with some sup- plans portive evidence but not as much, we’ve said: Let OMB intends to develop an improved performance man- us try those too, but rigorously evaluate them and agement framework aligning high-priority performance see whether they work. If not, we’ll redirect their goals, GPRA performance reporting, and many program- funds to, more promising efforts. This two-tiered level PART measures. It will focus on outcomes, allow structure will provide objective criteria to inform comparisons across programs and agencies, and illustrate our decisions on which models to invest in. It will performance trends over time. It will focus on information also create the right incentives for the future. Or- supporting long-term decisions about targets and organiza- ganizations will know that to be considered for tional strategies, as well as key decision makers in the ser- funding, they must provide credible evaluation re- vice delivery chain, including those in federal field offices, sults that show promise, and be ready to subject state and local partners, nonprofit grantees, and contrac- their models to analysis. As more models move in- tors. It will use new information technologies to make this to the top tier, it will create pressure on all the top- more feasible, less cumbersome, and far more useful than tier models to improve their effectiveness, so they past alignment efforts. In addition, the administration is continue to receive support. By instilling a culture proposing historic investments in comparative effective- of learning into federal programs, we can build ness research and evaluations, integrated with performance knowledge so that spending decisions are based management efforts. not only on good intentions, but also on strong evidence that carefully targeted investments will Increased emphasis on program evaluations produce results. In June 2009, OMB Director Peter Orszag wrote in his reg- ular blog on the OMB Web site that New evaluation initiative Most agencies review ongoing programs, but not robust- [W]e have not been making the right investments ly enough to provide meaningful data about outcomes and to build a new foundation for economic prosper- progress toward the objective. Many agencies lack the ca- ity—and we need smarter investments in educa- pacity to carry out a rigorous, strategic research agenda, and tion, health care, and social services. Much of the policy priorities are established without evidence to back time, it is hard to say whether a program is work- them up. Programs are continued year after year, without a ing well or not. Many initiatives drive funds to the hard, objective look at their effectiveness. local level, but do not track how they are spent; On October 7, 2009, OMB asked agencies to volun- others spend dollars bit by bit, so the results are teer for an initiative designed to strengthen rigorous, ob- hard to see. Too often rigorous evaluations do not jective assessments of existing federal activities to improve happen. This has to change, and I am trying to results and better inform funding decisions. Ongoing pro- put much more emphasis on evidence-based poli- gram evaluation research will be published online, and an cy decisions here at OMB. Wherever possible, we interagency task force will identify and shape evaluations should design new initiatives to build rigorous da- of programs that cross over several agencies. Participants 65 M&E SYSTEMS AND M&E TOOLS will serve as demonstration projects through which we can ory, staff members have been told that the annual budget test approaches to improve program effectiveness and ef- reviews with the director for all parts of the federal budget, ficiency, share best practices, and further improve perfor- not just social programs, must give prominent attention to mance. After assessing the initiative in fiscal year 2011, the quantitative evidence of program performance and results. administration will implement governmentwide evalua- As a top manager said in a meeting on the program evalu- tion metrics. ation initiative this week, this is a very “fluid and dynamic In addition, OMB will work with agencies to make in- process,” and more developments are sure to come. formation readily available online about all federal evalua- tions focused on program impacts that are planned or al- John R. Pfeiffer ready under way. Along with other White House offices, Program Examiner, Energy Branch, OMB will establish a new interagency working group to Office of Management and Budget, United States promote stronger evaluation across the federal government. OMB will allocate a limited amount of funding for agen- cies that volunteer to Institutional Arrangements for M&E Systems in Latin America: The Case of Mexico: Performance • Show how their fiscal year 2011 funding priorities Evaluation System are evidence-based or otherwise subject to rigorous evaluation Background • Assess their own capacity to support evaluation and In Mexico, the tradition of carrying out monitoring and suggest pathways for strengthening that capacity evaluation (M&E) programs and projects dates back to • Propose new evaluations that could improve gov- the 1970s. The Secretariat of Finance and Public Credit ernment programs in the future and the Bank of Mexico promoted the evaluation of pro- • Identify impediments to rigorous program evalua- ductive and infrastructure projects through development tion in their statutes or regulations banks, with assistance from the World Bank and the Inter- American Development Bank (IDB). Efforts to monitor The initiative initially will focus primarily on impact and evaluate the management of public entities were en- evaluations, that is, evaluations aimed at determining the hanced in the 1970s with the 1976–82 Administrative Re- causal effects of programs. Although the administration is form Program, which introduced a new planning and con- committed to improving a wide range of evaluation and trol model that grouped the various offices and entities of performance measurement activities, the initiative initially the federal public administration into various sectors, orga- will focus primarily on social, educational, economic, and nizing them into sector-based groups and establishing the similar programs whose expenditures are aimed at improv- fundamental elements of the planning, programming, bud- ing life outcomes (such as improving health or increasing geting, and evaluating cycle. productivity) for individuals. OMB will consider on a case- In 1982, the Mexican government created the Secretari- by-case basis the inclusion of evaluation efforts in other at of the National Comptroller General (SNCG), a govern- spheres, such as procurement, construction, taxation, and ment office with a broad spectrum of responsibilities and national defense. Because drug and clinical medical evalua- functions, and especially the operation of the National Sys- tions independently have received extensive study, they are tem for Government Control and Evaluation. This system not included. operated through the internal control bodies within feder- al public administration offices and entities, and the comp- Conclusion troller’s offices in state governments, as well as through This is an exciting time for those interested in improving public commissioners and external auditors. Although its program evaluation in the federal government and in see- responsibilities included verifying the efficiency, effective- ing that the results of those studies have a more direct im- ness, and operational economy of the offices and entities, pact on federal decision making. For the first time in mem- it was essentially more oriented toward the auditing and 66 S ession 3 : I nstitutional A rrangements for M onitoring and E valuation S ystems in L atin A merica supervision of public spending. In 1986, the Federal Law sults, within a framework of greater normative flexibility on Para-State Entities went into effect, and in 1990, the (see Figure 11). Mexican government created the National Program for the The NPS was defined as “an instrument for planning, Modernization of Public Enterprises. programming and budgeting, with the objective of improv- In the 1990s, an important change directed by the Sec- ing the quality of results-based spending, through the re- retariat of Finance and Public Credit was implemented. composition of programmatic structure.” With the NPS, While the control functions carried out by SNCG con- the federal government sought to achieve greater interac- tinued, a strong impulse was given to the systematization tion among budgeting authorities in government offices of both monitoring and evaluation before, during, and af- and entities as well as among the units responsible for and ter the completion of programs and projects. The Mexican the individuals charged with strategic planning. The bud- government incorporated the socioeconomic evaluation of get was defined as a vehicle for establishing greater links be- projects so that, in the structuring of infrastructure proj- tween the objectives of public policies and those making ects, prior formulations and evaluations would be carried expenditures. With these elements, the goal was to improve out to demonstrate whether or not the investments pro- implementation, control, evaluation, and accountability posed were socially profitable. Development banks, par- in public spending. The NPS included new programmat- ticularly the National Public Works and Services Bank, ic categories such as the following: function, sector-based were pioneers in ensuring that socioeconomic evaluations program, institutional activity, and investment project. The were conducted for the infrastructure projects linked to the mission, objective, and indicator goal were considered in Fund for Investment in Infrastructure and selected for fi- the new programmatic elements. nancing (loans, venture capital, nonreturnable financing, The use of indicators to improve the achievement of or a combination of the three). strategic objectives in national planning and efficiency in During the 1995–2000 period, the composition of the management of public spending has followed an evo- public spending changed, with increasing resources used lutionary process that has prepared the federal public ad- by legislative and judicial branches of government, com- ministration for more advanced phases in conducting eval- pared with those used by the executive branch. Specifical- uations. Between 2000 and 2006, the NPS encouraged the ly, federal resources transferred to states and municipali- use of performance evaluations, urging government offic- ties increased, and resources corresponding to the federal es and entities to report information on their progress in level decreased. This strengthened the orientation of pub- reaching goals. Indicators were reported on the basis of lic spending toward social development with a regional perspective. The rules of operation for the programs that FIGURE 11: grant subsidies and transfers to the population were pub- Performance Evaluation System, First Stage lished for the first time in 1999, with the aim of assuring their efficient, effective, fair, and transparent application, Performance agreements and achieving the anticipated goals and objectives, their ap- plication in the target population, and the corresponding evaluation of results. The process of reforming the budgeting system be- Audits PED components gan in 1998. A New Programmatic Structure(NPS) was (Evaluation of government programs) established with the aim of contributing better quality information on public programs and the orientation of Information Technology spending. With the establishment of performance agree- ments between the Secretariat of Finance and Public Credit, the Civil Service Secretariat, and semiofficial en- Surveys of beneficiaries tities, institutional incentives were established for these government entities to make decisions with a focus on re- Source: Author. 67 M&E SYSTEMS AND M&E TOOLS institutional activity and the responsible unit, and conse- At the same time, the performance evaluation system quently the monitoring of goals was carried out with a di- (PES) was conceptualized as “the set of methodological el- versity of priority programs, projects, and activities that ements that allows for making an objective assessment of added this cost into the corresponding programmatic cat- program performance, in line with principles focused on egories. From 2005 to 2006, federal public administration verifying the degree to which goals and objectives are be- offices and entities reported information on meeting their ing met, and based on strategic and management indicators goals on the basis of selected indicators. that make it possible to determine the social impact of pro- grams and projects.” The two main components are (1) the Performance evaluation system: results-based objective evaluation of federal programs, and (2) the ongo- management strategy ing evaluation of the functioning and results of public in- With the leadership of Secretariat of Finance and Public stitutions. To strengthen this system, important legal and Credit, it was determined that the programming-budget- institutional measures were carried out, and the necessary ing process would be carried out under the results-based support was provided to turn this system into a starting management (RBM) strategy, which uses performance in- point for major administrative reform. Beginning in 2007, formation to improve decision making, incorporates stra- the implementation of this redefined system was promot- tegic planning tools, applies logical models, and monitors ed actively by the federal government, with support from and evaluates results. The strategy focused on (1) concen- the National Congress, and aimed at making the constitu- trating decisions on results, and aligning strategic planning, tional and legal reforms necessary for its implementation programming, monitoring, and evaluating with results; (2) to be comprehensive. The participation of multilateral de- orienting information systems toward results; (3) focusing velopment banks, both the IRDB and the IDB, has been management on results; and (4) making use of informa- important. In June 2007, the Secretariat of Finance and tion for learning, decision making, and accountability. The Public Credit, representing the Mexican federal govern- strategy was conceived as a strategy with the capacity to be ment, signed the Non-Reimbursable Technical Coopera- used in all public entities. tion agreement (No. ATN/OC-10476-ME) with the IDB- FIGURE 12: Results-Based Management Strategy • Center decisions on results in all phases • Align strategic planning, monitoring and evaluation with results • Keep measurement and information simple • Manage for, not by, results • Use information from results for learning and decision-taking as well as accounting Results-based Management • Link to PND (NDP) and its programs Results based budget Results based Budget • Strategic objectives of the office (RbB) Indicators for results • Budgetary programs New budget process organization • Indicator Matrix SHCP Results-based evaluation allocation Performance Evaluation Program Management System Evaluation improvement Types of evaluation Framework program Annual evaluation program Institutional performance Monitoring and Evaluation System evaluation Commitment agreements For results To improve management Source: Author. 68 S ession 3 : I nstitutional A rrangements for M onitoring and E valuation S ystems in L atin A merica to carry out the implementation of a Results-Based Budget monization of the government’s accounting system System (RBBS) in the federal public administration. for the three levels of government, and earned val- The RBM strategy to carry out the RBBS-PES was part ue accounting records, there will be a solid source of a much more comprehensive effort in the country, ac- of information for strengthening budget allocation companied by a state vision that promoted efficient, ef- and accountability processes. fective, and accountable government. The Comprehen- sive Reform Initiative of the Public Treasury, presented by Results-based budgeting the federal executive branch to the National Congress on Results-based budgeting (RBB) is a central element in the June 20, 2007, for its consideration, included measures for implementation of the PES. The process of developing the strengthening tax collection and greater coresponsibility Federal Expenditure Budget (FEB) Project for fiscal year through participation by state-level governments, as well as 2008 made adjustments to the programmatic structure. better and more transparent public spending. This initia- Two adjustments especially worth mentioning were the cre- tive was approved on September 13 of that same year by the ation of the budgetary program to replace the institution- House of Representatives, and was given the name “Trea- al activity category, and the implementation of the objec- sury Reform for the Most Disadvantaged.” tive-indicator-goal model. The RBB was defined as “the set Behind this reform is the idea that taxpayers should of activities and tools that make it possible for the decisions have greater certainty that the taxes they pay are used well, involved in the budgetary process to systematically incor- and that they are applied to government programs and in- porate considerations regarding the obtained and expected vestment projects that offer tangible results to the popula- results from the use of public resources, and that motivate tion, as proposed in the 2007–12 National Development government offices and entities to achieve such results, with Plan (NDP), the 2008–12 National Program for Develop- the objective of improving the quality of public spending ment Financing, and the Live Better strategy. The follow- and accountability.” ing are the most outstanding elements in the reform: Institutional framework • To grant constitutional status to the obligation for With the Federal Budget and Fiscal Responsibility Law, the three levels of government (federal, state, and constitutional reforms of public spending and supervision, municipal) to ensure that the economic resources and the subsequent legal reforms in the area of the RBB- at their disposition are evaluated by technical en- PES, an institutional framework was established to provide tities created by the federal and state governments. greater legal and economic certainty and transparency in • To establish that federal economic resources used the overall budgetary process. Simultaneously, a combined by states and municipalities will be subjected to the M&E system was established, with the necessary network- provisions in regulating laws, specifically the Federal ing and coordination elements based on provisions current- Budget and Fiscal Responsibility Lawthe Fiscal Co- ly in force, with the participation of the Secretariat of Fi- ordination Law, and an assessment by technical enti- nance and Public Credit, the Civil Service Secretariat, the ties. In other words, the use of resources will be sub- National Council for the Evaluation of Social Develop- jected to stipulations for performance evaluations ment Policy (CONEVAL), and the House of Representa- aimed at verifying the degree to which goals and ob- tives, through the Federal Auditing Department. jectives are being met, on the basis of strategic and At the same time, federal public administration offices management indicators, among other aspects. and entities have the responsibility, through higher clerk- • To establish the foundations for the uniform pre- ships or the equivalent, to plan and evaluate budgetary pro- sentation of information on income and expenses, grams and administrative units responsible for public pol- as well as assets, by the federal, state, municipal, icies and related federal programs. In accordance with the and Mexico City governments. One of the main normative structure of the RBB-PES, regulators and those achievements is the General Law on Government responsible for making expenditures collaborate. In prac- Accounting of December 31, 2008. With the har- tice, collaboration has been increasing as training programs 69 M&E SYSTEMS AND M&E TOOLS have advanced. In this collaboration between regulators makes it possible to identify the allocations and fi- and expenditure-makers, another factor has had a decisive nal destination of federal public spending, based on impact—that is, the importance placed on performance the classification of groups and modalities. evaluation in expenditure budget projects as an important • Regulating the evaluation of federal programs. element in budgetary decision making. With the issuance of “General Guidelines for Eval- The PES’s combined institutional design has important uating Federal Programs of the Federal Public Ad- advantages. On the one hand, it makes it possible for the ministration,” the Annual Evaluation Program executive branch to exert decisive influence on program- (AEP) was created. Its objective is to refine perfor- ming by linking the NDP and its programs with the strate- mance evaluation as an instrument for gradual im- gic objectives of government offices and entities, and with provement in the quality of public spending. budgetary programs, and by adopting qualitative improve- ments in programming structures. It allows for consider- This work made it necessary to establish the necessary ing results when making budgetary allocations; improving types of evaluation as well as the development of a set of the management and quality of public spending; providing methodologies for performance evaluation that would fa- follow-up to budgetary management; improving policies, cilitate the systematic, comparative evaluation of budget- programs, and institutions; and strengthening the account- ary programs. It was also established that the evaluations ability process. In addition, it makes adequate external su- of budgetary programs would be conducted by indepen- pervision possible, conducted by the Federal Superior Au- dent consultants. Also established was the requirement to diting Office, through the Special Performance Auditing make evaluation results public, making them available on Office, which, according to the Federal Budget and Fiscal the Web sites of the Secretariat of Finance and Public Cred- Responsibility Law, covers the control and evaluation of it, the Civil Service Secretariat, and CONEVAL. It was al- federal public spending, among other tasks. so necessary to With the recent publication of a September 2009 decree that reforms, adds to, and repeals various provisions in the • Establish mechanisms for monitoring evaluations. regulations for theFederal Budget and Fiscal Responsibili- Implemented in October 2008 were “Mechanisms ty Law, PES coordination is enhanced through the creation for monitoring aspects susceptible to improvement, of a permanent working group in evaluation and monitor- derived from external reports and evaluations of ing of budgetary performance, and the results of that per- federal programs.” formance. This group is composed of the Secretariat of Fi- • Develop the necessary systems for monitoring the nance and Public Credit, the Civil Service Secretariat, the goals for indicators established in the FEB pro- President’s Office, and CONEVAL. grams. The functionality of validation, review, de- finitive updating, and selection of budgetary pro- Implementation of the RBB-PES gram indicators was developed in the RBB-PES Various actions in the implementation of the RBB-PES, module included in the Secretariat of Finance and carried out to date, make the following possible: Public Credit’s Web site. • Train and educate public servants for participation • Modifying the budgetary process to be able to iden- in the RBB-PES implementation process, through tify the destination of public spending, and link- long-distance training and regular courses on Logi- ing such spending with strategic objectives in na- cal Framework Methodology. tional development planning. Budgetary programs were redefined to facilitate the measurement of re- Matrix of Results-Based Indicators sults. The programmatic category of “Budgetary Encouraging changes have been achieved through the pol- Program” was created to replace the previous “Pri- icy of strengthening PES. For fiscal year 2008, 398 of the ority Activity” category, which consisted of four ge- 1,183 budgetary programs had matrix of results-based in- neric modalities. The new programmatic category dicators (MRIs), with a total of 3,290 indicators. The fed- 70 S ession 3 : I nstitutional A rrangements for M onitoring and E valuation S ystems in L atin A merica eral public administration offices and entities identified type of resources allocated, as well as the proposed results 265 indicators, corresponding to 217 budgetary programs, for the benefit of society. In addition, 294 strategic indica- as most representative for measuring and assessing perfor- tors that most adequately measure the objectives of these mance. For 2009, 512 of the 1,108 budgetary programs programs were identified and included in the 2010 FEB have MRIs, with a total of 4,634 indicators. Of these, 246 project. It is important to point out that the indicators se- strategic indicators corresponding to 160 priority budget- lected correspond to the most relevant budgetary programs ary programs were selected. During the process of develop- in federal public administration, to the degree that they di- ing the FEB Project for the 2010 fiscal year, federal pub- rectly or indirectly provide federal subsidies to target pop- lic administration offices and entities, in collaboration with ulations. the Secretariat of Finance and Public Credit, Civil Service Secretariat, and CONEVAL, reviewed and improved the Annual Evaluation Program MRIs, particularly those with 2009 selected indicators, to The AEP brings together the evaluation proposals from summarize and thereby achieve greater certainty in the in- the Finance and Public Credit and Civil Service Secre- formation generated. MRIs were registered for 550 budget- tariats and CONEVAL, addressing the evaluation strat- ary programs, with 4,422 indicators that will serve as the egies used by each of them. In 2007, 2008, and 2009, basis for the PES during the 2010 fiscal year. a total of 126, 192, and 196 evaluation strategies have Of these indicators, 294 were selected from 199 pro- been used, respectively. While design evaluations domi- grams, considered to be the most representative of the fed- nated during 2007, specific performance evaluations be- eral public administration, both due to the amount and gan to take shape in 2008 and 2009, demonstrating FIGURE 13: Performance Evaluation System, Second Stage SED, SECOND PHASE Legal Framework PND & ITS Performance Indicators Programme Implementation Programmes Planning Strategic Management Design of: • These are the expresion of the objectives & results expected or desireable to be achieved by the Budgeted Exercise Politcies and Programmes programme Clear Definition of: • These are a referentto measure the advances reached by the objectives Objectives and Results & results during programme operations (follow-up) • These are relevant means of SED To those having allocated resources: information for the evaluation of FOLLOW-UP policiesand programmes EVALUATION Budgeting (during • These are an instrument for (of results) implementation) accountability and transparency PbR Indicator Targets: Budgetary Decisions Baseline Policies Programmes 2008 Resources Reallocation Performance of Institutions Allocated of Resources 2012 Vision 2030 Quality of Accountability Transparency the Expense Continuous Improvement Source: Author. 71 M&E SYSTEMS AND M&E TOOLS progress in meeting program goals and objectives on the • Monitoring: (1) monitoring budgetary programs, basis of a summary of information contained in the PES and their objectives, indicators and goals; (2) pre- and through an analysis of strategic and management in- paring reports on progress made in meeting the dicators that made it possible to verify the results of the goals for budgetary program indicators. use of budgetary public resources. The AEP for 2010 has • Evaluation: (1) systematic evaluations in accor- a strategy different from those used in previous years. The dance with the AEP, the results of which have im- purpose of the 2010 AEP is to expand the scope of the proved the design of the budgetary program, and in evaluation of the federal public administration, by pro- some cases, its operation and impacts; (2) commit- gramming evaluations of areas not previously included. ments for results and improvements in budgetary Examples are the Interior Secretariat (Area 4), the Foreign program management, based on aspects susceptible Relations Secretariat (Area 5), the Secretariat of Commu- to improvement derived from evaluations. nications and Transportation (Area 9), and the Federal At- • Transparency and accountability: public accounts torney General’s Office (Area 17). Also acquiring impor- that bring together information on the monitoring tance in the 2010 EAP is the strategic evaluation of social of goals for indicators in each budgetary program. protection policies, microcredit policies, employment pol- • Impact: for information on program performance, icies, and highway subsector policies. generated by M&E indicators, to have an impact on budgetary allocations, it is necessary to make Continuous improvement ongoing efforts in consolidating the quality of the In the context of the implementation of the RBB-PES, a information incorporated into the PES, expanding series of improvements have been incorporated into the the use of the MRI in analyzing program perfor- budgetary process. Together, these improvements make it mance, and rooting results-based management in possible to link planning, programming, budgeting, use, the daily activities of public servants, in the annu- monitoring, evaluating, transparency, and accountability, al budget proposal, and in the program, and proj- exerting influence on the systematic feedback from pro- ect review process. grams and contributing to the analysis, discussion, and decision making regarding resources allocated to the pro- Performance evaluation at the local level grams. These improvements consist of the following: At the local level, during fiscal year 2009, the initial per- formance evaluations observed significant advances in the • Planning: (1) alignment of budgetary program ob- implementation of strategic and management indicators. jectives with the objectives and indicators of the They were defined for General Area 33, Federal Contribu- NDP and its programs; (2) Alignment of budget- tions for States and Municipalities, allocated in the context ary program objectives with the strategic objectives of the existing schemes for intergovernmental coordination of the government office or entity. and with the participation of coordinating offices for each • Programming: (1) defining the modality of bud- fund. A total of 29 indicators with varying frequencies were getary programs; (2) integrating the budgetary pro- agreed on for the eight funds included in the area. Of these, gram into the programmatic structure; (3) devel- 18 are strategic and 11 are management indicators. State oping and updating the MRI for the budgetary and municipal governments report progress in the indica- program, including the creation and continuous tors through the information system designed by the Secre- improvement of strategic and management indica- tariat of Finance and Public Credit. This progress is report- tors. ed to the National Congress through quarterly reports. In • Budgeting: specific budgetary allocation for each the near future, the MRIs and performance indicators for budgetary program, bringing together the different federal public resources transferred to local governments in expenditure-makers participating in each program. General Area 23, Wage and Economic Provisions, will be • Use: linking the rules of operation for the budget- defined for each of the funds focused on regional develop- ary program with the MRI. ment. The strategy for implementing the PES at the local 72 S ession 3 : I nstitutional A rrangements for M onitoring and E valuation S ystems in L atin A merica level requires each state (1) to reform its constitution and question, the PES increasingly is considered to be a vehi- local budgeting and accounting laws, (2) to make progress cle for change, as it exerts influence on parliamentary de- in implementing methodologies for results measuring us- bates on the efficiency and effectiveness of public spending. ing its own resources, (3) to strengthen the technical capac- Similarly, it has become important in relation to the welfare ities and independence of the supervision entities of local economy, the allocation of budgetary resources over time, congresses, (4) to evaluate the results from the use of fed- and the very structure of economic policy, in terms of its eral public resources received on the basis of strategic and capacity to implement economic policy to achieve the ob- management indicators, and (5) to improve accountability. jectives proposed. It has become important as an element for evaluating public policies and institutions. Because the A critical assessment of performance evaluation RBB-PES is relatively new, it is necessary to strengthen its in Mexico institutional design, broaden its coverage in terms of pro- To strengthen the PES, it is necessary to overcome a se- grams and projects, and expand the context in which it is ries of challenges, all of which merit examination. Without applied to include other branches and levels of government. FIGURE 14: Budget Programming Structure BUDGETARY PROGRAM STRUCTURE PLANNING PROGRAMMING Link between programming and planning to achieve objectives, PROGRAMMATIC STRUCTURE policies, strategies, priorities and targets based on performance indicators included in the PND and its PROGRAMMATIC CATEGORIES PROGRAMMATIC ELEMENTS programs Functional Classification Programmatic elements are integrated through the Results-based National Development Plan 2007–12 Functional Group Indicator Matrix (MIR), which permits the identification, Public Policy Axes Functions measurement and evaluation of social and economic impact of Major Theme Sub-functions the allocation, implementation and quality of public expenditure. Objectives Programmatic Classification Mission (at office or entity level) Strategies Institutional Activity Planning-Programming link Budgetary Program (Pp) Lines of Action Program Priorities Results-Based Indicator Matrix Strategies Objectives Pp modalities (Programmatic point) Objectives Target Indicators Means of Suppositions Sector Programs S Subject to operating regulations Verification National Infrastructure U Subsidies Aim Performance Program E Public Services Purpose Indicators Basis B Public Property Component for functioning of P Planning and Public Policy Activity Performance F Promotion and Development Evaluation System G Regulation and Supervision Programmatic Key A. Armed Forces Unit responsible (Responsible for MIR follow-up and evaluation) R Specific Digits Category M Support Activities 1 Functional Group O Internal Support Organism 2 Functions L Jurisdictional responsibility 3 Sub-functions N Natural disasters Emphasis 4 Institutional Activity Type of Indicators Strategic/ J Pension and Retirement Management Public Goods/Services 1. Modality Identifier T Tax Contribution 2. Program Number Dimensions of Indicators Y Compensation Funds Citizen Satisfaction 5 Federative entity Efficiency, Efficacy, Z Pension Investment and Quality, Economic Restructuring fund All planning and programming processes are self implemented Investment Portfolio through the integral programming Federalized Account and budget systems (PIPP) Investment Projects(K) Social or Economic Indicators Targets Geographical Classification Cost–benefit Federative only Unit responsible for project budgeting Source: Author. 73 M&E SYSTEMS AND M&E TOOLS Various measures elevated the PES to the level of the tion of results obtained with this new focus. To that end, 2010 FEB Project. This guaranteed the PES’s sustainabili- it is vital to establish the necessary incentives to promote ty. The government offices and entities responsible for pro- the fulfillment of strategic objectives in programs, policies, grams must carry out a process of continuous improvement and institutions with the objective of innovating and mod- and must maintain the MRIs updated. To accomplish this, ernizing how organizations work, and the budgeting and it is necessary to consider the advances and results obtained spending processes. Ultimately, the information obtained from monitoring whether program goals are met, whether from monitoring the program indicator goals and the eval- evaluations are conducted in line with the AEP and wheth- uation’s results should be included in budgetary decisions er the criteria defined by the Secretariat of Finance and and in program management. Public Credit and CONEVAL are met. In addition, the The RBB-PES has achieved considerable success, par- MRIs should consider, in the programs in which it is nec- ticularly in modifying the normative framework and in essary, the transversal focuses of equality in relation to gen- generating and disseminating information on public man- der, youth, disability, and ethnicity. agement performance. Nevertheless, this success has been Improvements in the MRIs should align their objectives achieved due to the obligatory nature of the legal frame- with the strategic objectives of government offices and en- work. Institutions and functionaries have not yet taken tities, and with the strategic objectives of the NDP and the ownership of the entire system. It will be possible for the programs derived from that plan. The rules of operation PES to make significant steps forward if a positive scheme should be consistent with the information in the MRIs, in of incentives is incorporated for encouraging the use of per- response to their improvement and updating. Government formance information in decision making, and if the rating offices and entities must develop a work plan for following of public servants can be incorporated into program perfor- up on the primary results from the evaluations conducted, mance. Institutional incentives also should be established, and must include aspects that could be improved in the de- to allow public entities to benefit from the programs they signing of public policies and the corresponding programs. carry out, through the savings obtained from applying effi- Commitments will be formalized through specific instru- ciency and cost-effectiveness measures. ments. The advances and results achieved through the PES At the regional level, it is necessary to move forward will be reported and published in the terms of applicable with parallel actions that strengthen financial systems. Ac- provisions so that the information obtained from the mon- tions will align accounting systems in states and municipal- itoring of commitments on improvements, corresponding ities with those in the federal government. On December to the fiscal years 2007 to 2010, will be taken into account 31, 2010, at the latest, the federal government and states as part of a gradual, progressive process during 2010 and must have developed indicators for measuring physical-fi- for the subsequent budgetary processes. nancial advances associated with federal resources and with PESs are in a constant state of change. Challenges arise different points in time during the accounting process. It as their application is extended throughout public insti- is estimated that it will be possible to have implemented a tutions, causing changes in the behaviors of institutional uniform accounting system by 2011, and that by 2012 all agents, and generating acceptance, indifference, or rejec- records will be maintained under the new system. tion. The main challenge lies in finding a way to motivate The PES must be appropriate for its users. It is neces- behavior favorable to the PES. A change in traditions to- sary to encourage the main actors in the budgetary process ward conducting evaluations requires not only the training to use performance information, particularly what is gen- and dissemination that have been carried out on an ongo- erated in the framework of the RBB-PES, and to improve ing basis, but also the generation of a scheme of incentives the clarity and transparency of the results presented to so- that rewards the institutions that operate programs with ciety. The challenge is to convince users of the positive as- better results and allows them greater degrees of freedom. pects of using the system, and above all, in instilling the no- To enhance the probability of success and permanence, tion that a budgetary program is more valuable when the it is vitally important for the RBB-PES to be accompanied genetic code characterizing it is fully identified. Lastly, the by an adequate system of incentives and by the dissemina- RBB-PES is a fundamental part of the effort to build and 74 S ession 3 : I nstitutional A rrangements for M onitoring and E valuation S ystems in L atin A merica establish a budgetary tradition among the population, so forth as an objective the evaluation of higher middle-level they can be aware of the use, destination, and performance education policies. The objective of the Planning and Eval- of public resources in a clear, simple, and direct manner, uation of Educational Policy Unit (UPEPE) is “to develop and so they can participate in the way that such resourc- valid and reliable evaluations for supplying pertinent and es are allocated. timely information to federal and state authorities.” How- ever, the evaluations carried out by the UPEPE are focused AgustínCaso on basic education (primary and secondary); thus, they do Deputy Director General, Performance Evaluation System, not evaluate education policies at the higher middle level. Ministry of Finance and Public Credit, Mexico On the other hand, the National Institute of Evaluation of Education(INEE is in charge of evaluating the education system as a whole, without specifically evaluating HME The Creation of an Evaluation System of Higher public policies). As will be seen later, the UPEPE and the Middle Education in Mexico INEE are a basic part of the Higher Middle Education Eval- uation System (HMEES). Additionally, the National Eval- Introduction uation Council of Social Development Policy (CONEVAL) Information and knowledge are prerequisites for well- has as its objective to establish the rules for and coordinate founded decision making, which in turn are necessary for the evaluation of social development policies and programs. continuous improvement or progress. In the public are- The creation of an HMEES (Sistema de Evaluación de la na, information and analysis aid us in obtaining diagno- Educación Media Superior, SEEMS) comprises the first ef- ses that identify the main problems to be solved. Identifi- fort in this direction. SEEMS was created to comply with cation of the problem and its probable causes provides us two general objectives: with the necessary elements to design public policy pro- grams or interventions. When time and the nature of the • Improve the quality of EMS by means of decision problem permit, implementation of the program (or inter- making based on diagnosis, design, and measure- vention) should be preceded by a pilot program to discern ment its impact. Finally, once the public policy is implemented, • Inform society, through accountability, of the status it should be subjected to continuous monitoring and peri- maintained by the EMS and the manner in which odic measurement of the impact that the intervention had. EMS policies, programs, and interventions exert an The three components necessary for the design and im- impact on the quality of this education level plementation of effective public policy (diagnosis, interven- tion design, and impact measurement) can be seen as part of The creation of SEEMS arises within the framework an integrated evaluation program. Without one of these ele- of the Integrated Reform of Higher Middle Education ments, the public policy decision runs the risk of not having (IRHME). IRHME sets forth an exhaustive review of the the expected impact. Interventions without impact not on- EMS system, including education content, the installment ly fail in their attempt to resolve the problem for which they of a mechanism to assign and evaluate the education cen- were designed, but also waste limited public resources. Pub- ters’ directors, and the design of an instrument to measure lic policies, whose design and implementation are based on academic achievement, among others. Therefore, the entry integrated evaluation, resolve a specific problem efficient- of IRHME is a unique opportunity to carry out a complete ly and reveal the effectiveness of the use of public resources, diagnosis of EMS, to evaluate the impact of policies set in thereby increasing support for the intervention. motion as part of the reform, and to launch pilot programs Today, Mexico does not have in place the institutional- for designing and evaluating future interventions. ized mechanisms or processes to evaluate public policies ori- ented toward the improvement of higher middle education Evaluation within the IRHME framework (Educación Media Superior, EMS). Two public institutions EMS in Mexico has been characterized by a large number evaluate public education; however, neither of these sets of subsystems with different study plans, quality standards, 75 M&E SYSTEMS AND M&E TOOLS entrance and exit criteria, objectives, and goals. Consequent- and the transparency of the academic management of the ly, there have been high dropout rates, poor results in cogni- center’s director, among others. IRHME conceptual plan- tive achievement tests, and a weak link with the labor market. ning and practical implementation rests on four basic pil- The IRHME was put into action in 2007.25 The objec- lars (see Figure 15). tive of IRHME is to endow identity and order on a segment of the education system in Mexico that has been character- Education content ized by the absence of common policies in all subsystems This pillar includes definition and construction of a com- and a lack of regulation in the supply. The manner by which petency-based common curricular framework. The com- IRHME seeks to achieve this ambitious objective is through mon curricular framework allows for free movement of the creation of a National Baccalaureate System (NBS), students among different subsystems without the need to which defines IRHME entrance and exit criteria, standard- begin studies from zero, as occurred until recently. Addi- ized quality standards, and study plans that are common to tionally, it includes the redefinition of a memorization to a all subsystems without sacrificing their diversity. competency-based teaching paradigm model. To pave the way for the creation of the NBS, federal, state, and other authorities defined the profile of the higher Diverse supply options middle school graduate. With this in mind, they identified This pillar includesdefinition and recognition of the differ- the minimal academic resources to guarantee that all NBS ent supply options of EMS. students, no matter the state, subsystem, or type of school- ing, graduate with the necessary tools to face the challenges Operating mechanisms of the twenty-first century (the exit profile). The academic This pillar includes instrumentation of programs and inter- resources necessary include the education content, the ed- ventions for the construction of the NBS. These programs ucation center’s physical plant, the profile of the teachers, or interventions represent the operative part of IRHME, FIGURE 15: National Baccalaureate System National Baccalaureate System (NBS) I. Educational Contents II. Diverse Supply Options III. Support Mechanisms Three Competency Official Recognition of Validity of 1. Teacher profile Types Studies (ORVS) 2. Construye-T (Build Yourself) program a) Generic 1. Presential 3. Infrastructure b) Disciplinary 2. Intensive 4. Support for the demand Common Basic 3. Virtual (scholarships) Curricular Extended 4. Self-planned 5. Public competition Framework c) Professional 5. Mixed 6. Academic management Exit (graduate) 6. Certification by partial 7. Disability program profile examinations 8. HME Evaluation System 7. Certification by examination IV. NBS Entrance Process and Certification 1. NBS Board of Directors 2. Entrance/Exit Rules 3. Accreditation and Evaluation 4. State Governments/Higher Education Institutions (HEI) Agreements Source: Author. 76 S ession 3 : I nstitutional A rrangements for M onitoring and E valuation S ystems in L atin A merica that is, the pathway to achieve the NBS objectives. The Programs within IHRME programs or interventions included, among others, the fol- As previously mentioned, the operation of IHRME is ac- lowing: the Teacher Training Program with the adequate complished through programs or interventions that seek to profile, the Academic Infrastructure Program, the Director endow the centers belonging to the NBS with the academ- Training Program, the EMS Scholarship Program (support ic resources necessary to guarantee the graduate’s exit pro- for demand), the Academic Management System, and the file. The programs or interventions can be divided into the creation of the EMS Evaluation System. following two broad headings: (1) programs that are in pro- cess, and (2) pilot programs. NBS entrance and NBS certification This pillar establishes clear rules of NBS entrance and cer- Programs and interventions in process tification, and national certification for all NBS gradu- • Infrastructure Programs: drive the growth of the ates. The 2009–10 academic year marked the beginning educational supply in EMS; endow and strengthen of the NBS, following two years of work on the definition the existing infrastructure. of the common curricular framework, NBS entrance and • Teacher Training Program: develop the teaching exit rules, and implementation of the necessary programs competencies that are required for achieving the and interventions. In 2009, all centers demonstrating that NBS exit (graduate) profile. they had participated in the programs and interventions re- • EMS Scholarship Program: broaden education op- quired to guarantee the graduate profile could request their portunities between students residing in a house- entrance into the NBS. As agreed to by all the education hold that falls under the asset poverty line. authorities, entrance into the NBS will be determined by • Program of Attention to Persons with a Disabili- independent evaluator organisms based on criteria prees- ty: ensure the equality of educational opportunities tablished by the board of directors. among persons with special needs. • Construye-T (Build Yourself ) Program: prevent risk The Higher Middle Education Evaluation System behaviors among EMS youth and promote the SEEMS plays a fundamental role in IHRME in evaluating construction of a life project. all policies and interventions carried out within the frame- • The Impúlsate con Inglés (Drive Yourself with Eng- work of reform (“operating mechanisms”). Additionally, the lish) and Computer Program: promote the use of evaluation system designs and evaluates pilot programs for new technologies and the tools necessary in a glo- future interventions that contribute to IHRME objectives. balized world. SEEMS includes four evaluation spheres according to • School Management System (SMS): foster trans- their aggregation level. The four levels are (1) system and parency and accountability among EMS center di- subsystems, (2) school or center, (3) teachers, and (4) stu- rectors. dents. Each of these areas of influence plays a basic role in • Public Competition for Assigning Center Direc- the learning process; thus, it is necessary to possess a di- tors: develop a merit-based employment-assigna- agnosis and a referent for each. The diagnosis of each ar- tion system and thereby foster transparency in the ea together with the referent (or point of comparison) per- use of resources. mits identification of the sphere in which EMS presents the greatest lag, that is, where we require redoubling of effort. Pilot-phase programs For example, in the case of the NBS student area, the ref- • Prepárate (Prepare Yourself ): broadens EMS cover- erent (the “should be”) is defined by the exit profile of the age by means of distance learning (online). higher middle graduate. Having a comparison between di- • Incentives-Alignment (IAL) Program to Strengthen agnosis and referent for each ambit also permits us to evalu- Academic Achievement: identify a better incentives ate the efficacy of the policies and interventions defined by scheme to improve academic performance. IHRME and based on their capacity to focus resources on • Program on Perceptions and School Dropout: areas with the greatest lag. test the effectiveness of reducing school dropout 77 M&E SYSTEMS AND M&E TOOLS { in an intervention whose sole component in- 1  the individual “i” decides to leave the cludes information on the benefits associated Dit = EMs in time “t” with schooling. 0  the individual “i” decides not to leave the EMs in time “t” Advances in SEEMS consolidation In 2009, the construction of SEEMS began with the We can define the probability of dropout, Pr(D), as a participation of the National Institute of Evaluation of linear function of a variable, TBi, which indicates that in- Education(NIEE), the Planning and Evaluation of Ed- dividual i has an HME scholarship, and a series of controls ucational Policies Unit of the Secretariat of Public Ed- X. Therefore: ucation (Secretaría de Educación Pública, SEP), the Na- tional Center for the Evaluation of Higher Education Pr(Di) = α + βX + δ1TBi + εi (1) (CENEVAL), CONEVAL, the Latin American Faculty of Social Sciences (FLACSO), and the Under Secretari- Where β and X are a vector and a parameter matrix, and at of Higher Middle Education of the SEP (Subsecretaría independent variables, respectively; δ1 is the parameter de Educación Media Superior, SEMS). SEEMS members that measures the effect of the HME scholarship on the designated a group of experts to coordinate the tasks of probability of dropout. That is, δ1 is our parameter of in- diagnosis and evaluation within each SEEMS area. In terest. And last, εi is a randomized term with a mean of parallel, some of the programs or interventions are in zero and a distribution that asymptotically approximates the process of evaluation, in collaboration with univer- normal. sity academics of international prestige, and even with The validity of model (1) for obtaining the impact of the multilateral organisms. scholarships on dropout depends on the manner in which these have been assigned among treatment groups—those Evaluation of HME scholarship programs who received a scholarship—and the control group—those Evaluation of the scholarship program is performed by who did not receive a scholarship. If the treatment (EMS Orazio Attanasio and Costas Meghir of the University scholarships) was assigned randomly, differences between College London, who work in evaluation issues that uti- the two groups (treatment and control) can be attributed to lize an experimental design. The HME Scholarship Pro- the scholarship program; thus, we will know its impact on gram (HMESP) consists of a monthly monetary transfer dropout, absorption, and academic achievement. to all EMS students who comply with all of the require- A questionnaire with all the necessary information to es- ments established by the program’s rules of operation as timate the model (1) was designed to evaluate the program. follows: In 2010, the National Public Health Institute (NPHI) sur- vey will be applied to 3,500 households of youths belong- • Mexican citizenship ing to the control group and the identical number of house- • Enrolled in a participating public HME institution holds in the treatment group. Final results will be available • Under 21 years of age at the end of 2010. • Live in a household that falls under the asset-based poverty line Evaluation of the Construye-T (Build Yourself) • Request government aid by filling out an applica- program tion form26 The Construye-T program is made up of the following six special dimensions for the development, protection, and The scholarship program’s objectives are to increase ab- construction of life projects of youths in EMS: sorption (the transition from secondary school to the bac- calaureate level), reduce school dropout, and increase levels • Knowing one’s self: to develop the creative capac- of academic achievement. In statistical terms, Di,t is defined ity, as well as the expression of one’s inner world, in the following manner: to find answers when confronted by obstacles and 78 S ession 3 : I nstitutional A rrangements for M onitoring and E valuation S ystems in L atin A merica risks, to overcome fear in the face of new paths, and is, the difference between the differences. If ETf,t and ETnf,t to recognize affections and emotions. are defined as the terminal efficiency between federal and • School and family: to improve the relationships nonfederal centers in time t, respectively, then the impact among male and female students and their families, of the program can be measured by means of the follow- and the teachers. This consists of two subdimen- ing equation: sions: school environment, and family. • Healthy life: to strengthen the resources and indi- Impact = [(ETf,t+1—ETf,t)—(ETnf,t—ETnf,t)] vidual and social elements of protectionfor promot- | (Xf,t+1, Xf,t, Xnf,t+1, Xnf,t) (2) ing health and obtaining a better quality of life. It embodies three subdimensions: consumption and where X is a vector of observable characteristics at the sub- health, consumption of addictive substances, and system level. The same model can be estimated for the case sexuality. of risk behaviors, with the difference that impact would be • Youth participation: to educate male and female rendered at the student level, and not at the center level, as students as citizens and to link their performance in the case of terminal efficiency. in school and in the community, for the benefit of the environment. The Prepárate (Prepare Yourself) pilot program • Nonviolence: to offer elements for the develop- Online education in Mexico has been directly mainly to- ment of a culture that promotes inclusion, solidar- ward the adult population and in many cases, it is already ity, and cooperation for finding peaceful solutions inserted in the employment area. The Prepárate pilot pro- to conflicts. gram attempts to determine whether providing EMS on- • Life project: to foster the development of compe- line to persons under 17 years of age can be an alternative tencies in male and female students for them to de- to face-to-face education, that is, whether online education fine and conduct their future, in such a way that achieves at least the same learning, failures, and dropout they become the owners of their lives and they dis- levels. UNESCO is in charge of evaluation of this pilot pro- cover their values, capacities, potentialities, and the gram, whose four basic objectives are as follows: sense of their existence. • Given that distinct academic institutions with dif- Evaluation of the Construye-T program is carried out ferent programs of study participate in the pro- by the United Nations Educational, Scientific and Cultural gram, attempts have been made to systematize Organization (UNESCO) in collaboration with the Inter- these academic variants of program applications, national Institute of Education Planning in Buenos Aires. while citing the common and different compo- Variables of interest include terminal efficiency (the per- nents and determining their importance. In this re- centage of first-year EMS students who finish on schedule) spect, the strengths and weaknesses of each variant and the risk factors to which youths pursuing EMS are ex- will be identified, highlighting the lessons learned posed. by the institutions involved and analyzing the pro- To carry out the evaluation, in 2007, the year in which gram management design. the federal-subsystem centers program began operation, • To conduct a qualitative evaluation by means of the SEMS and the NPHI implemented the first survey workshops or focus groups in terms of student per- “Exclusion, Intolerance, and Violence in the Higher Mid- manence and failure dle Education.” At the end of 2009, a second survey will be • To carry out an evaluation on learning achievements conducted, with the objective of analyzing whether there through employment of the standardized Higher progress has been made in federal centers compared with Middle National Evaluation of Academic Achieve- other subsystems. Evaluation equipment will analyze the ment in Academic Centers (ENLACE) test in which difference in terminal efficacy and risk behaviors through- the face-to-face and online modalities of the insti- out time and between federal subsystems and others, that tutions will be compared. To render these groups 79 M&E SYSTEMS AND M&E TOOLS comparable, the students selected in both groups Although the program seeks to increase student learn- will have similar socioeconomic characteristics. ing, it will analyze the impact on some education indi- • Finally, the development of abilities, practices, and cators of intermediate-results, such as the dropout rate, study habits in relation to information technologies transition rates among school grades, terminal efficiency, and communication will be evaluated. This evalua- university attendance rates, and cognitive achievement in tion will be performed through focus groups. mathematics. Incentives-alignment program Perceptions and school dropout program The objective of the IAL program is to identify the best From the viewpoint of economic rationality, the decision incentives scheme (monetary) to improve student achieve- to drop out of school is determined by (1) the salary that ment in mathematics. This pilot program is the result of the student can obtain in the labor market (without hav- collaboration among the Inter-American Development ing finished baccalaureate studies), (2) the student’s percep- Bank (IDB), the University of Pennsylvania,27 and SHME. tion of the returns (or economic earnings) associated with The IAL program operates in 88 federal technical-bacca- education, and (3) the relative importance that the student laureate higher middle education centers, in which nearly assigns to present versus future remuneration. Thus, an in- 50,000 students form part of this academic achievement– crease in remuneration in workers without finishing bac- conditioned transfer program. The 88 centers forming part calaureate studies will tend to increase school dropout in of this program belong to the following four groups: higher middle levels; likewise, an increase in the perception of higher middle students on the returns of education will • T1: students receive monetary transfers based on tend to reduce dropout. their achievement in a standardized mathematics In the absence of restrictions in credit markets, that is, test. assuming that students have the opportunity to finance • T2: teachers receive monetary transfers based on their studies, higher middle students will drop out when their students’ achievement in a standardized math- the labor-market salary is higher than the perception of ematics test. the future income that they could acquire if they finished • T3: students, teachers, and directors receive mone- their baccalaureate studies. Therefore, part of the drop- tary transfers in terms of their individual and group out problem could be associated with an erroneous per- achievement in a standardized mathematics test ception, that is, with imperfect information. On the other (incentives-alignment). hand, within a context of imperfect credit markets, school • C: No monetary transfer is received. dropout can obey economic restrictions, which push the student into opting for present versus future consump- The centers were assigned to each of four groups ran- tion (for example, if students reside in households below domly, which guarantees that the differences in mathemat- the poverty line, their capacities and preferences for in- ical achievement between the control group and each of vesting in their futures will be limited). This economic re- the treatment groups can be attributed to the incentives striction can be eased by providing scholarships for pover- scheme. In addition, a pre- and post-course questionnaire ty-level students. is applied every year to control for changes in observables The task of this project is to identify the role played that could be treatment related (incentives). by erroneous perception (or imperfect information) in the The mathematics test with which the achievement of decision to drop out of school. We know that, in the case the students is evaluated was designed especially for this of poverty, economic restrictions can be more important program by CENEVAL. Students, teachers, and directors than lack of information; thus, we differentiate between receive the monetary transfer in a sole annual exhibition, in the impact exerted by erroneous perception on situations terms of achievement in the standardized test. The results with and without budgetary restrictions (students with and of this important pilot program will be announced at the without scholarships to study at the higher middle level). end of the 2010–11 school year. The methodology is divided into two stages: 80 S ession 3 : I nstitutional A rrangements for M onitoring and E valuation S ystems in L atin A merica • Learn the perception of students concerning the re- (TM2) on the probability of dropping out or not entering turns of education and the nonmonetary benefits higher education. That is, these are our parameters of inter- associated with this, and comparing these percep- est. Last, ηi is a random term with a mean of zero and a dis- tions with the returns observed. tribution that asymptotically approximates normal. • Carry out a later experiment, in which a group of Model (1) in combination with evaluation of higher students is provided with information on returns— middle scholarships will permit the evaluation of wheth- monetary as well as nonmonetary—in education er the presence of the two treatments exerts effects that are (measurement with the official information from above those of the impacts that would be obtained in iso- the “National Survey of Occupation and Employ- lated fashion. In other words, the impact of dropping out ment” in Mexico). on students who receive the scholarship and in addition receive information on the returns of education is greater In the case of EMS, in which the dropout problem is than the sum of the impacts of the treatments: concentrated on the first year of studies, the experiment will be carried out with first-entrance students. From Ho: γ2>γ1 + δ3 among the universe of first-time students, two treatment groups and one control group will be created: where δ1 is the isolated impact of the higher middle school scholarships on the probability of dropping out (see equa- • TM1: receives information on the monetary and tion (1)). nonmonetary benefits related to EMS. The program will follow up students in treatment and • TM2: receives information on the monetary and control groups within one year of their receiving informa- nonmonetary benefits related to EMS and a SHME tion to determine the impact that information on the re- scholarship. turns of education had on school dropout. In addition, • CM: does not receive information on the returns of SHME will apply a standardized test to students who education or on the SHME scholarship, but is giv- will participate in a pilot test to analyze whether informa- en a survey identical to that administered to TM1 tion exerts an impact on academic achievement (possibly and TM2. through the effort involved). In statistical terms, Di,t is defined as follows: The challenges and the future agenda { During the initial stage of SEEMS, the challenge was that 1  the individual “i” decides to leave the of integrating in a single conceptual document the ideas Dit = EMs in time “t” of a large and heterogeneous workgroup made up of di- 0  the individual “i” decides not to leave the verse governmental and nongovernmental institutions. On EMs in time “t” the other hand, the benefit of having a working group of a number of academic and evaluation institutions is that the We can define the probability of dropping out, Pr(D), SEEMS agenda can complement other work plans under as a linear function of treatments TM1 and TM2, and a se- the heading of monitoring and evaluation. The remaining ries of controls (X). Therefore: advantage of having a heterogeneous working group is that SEEMS can be nourished by the experience and informa- Pr(Di) = α + βX + γ1TM1i + γ2TM2i + ηi (3) tion from other systems or institutions with greater expe- rience. Additionally, possessing a workgroup composed of where β and X are vector and matrix of parameters and in- institutions within and outside of the federal government dependent variables, respectively; γ1 and γ2 are parameters lends credibility to the results. that measure the effect of information on the monetary and Another important challenge is that of finding an insti- nonmonetary returns of education (TM1) and the effect tutional mechanism by which SEEMS results (and those of of information in addition to a higher middle scholarship other evaluation systems) are taken into consideration in 81 M&E SYSTEMS AND M&E TOOLS decision making. The fact that the under secretariat is the path, the “budget model,” essentially seeks to use M&E to main promoter and a fundamental part of the SEEMS has improve decisions for budgetary allocations and has there- increased the interaction between evaluation results and de- fore been promoted by treasury ministries. The systems cision making. The true challenge is to create an institu- originally implemented, in Chile and Mexico, for exam- tional mechanism that guarantees that the evaluations and ple, illustrate this model, which comes from the hegemon- decision-making binomial is preserved in future adminis- ic current of thought on the modernization of public man- trations. agement, originating in the United States and Great Britain Perhaps we should begin creating suitable conditions to in the late 1970s. In the case of Mexico, in fact, a Program promote this link. At the present time, institutional design for the Modernization of Federal Public Administration in the exercise of federal government expenditure in Mex- (Programa de Modernización de la Administración Pública, ico—and in many other parts of the world—leaves little PROMAP) was implemented in 1995 (PROMAP 1995– space for rigorous evaluation-based decision making. 2000), and one of its main focuses was to initiate actions to The work agenda that we have outlined is ambitious and evaluate its results and pave the way for linking these results transcends the present administration, and even the Inte- to the resources allocated. grated Higher Middle Education Reform. For the first time The second path, the “planning model,” in contrast, in the history of Mexico, we will have EMS with identity, can be found in those countries that had decided to adopt clear, and common goals for all of the subsystems, and even M&E to improve political decisions at high levels of gov- its own system of evaluation in which cutting-edge interac- ernment and improve accountability. In this sense, it is no tions and decision making arise. We expect that the EMS accident that the institutional designs guided by this type results we will obtain during this administration will be the of motivation are focused on articulation with national de- main promoter of the culture of evaluation in Mexico. velopment plans and are inspired by ministries of planning. Colombia’s National System for Performance Management Rafael de Hoyos and Evaluation (SINERGIA) and Costa Rica’s National Advisor Coordinator, Under-Secretariat of Higher Middle Evaluation System (NES), in their origins, illustrate this Education, Secretariat of Public Education (SEP), Mexico type of model. Colombia, in fact, was the first country in Latin America to propose a design for a system to evalu- ate the management and results of public administration. Latin American Monitoring and Evaluation It accomplished this through the Constitution of 1991, in Systems and Their Governance which this task is assigned to the National Planning De- partment. In 1994 SINERGIA’s initial outline was dissem- Introduction inated, and between 1994 and 1999 it was validated by the Since their emergence in the early 1990s in Latin Ameri- World Bank and was promoted as a model to be followed. ca, institutional arrangements adopted to install the func- In Costa Rica, the National Development Plan (NDP)for tions of monitoring and evaluation (M&E) in public man- 1994–98 promoted a new organizational tradition with- agement results at the national level have been marked by in the public sector, with an orientation toward results and profound transformations. Currently, there is a tendency an emphasis on making public functionaries responsible. It toward convergence, which is likely associated with the in- proposed the creation of a system for evaluating manage- creasing consolidation of a tradition of results-based man- ment as one of its fundamental pillars. The experiences of agement. However, there are some remnants from two Great Britain and Colombia were used as the primary ref- types of different arrangements, which in their origins were erents for NES’s conceptualization. linked to two different focuses with respect to the functions Brazil is the only country in which the original institu- to be fulfilled by M&E.� tional design differed from the two models; it represented Along these lines, it is clear that the countries that de- a hybrid model that would later tend to prevail. Its expe- cided early on to institutionalize M&E in their public sec- rience in results-based management began in 1996 within tors did so by adopting one of two different paths. The first the framework of the 1996–99 Multiyear Plan although it 82 S ession 3 : I nstitutional A rrangements for M onitoring and E valuation S ystems in L atin A merica was defined by 42 strategic programs. In the Multiyear Plan priorities defined in NDPs, the Treasury Ministry and the for 2000–03, with the support of the World Bank and with National Comptroller General demand—as a requirement Canada as a model, the following five elements were estab- for the assessment and approval of the annual budget pro- lished as the focus for the public management model to be posal—a resolution for linking the AOPs to the NDP is- implemented: sued by the MIDEPLAN. Currently, some progress—at least expressed as a for- • Organization by programs mal connection between the MIDEPLAN and the Treasury • Strengthening of a strategic orientation Ministry—is evident in the “Technical and Methodological • Selectivity Guidelines for Strategic Sector-based and Institutional Pro- • Management gramming and Sector-based Monitoring and Evaluation,” • Evaluation developed jointly by the two entities in April 2009. A key impetus for the construction of shared leaderships An important aspect of the Brazilian case was the inte- has been offered by the creation of joint entities, sometimes gration, within the Multiyear Plan management method- having important decision-making powers. In Costa Rica, ology, of the teams from the areas of government planning an Inter-institutional Technical Commission has been in and government budgeting. Thus, it appears to be possi- place since 2002. It functions as an arena for coordinating ble to address one of the primary conflicts characterizing actions between these three entities. In this context, techni- M&E institutional designs, although it was not totally re- cal and methodological guidelines for developing AOPs are solved in this country. Current tendencies point to a search formulated and issued. for solutions along these lines, at least partly explaining the In Colombia, also in 2002, the Intersectoral Committee reasons why emerging convergences appear to surpass the for Results-Based Management and Evaluation (ICRME) original models and bring greater governance to M&E sys- was created as an arena for technical coordination and a tems. Some advances in this direction will be described and connection between evaluation and budgetary program- then pending challenges will be identified. ming. It is composed of the Presidency, the National Plan- ning Department, and the Treasury Ministry, in addition to Some advances in governance those invited by each. The boost given to ICRME is evident in the most recent document CONPES 3515 (2008) pub- Toward shared leaderships lished by the National Economic and Social Policy Council In countries in which M&E systems have been instituted (Consejo Nacional de Polítca Económica y Social, CONPES), under the jurisdiction of planning ministries, competition which strengthens and institutionalizes the functions and with treasury ministries has tended to limit the scope and role played by this committee. effects of the systems. Evidence, however, suggests the pos- A concrete expression of the ties between the Treasury sibility of building shared leaderships, although certainly Ministry and the Planning Ministry could be observed in not without some tensions. Colombia in 2007, when as part of the Framework for In the case of Costa Rica, the trigger for possible coordi- Medium-Term Spending process, 19 sector-based budget nation between the National Planning and Economic Pol- committees were created to establish the primary goals to icy Ministry (Ministerio de Planificación Nacional y Políti- be achieved in a medium-term budget context. ca Económica, MIDEPLAN) and the Treasury Ministry was the Law on Financial Administration and Public Budgets From isolated to more coordinated uses of that went into effect in 2001. This law stipulated that both information government entities, with prior consultation with the Na- In Colombia, coordination with other actors began to im- tional Comptroller General, must issue on an annual ba- prove after the Inter-sectoral Commission on Information sis the technical and methodological guidelines for formu- Policies and Management for Public Administration began lating annual operational plans (AOPs) for all institutions. to operate. The functions of this commission include elim- Since the law establishes that AOPs must be subject to the inating duplicity in requests for information and resolving 83 M&E SYSTEMS AND M&E TOOLS situations in which two or more public administration en- each program. Along these lines, one of the most impor- tities have influence or control over the same information. tant advances in this component is that efforts were begun Also having an impact was SINERGIA’s reorientation to- to define the programs to be evaluated on the basis of nego- ward M&E of exclusively macro-level results (sector-based tiations and coordination among the seven sectors of public policies and programs), as well as the creation of the pro- administration considered to be of high priority, since most gramming and management by objectives and results sys- of the national budget was allocated to them. For each sec- tem, which made it possible to simply and unify process- tor, the ICRME and the evaluation department (Evaluación es for obtaining information on management results from de Políticas Públicas, DEPP) team define the programs they various public entities. consider to be the most important for evaluating. Transparency laws, which have been increasingly pro- Brazil provides an example, in the strictest sense, of a re- moted in recent years, appear to be contributing to great- al “system” in relation to the M&E of the Multiyear Plan. er, more comprehensive use of information. In the case of It is composed of a Monitoring and Evaluation Commis- Mexico, for example, the creation of the Federal Institute sion (MEC) as a joint interministerial entity, and Moni- for Access to Public Information in the framework of the toring and Evaluation Units (MEUs) in each ministry. Be- Law for Access to Public Information facilitates access to ginning with the 2008–11 Mutiyear Plan, the MEUs have information on the performance of federal programs and been subordinated directly to the sector-based entity, sig- federal public administration offices and entities. nifying greater influence in the highest ministry offices. The MEC is the engine of the system, with representatives Creating entities for negotiation from the Civil Cabinet at the President’s Office, the Trea- Relationships with sectors, particularly with the highest sury Ministry, the Environment Ministry, and the Ministry ministry offices, have constituted one of the most critical of Planning, Budgeting, and Management, where the Exec- matters in a number of M&E systems, insofar as it some- utive Secretariat is located. times has led to a weakening of these ministry offices, and consequently, of the political leadership over sectors. Ac- From isolation to interconnection among cording to what was established at the Latin American M&E systems Center for Development (Centro Latinoamericano de Ad- One of the main problems reported by organizations is the ministración para el Desarrollo, CLAD) Congress in 2008, repeated requests for the same information from different the contrasts in the use of information on results between systems. In the end, they feel as if they are being invaded by developed countries and Latin America constitute one of requests. Their time is wasted and they are distracted from the aspects that limit results-based management in the re- more important objectives. What is worse, the legitimacy gion, particularly the institution of results-based budgets. of M&E is undermined. This situation is found particular- Industrial countries tend to develop a “bottom-up” process ly in countries in which various national systems have not based on the “responsibilization” of services, involving ne- been able to join together “at the base level.” gotiations and agreements with ministries and allowing the Some institutions have attempted to resolve this type of use of results as a primary source of information. However, problem. In Brazil, in addition to the Ministry of Planning, the process in Latin America tends to be “top-down” and Budgeting, and Management that coordinates the M&E based on mistrust, supervision, sanctions, and avoidance of system in the Multiyear Plan, at least three other institu- responsibilities at the base level. tions systematically conduct M&E. The President’s Office Concrete actions toward the creation of communication monitors a portfolio of goals associated with the Growth spaces that allow for negotiating M&E agendas as well as Acceleration Program (which is a set of high-priority ini- goals and indicators have begun to be carried out more vig- tiatives that overlap to a significant extent with PAP pro- orously in recent years. For example, in the case of Colom- grams, although are not limited to them); the Ministry of bia, efforts began at the end of 2008 to develop Multiyear Social Development and the Fight against Poverty moni- Evaluation Agendas, in which the idea is to avoid isolat- tors and evaluates social policies and programs; and the Na- ed evaluations subject only to the demands and interests of tional Accounts Tribunal carries out quick assessments of a 84 S ession 3 : I nstitutional A rrangements for M onitoring and E valuation S ystems in L atin A merica limited set of government programs, as part of its mandate New foundations for sustainability for monitoring performance. What should be emphasized Normative frameworks are not sufficient for sustaining here is that the purposes of the different systems comple- M&E systems, but there is no doubt that they can contrib- ment each other, and there are significant areas of over- ute to this goal. Significant steps are being taken in the re- lap. However this does not signify unnecessary repetition gion along these lines. In Costa Rica, for example, the re- or parallelism. In fact, both formal channels (particularly forms in Article 11 of the National Constitution (2000) the MEC, given its interministerial composition) and in- and the Law on Financial Administration and Public Bud- formal channels of communication are broadly used, ac- gets (2001) constitute a renewed platform for strengthen- cording to reports from sources. Furthermore, the moni- ing NES. Through these reforms, the evaluation of results toring of the Growth Acceleration Program is integrated at and accountability is introduced as the guiding principle of the operational level with the Multiyear Plan monitoring, public management. The new law seeks to coordinate the since it uses data from its information system to monitor powers granted to the MIDEPLAN, the Treasury Minis- and evaluate its own initiatives. In addition, the evaluation try, and the National Comptroller General in the area of of social policies and programs—since this involves exter- budgetary evaluation and institutional management eval- nal evaluations—provides inputs for the evaluation of re- uation. lated PAP programs, complementing and testing their self- In Colombia, through Article 132 of Law 1151 of 2007 evaluations. (through which the NDP is regulated), a step forward was In Mexico, M&E’s close connection to public poli- taken in recognizing the importance of evaluating the pub- cies and programs is noteworthy. The National Coun- lic sector. The Article, titled “Evaluation of Management cil for Evaluation of Social Development Policy (CONE- and Results,” indicates that VAL), the Treasury Ministry, and, to a much lesser degree, the Civil Service Secretariat coordinate the Performance according to what is stipulated in Article 29 of Evaluation System. One of the expressions of the joint ef- Law 152 of 1994, in relation to the design and or- forts among these three entities is the 2007 establishment ganization of systems for evaluating management of guidelines for evaluating federal programs, initially only and results, the Inter-sectoral Committee for Eval- applicable to federal programs subject to rules of operation, uating Management and Results is created as an yet later including other programs as well. entity for establishing priorities for program eval- uation, approving methodologies and their scope Greater focus on policies and accountability and considering results that may contribute to im- In various M&E systems, there is a noteworthy change in proving processes for formulating policies. The focus toward the policies and strategies. This can be clear- National Planning Department will design mech- ly observed in Colombia, Costa Rica, and Uruguay. In Co- anisms for efficient use of the corresponding in- lombia, the turning point for SINERGIA began in 2002, formation.29 when the Uribe administration reoriented the system to- ward a focus on monitoring at the program and policies- Challenges sector levels. Also noteworthy is the emphasis current- ly placed on the function of accountability to society. In Immediate challenges for the M&E systems 2008, various efforts began to address the topic of a Na- Some of the immediate challenges vary according to the tional Accountability Policy. This initiative was primarily system. For example, in the case of Costa Rica’s NES, the focused on the system serving as a source of information importance of consolidating the self-evaluation process for- for citizens, control entities and Congress, and not only as mally promoted since 2006 is recognized, specifically, in re- an information tool for decision making by the executive lation to the development and implementation of means branch (CONPES 3294, 2004). Nevertheless, these areas of verification that offer transparency and legitimacy to the have not yet been adequately developed, and it is important results reported. In Colombia, one of the challenges is to to continue to strengthen them. achieve the systems’ institutionalization in the executive 85 M&E SYSTEMS AND M&E TOOLS branch and the other sectors of public administration. In Latin American public administrations, and consequently, Brazil, the challenge is to introduce multiple evaluation the type of institutions characterizing the region. These re- mechanisms, with participation by other actors, primarily fer to the type of power and influence relations, and the val- users, for the purpose of diminishing the tendency toward ues that have been consolidated through the various state self-complacency generated by self-evaluation. reform processes. Other challenges are shared, and basically involve ex- panding the positive tendencies that are beginning to ap- Medium-term challenges: The new public arena pear, with the understanding that these are still unfinished and its values30 tasks. The recurrent challenges involve achieving greater And along these last lines, another set of challenges emerg- coordination between the processes of planning, budget- es. Without doubt, changes of uncommon magnitude have ing, implementation, M&E, and specifically, the need to been generated in recent decades, and not only are we con- develop new mechanisms and strategies for coordination fronting a new universe of services that are public but in with treasury ministries to achieve a greater link between private hands, but also we are finding that the logic of the M&E information and budget decisions, as well as to en- market has been established within the state. sure that the latter take political strategies more seriously In fact, most institutional incentives (new institutions) into consideration. The fact that there are still cases of strict that have been promoted through state reform and pub- compartmentalization at the level of units of analysis cer- lic management processes are oriented toward reproducing tainly makes this task more difficult. For example, in Co- the logic of the market within the public sector—and this lombia, the unit of analysis used by the Budget Bureau of is even acknowledged without hesitation by the promoters the Treasury Ministry and the one used by SINERGÍA of- themselves. But these incentives go even further, seeking to ten do not coincide. For some, the unit of analysis is a proj- change the state’s role in the public sector, and specifically, ect, and for others it is a public program or policy, and this to diminish state power in the sector. This is the real insti- means that objectives, baselines, and indicators are difficult tutional change being prescribed. to share between them. One of the most noteworthy effects of the introduction There is still a great deal of work to be done in terms of of market mechanisms in the public sector is the change working in coordination with ministries and government in its very nature. Recently, the Organisation for Econom- entities, to ensure that they use the information generat- ic Co-operation and Development (OECD) has even pro- ed by the systems. And there is a similar need in relation to posed a new concept, that of “public domain,” to define the improving the integration of different systems and their co- public sector that has been newly configured, with the as- ordination with society. The need to build trust points to sumption that it no longer only includes entities belonging the need to replace general policies that return powers by to or controlled by the government, but also includes ser- selective mechanisms with those in which flexibility is part vices financed (directly or indirectly) by the government, of what is recognized as good management. There is also a yet provided by private organizations. need to design mechanisms that permit the distribution of In the case of Latin America, the fragmented data avail- power among intersectoral entities. able suggest a tendency toward growth particularly in the Most of the challenges are not of a technical nature. social services provided by commercial entities. Conse- They refer more to matters of systems governance, in which quently, Latin America (with a few exceptions) has adopt- coordination, negotiation, and collaborative formulation ed the logic of the market for the provision of public servic- of policies among diverse actors—and here it is important es in more extreme terms than the countries in which this to identify them as political matters that affect the gover- type of focus initially emerged. In this regard, in the Unit- nance quality of M&E systems—become increasingly im- ed Kingdom, immediately after the period characterized portant. And of course, if we divert our attention to these, by the deepest transformations in this direction, it was ac- others of even greater magnitude come to the fore. One is knowledged that although the public sector might have lost clearly related to the political economy of M&E. Anoth- its economic role, it had maintained a strong social role. It er concerns the compartmentalization and segmentation of continued to finance and provide vital goods and servic- 86 S ession 3 : I nstitutional A rrangements for M onitoring and E valuation S ystems in L atin A merica es, including health, education, research and development, M&E systems are not free from potentially negative as- criminal justice, and social security. pects, in relation to the type of institutional arrangements Among the issues emerging since these new realities and type of values established in the public arena, and in became apparent is the lack of data for identifying the the end, in society. They do not exist in a vacuum, but rath- costs, inputs, and, especially, the results of the new ar- er are affected by both the features of the sociopolitical con- rangements, which are needed to adopt duly established text in which they are being developed, and the patterns of public policies in this regard. The premise, in all cases, reform in the public sector that frame their gestation and is that the private sector, whether commercial or non- development. commercial, has comparative advantages compared with A significant medium-term challenge is related to the the public sector—although the indicators for regarding type of state and public administrations under construc- whether or not to support this statement are not specified. tion. The old pattern of the state’s configuration was in- In fact each modality or instrument should be evaluated efficient, and promoted rentier societies and irresponsible for efficiency and cost, quality and effectiveness, and cov- bureaucracies. The new pattern, on the other hand, is lead- erage and fairness. ing to divided, individualist societies. Now that the state Another phenomenon that is largely a consequence of is once again recovering its importance, it is time to move the application of a market focus in the public sector is away from both of these focuses. For this to happen, to the over assessment of measurements, with biases that bor- begin to acknowledge the complex interdependencies be- der on formalism and with an emphasis that generates ad- tween values, institutions, and behavior norms, as well as verse effects on fairness. The assumption that what cannot between the search for fair distribution and for produc- be measured does not exist, and the tendency to manage tion efficiency, requires more extensive research than pro- “for” indicators instead of the opposite, constitutes some posed. Such research, which directly involves M&E sys- of its expressions. One effect of this is that the attention tems, should begin by acknowledging that we currently are taken away from the ends becomes caught in the means. In confronting a black box (public domain), in which a signif- fact, many measurements are currently being made, but in icant degree of the production of public goods and servic- general we do not have access to real evaluations of chang- es takes place, and by admitting that no goals or indicators es in pertinent government capacities, or their impacts on are totally free from outside interests, particularly if the un- society. derlying logic is one of the market, not of what is public. Nevertheless, what is most significant is the connection The governance of M&E systems is, without a doubt, an between incentives, private calculations of personal bene- immense task that requires influencing patterns of state re- fits, and measurements, which involve the logic of the mar- form and public administration. ket established in the state and its consequences for max- imizing values. One of the main findings suggested by Nuria Cunill Grau recent studies is that this connection can be translated into Special Advisor and Former Program Director, CLAD a lack of attention to fairness, as a maximizing value from the public sector. In addition, it has been found that goals can be converted into negative, strategy-structuring ele- Session 3 Notes 21 ments among actors, and can configure relations character- See the MIDEPLAN Web site, http://www.mideplan.go.cr/con- ized by competition rather than collaboration. tent/view/87/438/ (accessed on October 1, 2009). 22 See the Deinsa Delphos Web site, http://www.deinsa.com/del- All of this warns us of the type of tradition underlying phos/index.htm (accessed on October 2, 2009). the new institutional arrangements. It proves that it is not 23 See Baltimore City Web site, http://www.baltimorecity.gov/Gov- enough to focus on the positive result from recent transfor- ernment/AgenciesDepartments/CitiStat.aspx mations in the public sector, pointing to the establishment 24 See White House Web site, www.whitehouse.gov/omb/ of a tradition of goals and indicators, knowing that this blog/09/06/08/BuildingRigorousEvidencetoDrivePolicy 25 new tradition eventually may lead to negative effects on the All information on IHRME may be obtained at http://www.re- forma-iems.sems.gob.mx/ values of the public arena and its governance. 87 M&E SYSTEMS AND M&E TOOLS 26 28 For information on ProbHME, see http://www.sep.gob.mx/be- For further discussion of the M&E systems described in this casmediasuperior/index.html paper, see the SIARE-CLAD Web site, http://www.clad.org/si- 27 The University of Pennsylvania research team is composed are, particularly the section on the evaluation of results. 29 of Jere Berhman, Susan Parker (CIDE), Petra Todd, and Ken Article 29 of Law 152 of 1994. 30 Wolpin. See Cunill Grau (2009) for further development of the issues presented in this point. 88 Special Session: National System for Performance Management and Evaluation (Sinergia) National Management and Results Evaluation Fifteen years of learning System Achievements, Challenges, and The Colombian results-based model emerged with the Po- Perspectives litical Constitution of 1991, as part of an intense process of During the Fifth Meeting of the Monitoring and Evalua- modernization involving a series of economic and political tion Network of Latin America and the Caribbean: Chal- reforms. Regarding institutional issues, one of the chang- lenges and Opportunities for the Institutionalization of Re- es stipulates that the state must have suitable monitoring gional Systems, a special session was dedicated to Colombia’s and evaluation (M&E) tools and a system that combines National Management and Results System (NMRS) to com- the various institutional efforts of a reform-oriented gov- memorate the 15 years since its inception. During the ses- ernment. sion, the achievements, weaknesses, and challenges of the Three years after the new constitution, Law 152 of system were discussed and an analysis conducted of its advan- 1994 (Organic Law of the National Development Plan) es- tages, roles, and aspects that have influenced its development. tablished the guidelines for the follow-up of policies, stipu- The panel included Director of Public Policy Evalua- lated that it must be based on the priorities and guidelines tion (PPE) of the National Planning Department (NPD) of the National Development Plan (NDP), and made the Diego Dorado; Presidential Advisor Claudia Jiménez Jara- NDP responsible for the performance of these functions. millo; Budget Director of the Ministry of Finance and Pub- These constitutional and legal mandates led to the emer- lic Credit Fernando Jiménez; Director of the National As- gence of Columbia’s NMRS. This tool was designed to en- sociation of National Financial Associations (ANIF) Sergio hance the effectiveness of the formulation and implemen- Clavijo; and Deputy Director of the NPD Juan Mauricio tation of policies, programs, and public projects to provide Ramírez, as chair.� timely measurements of the management results of public This document includes the central points of these in- administrators, improve public investment management, terventions, complementing their analyses and addresses and provide appropriate information for assigning funds. with bibliographies by authors that have studied NMRS as At the same time, the system would focus on follow- a reference. The document is divided into four parts. It be- up and evaluation with particular emphasis on the achieve- gins with an examination of the experience of NMRS dur- ment of results, as reflected in the 1994 National Eco- ing these 15 years of learning and then explores NMRS’s nomic and Social Policy Council (CONPES), document links with other central government entities. It explores 2688/94, which stipulates the “introduction of an evalua- views on the system from outside government and ends tion culture within organizations in order to boost public with a description of the challenges and perspectives for investment management.”� NMRS for the next few years in a context of strategic re- Some participants suggested that the system can be di- alignment of the system. vided into two generations of evolution: 1995–2001 and 89 M&E SYSTEMS AND M&E TOOLS 2002–09. The so-called first generation (1995–2001) saw • There was no clear separation between the concepts the development of the basic conceptual, legal, and in- of monitoring and evaluation. strumental framework of the evaluation function. Results- • There was a lack of incentives to demand evalua- based management practices were incorporated into cen- tion on the part of the various institutions and re- tral administration and the continuous use of management sults were not disseminated, thereby undermining indicators was introduced, with certain planning by goals accountability. and results tools in central government bodies. These plans • There was a lack of knowledge of the actual effects included (1) the indicative plan (tools for following up the of investments. main NDP programs, designed for the strategic managerial • Knowledge for decision making was insufficient. sphere), (2) the action plan, (3) efficiency agreements (in- tended to improve coordination between the organizations This scenario, coupled with Álvaro Uribe’s assump- responsible for the various levels of expenditure results, be- tion of the presidency, raised the need to restructure and ginning with a clear distinction between their skills and the strengthen NMRS’s operating scheme. This point of inflec- definition of what their contribution should be to emerg- tion would enable the system to change its approach “from ing from a fiscal crisis), (4) management indicators, and (5) monitoring at the agency-sector level to the program and commitment and evaluation (special documents that in- policy-sector level. On the basis of this reform, the system cluded the decisions made on the sector goals between the made a clear distinction between the monitoring and evalu- ministers and the president on the basis of the priorities es- ation activities and stressed the results-based assignation of tablished in the NDP). public expenditure” (Villareal 2007). During this stage, the World Bank provided continuous This new approach has been reflected in various impor- support for NMRS, through the Modernization of Public tant landmarks, such as (1) the implementation of the Sys- Financial Administration (MPFA) project. This first gener- tem of Government Programming and Goal Management ation ensured the methodological solidity and conceptual (SGPGM) in 2002,(2) the Strategic Plan for 2006–10, and maturity of the system’s M&E instruments. It permitted the (3) the drafting of the 2004 National Council of Econom- creation of a space to encourage a results-based management ic and Social Policy (NCESP) 3294 document, “Renova- culture and to learn more to ensure its proper implementa- tion of Public Administration: Results-based Management tion. It also provided a better diagnosis of the institution- and the Reform of the National Evaluation System.” The al context, which subsequently made it possible to design a SGPGM permits real-time monitoring and provides citi- more efficient follow-up system to meet the country’s needs. zens with access to the state of management as a function The second generation (2002–08) began with a com- of the set goals and a base line, corresponding to the start of plex scenario for NMRS, determined mainly by the follow- the administration. It identifies those responsible for carry- ing aspects: ing out every policy and program, while linking results and • The commitments and tools of the government FIGURE 16: bodies were detached from the budgetary assign- Milestones of Reform, 1991–2009 ments for each sector. • The various links, tasks, and responsibilities of the system were not clearly defined. • Data management was poor. Too many indicators (940 in 2002) focused on measuring management rather than results, and, in turn, discouraged the achievement of specific goals associated with this management. Their processing placed a burden on government bodies that was not reflected in pro- 1991 1994 1995 2002 2004 2009 ductivity gains. Source: Author. 90 S pecial S ession : N ational S ystem for P erformance M anagement and E valuation ( S inergia ) TABLE 9: Changes Introduced through NCESP 3294 NCESP 2688, 1994 NCESP 3294, 2004 Objectives Introduce a culture of evaluation within the Help improve the effectiveness and impact of policies, departments to boost the managerial capacity programs, and institutions; improve the efficiency and of public administration. transparency of programming and the assignment of resources; and promote transparency in public adminis- tration, by actively incorporating civic control. Components The system consists of two modules. The first Monitoring or follow-up of results. includes the mechanisms designed for the Focused evaluations. self-evaluation of departments regarding the Dissemination of results for accountability. management and results of the implementation of policies, programs, and projects. The second includes mechanisms designed for evaluation by the corresponding ministries and the NPD, of the projects, programs, and policies periodically selected for this purpose. Source: Head Office of Public Policy Evaluation, NPD. Note: NCESP = National Council of Economic and Social Policy; NPD = National Planning Department. goals to the investment resources assigned in the General the most significant commitments for each department at Budget of the Nation. The Strategic Plan promotes the in- the start of each administration. The tool for the follow- stitutionalization of NMRS within public administration. up is the System for the Follow-Up of Government Goals Table 9 summarizes the changes introduced through the (originally known as SFGG and now known as SISMEG). NCESP 3294 document. This system records the indicators for the 20 departments For its part, NCESP 3294 is the policy guideline that of the executive branch that are responsible for achieving made it possible to unify concepts and methodologies. It is the goals that will contribute to achieving the NDP goals. the road map that currently governs the system and defines Within these departments, a “goal manager” is designated the scope and limits of the development of its functions. In to assume responsibility to record progress toward the es- general terms, the modification of the evaluation and fol- tablished goal. low-up function in Colombia sought to achieve three ob- Its presence online and its virtual consultation environ- jectives: ment enable the system to obtain up-to-date information on physical and budgetary performance in real time from • Improve the effectiveness and impact of the poli- those directly in charge. This can reduce the transaction cies, programs, and institutions costs inherent in the process, and make management and • Increase the efficiency and transparency of the pro- follow-up more transparent. It also enables decision makers gramming and assignment of resources • Encourage transparency in public management, by actively incorporating civic control FIGURE 17: Results Road Map The reform divided the system into three complemen- tary components or strategic lines, by establishing con- Dissemination & Focused Results follow-up ceptual and instrumental differences between them (see accountability evaluation Figure 17). It then explored each of these lines or compo- nents in greater depth. Continuous Results of public Exhaustive analysis verification of policy evaluation of evolution and Follow-up of results achievement of goals and follow-up made impact of principal & objectives available to the public policies Thegoal is to verify continuously the achievement of the NDP’s goals and objectives. This component establishes Source: Head of Office of Public Policy Evaluation. 91 M&E SYSTEMS AND M&E TOOLS to obtain timely information without having to resort to FIGURE 18: the usual, traditionally established routes. National Development Plan Goals Through this system, citizens, Congress and entities re- 100 92.8 sponsible for control are able to exercise social control and 87.6 85.8 80.6 78.2 78.2 undertake consultations related to the commitments made 80 with those responsible for policies and programs and the progress achieved in their respective indicators. The exces- 60 sive number of indicators that once made up the system has been reduced.� 40 After the system records information on the indicators’ progress, a monthly report is drawn up to be analyzed by 20 the President’s Ministry, which ranks achievement of the 0 indicators by sector and organization, and gives details of Special High, Environmental Defense Poverty A better Dimensions sustained management policy reduction government poor or satisfactory levels of achievement and performance. growth Every three months, virtual publications are disseminated Source: Head Office of Public Policy Evaluation, NPD, cutoff date: Octo- on issues linking goals and investment by specific sector. ber 31 2009. An annual report, known as the Results Sheet, is submit- ted to the president of congress on each completed con- ture of the activity and its potential to be replicated or re- tract. The state of progress of certain indicators is given in produced. Figure 18. Likewise, the follow-up of results has achieved All the evaluations carried out to date use contracting the following: systems provided through a competitive bidding process, in which consultants or consulting firms outside govern- • Two development plans have effectively been mon- ment develop the required evaluation and the Public Policy itored (2002–06 and 2006–10). Evaluation Office gives technical support and monitors the • Early warnings are issued on the plan’s progress evaluation. In addition to carrying out robust impact evalu- (short-term follow-up). ations using experimental and quasi-experimental models, • 624 indicators are used. NMRS has developed less costly evaluations depending on • Monthly reports are available (average level of up- the program being analyzed. Thus, management, process, dating, at least 92 percent). institutional, results-based, and executive evaluations com- • Over the past year, 285 follow-up meetings have plement the range of methodologies used by the system to been held. evaluate programs. Focused evaluations According to NCESP 3294, evaluation is defined as an “ex- FIGURE 19: haustive evaluation of the causality between a state inter- Annual Evaluations, 2006–10 vention and its effects in determining its relevance, efficien- cy, effectiveness, impact and sustainability.” In this respect, NMRS has gradually incorporated annual and four-year- 7 Equity 6 10 23 ly evaluation agendas defined by the Inter-SectorCom- 2 Competitiveness mittee of Evaluation and Results-Based Management (see Figure 19).36 The scheme of prioritization of evaluations 3 Public includes such criteria as (1) the amount of resources used 1 5 3 administration 1 2 by programs, (2) the characteristics of the target popula- 2 tion, (3) the number of persons affected, the importance of 2006 2007 2008 2009 2010 the program for a specific sector, and (4) the innovative na- Source: Head Office of Public Policy Evaluation, NPD. 92 S pecial S ession : N ational S ystem for P erformance M anagement and E valuation ( S inergia ) To date, the NMRS has achieved considerable progress Technical Secretariat and thereby create top-level technical in evaluation issues. This has been borne out by the increase and academic debates on the issue of evaluation, follow-up, in the number of evaluations carried out per year (two eval- and accountability regarding public management. uations on average in 2006, 12 evaluations in 2009, and 30 Moreover, the NPD Web site has incorporated an planned for 2010), the links established between the eval- NMRS link that includes documents, products, and ser- uation results and policy formulation and reformulation, vices as well as control tools and input for accountabili- the increase in investment in evaluations (since 2006, near- ty. NMRS has a virtual documentary center of nearly 500 ly US$15 million have been invested), the diversification of articles related to evaluation, follow-up, and accountabil- the agenda, which now includes competitiveness and pub- ity. This progress has been made available in (1) NCESP lic administration issues, and the fact that evaluations are documents on the progress of public policies and evalua- carried out by widely recognized consulting firms indepen- tion reports, (2) evaluations in the “Public Policy Evalua- dent from government. tion” series, (3) an annual management report by and for At present, as part of its process of strategic realignment, the president of congress, and (4) technical support to en- NMRS is attempting to incorporate a component called sure national government’s accountability to the commu- RADAR to compile and collect the public policy evalua- nity through televised Council of Ministers meetings. In tions carried out in the various public organizations, terri- short, the dissemination of results and the availability of torial departments, research centers, and universities in the information have enabled the government to direct man- country. This initiative will make it possible to improve and agement toward results. Likewise, thanks to the adminis- optimize the use of evaluation studies, encourage coordina- tration’s technical support and credibility, during these 15 tion and complementarity in this sphere, use decision mak- years of learning, NMRS has managed to build a strong ing to achieve more effective management, and produce in- M&E system in government.� formation for the various social actors. This is particularly noteworthy because much of the in- formation produced by the system is intended for those re- Dissemination for accountability sponsible for implementing the policies evaluated. In this The main purpose of this component is to promote inter- respect, it is essential for governments, as policy imple- action between the citizens, the control entities, the exec- menters, to have the instruments for measuring manage- utive branch, and the other branches of government. The ment results and to participate in this process (Zaltsman component will implement the principle of practice in 2006). public management. In other words, it means making the results of the evaluation and follow-up of public policies NMRS and intergovernmental coordination: A tool available to the public. for guiding political and budgetary decisions As far as dissemination is concerned, the aim is to pro- NMRS’s links with other government bodies was crucial vide appropriate dissemination for NMRS products, by to the institutionalization of M&E throughout the govern- improving the production of documents in terms of qual- ment. This link is explicitly generated, insofar as the system ity and availability and ensuring that the information pro- produces quality technical information that the govern- duced is timely, clear, and reliable for its users. Thus, infor- ment uses as input in political decision making. At the in- mation has been delivered on the evaluation results carried tergovernmental level, the use of the information produced out on various national government programs and the fol- by the system operates within the context of the president’s low-up of the NDP goals. exercise of political control over the government cabinets, The various tools for dissemination include sending docu- and the assignation of in investment spending as an input ments physically to organizations, academia, and civil society for budget planning in the Ministry of Finance. organizations, and by e-mail to more than 9,000 addressees. Likewise, forums and seminars are held, some on an in- Political control ternational scale, to disseminate the work carried out by NMRS enables the executive and, more specifically, the the Head Office of Public Policy Evaluation as the NMRS president, to verify the achievement of NDP goals, monitor 93 M&E SYSTEMS AND M&E TOOLS the results of government management and those respon- certain approaches have been established for follow-up. sible, improve state planning on the basis of priorities, and This relationship is established through coordination with generally make results visible to make management more the follow-up group in the System for the Follow-up of transparent. Figure 20 illustrates this coordination. Government Goals. In addition to the ministry’s obliga- ��For the President’s Office and the Council of Minis- tion as a government entity to provide sector indicators, it ters, the information produced by NMRS is particularly uses the information available in the system and links it to important insofar as it is the primary source used for man- annual goals with the assigned investment budget, as a first agement controls. This use contrasts with the use it could step toward the programmatic classification of expenditure. be given by ministries in other scenarios to support deci- Another approach is in the organization of a sector ex- sion making. penditure committee. This committee undertakes a medi- The case of “Families in Action” has shown that the um-term expenditure framework exercise, follows a strate- information produced can be used to carry out structur- gic projection process, and prioritizes expenditures. This al or rapid changes in the implementation of the opera- facilitates the achievement of fiscal goals that look beyond tion. The most obvious structural change occurred because the annual horizon and permit better results within the of the evaluation results, whereby the number of beneficia- time required to assign resources, and it ensures that the re- ries tripled and virtually all the municipalities in the coun- sources are used more efficiently.� Lastly, where evaluations try were included. This last change involved the program’s are concerned, the Ministry of Finance forms part of the implementation in cities and the introduction of the popu- Intersectoral Committee of Evaluation and Results-Based lation displaced by violence as the new beneficiaries of the Management. program. Links between the NMRSand the Finance Ministry have been weak because, among other things, the relation- Budgetary planning ship has not been institutionalized. Although the informa- The relationship with the Ministry of Finance and Pub- tion on follow-up and evaluation may exert a limited in- lic Credit has yet to be formally institutionalized, although fluence over the distribution of the national budget in the FIGURE 20: Goals Management System NPD Head Office of Public Policy Presidential Board Evaluation Operations Technical Political Coordinator Coordinator Technical DAFP Verifier Administrators of Good Government Ministers and Administrative Department Goal System (NMRS) Directors Sector Coordinators Persons responsible for Goal Managers quality of information & reports on achievement of goals Goal 1 Goal 2 Goal 3 Internal Control Offices External Controls: Citizens, Congress, General Comptroller Source: Head Office of Public Policy Evaluation, NPD. Note: NMRS = National Management and Results System; NPD = National Planning Department. 94 S pecial S ession : N ational S ystem for P erformance M anagement and E valuation ( S inergia ) short term, considerable potential could exist for the find- cording to the constitutional mandate in Article 343 and ings in the medium and long term, particularly with clear Article 29 of the 1994 152 Law. Undertaking them is not evidence of the government’s performance in achieving necessarily “dangerous,” however, because follow-up and presidential goals. evaluation are enabling it to measure the degree of progress It is essential, then, to strengthen relations between the of the commitments set at the beginning, seeing how its Ministry of Finance and NMRS, because the influence that management goes, and trying to correct mistakes in time. M&E information can exert on decision making regard- Likewise, obtaining this information from the executive ing the budget is still unclear. Although under the current branch may be more practical insofar as ministries, imple- scheme it is the NPD’s responsibility to design the invest- menting organizations, and administrative departments are ment budget, it has absolutely no influence over the oper- the owners of administrative information, whereas the pro- ating budget or current spending designed directly by the duction of primary data by an external party would be vir- Finance Ministry. Likewise, coordination could be a pre- tually unviable. requisite for achieving a budget and planning by results. As for goal setting, the government’s definition does not One of the challenges raised by the new stage of necessarily mean that they measure nonessential things. On NMRS’s strategic realignment is to work in conjunction the contrary, goal setting is the main instrument through with the Ministry of Finance. To achieve this end, NMRS which a democratically elected government achieves the and the Ministry of Finance proposed an initiative to estab- mandate it has been given by its citizens. Goals will reflect lish a ranking of efficiency and effectiveness functioning in the priorities, capacities, and resources made available by public organizations to increase transparency and the opti- the government plan, as a condition of delegation to the mization of resources. Table 10 summarizes the channels of executive branch. One should not forget that the process communication between NMRS, the Ministry of Finance, of constructing indicators and goals as an objective unit and the Presidency. of measurement is agreed on at both the technical and the political level. This implies not only the consideration of Outside perspectives: How credible is the methodological and technical aspects, but also introduces information produced by NMRS? political validation by the top decision-making bodies that An external diagnosis of the reliability and usefulness of the give the process political muscle, together with a high lev- information produced by the system indicates that the out- el of appropriation. look is not that gloomy. In fact, to say that it is dangerous Likewise, NMRS has been interested in citizens’ opin- to carry out self-evaluations, to the extent that the “critical” ions and perceptions, as borne out by the fact that its vir- counterpart is absent, may be controversial. First, NMRS tual platform allows readers to make judgments about the performs self-evaluations to the extent and in the way that government’s performance and the progress of its commit- results-based management requires. The government car- ments and the accessibility, quality, and usefulness of avail- ries out an internal follow-up of the NDP and its goals ac- able information. On the other hand, in 2006, the PPE TABLE 10: Channels of Communication FOLLOW-UP EVALUATION PRESIDENT’S OFFICE Management Controls, Council of Ministers and Inter-Sector Committee of Evaluation and public accountability. Results-Based Management coordinates the Presentation of Report to Congress. definition of the evaluations agenda for every four- year period and the evaluations agenda for every fiscal period. MINISTRY OF FINANCE Functional Investment Committees to link goals Inter-Sector Committee of Evaluation and Re- and investment budgets. sults-Based Management technical coordination Ranking of effectiveness and efficiency of public and liaison between evaluation and budgetary entities. programming. Source: Head Office of Public Policy Evaluation, NPD. 95 M&E SYSTEMS AND M&E TOOLS undertook a survey on the quality of the results infor- some of the strategies NMRS has proposed for the next mation, asking more than 3,000 households in 13 capi- few years: tals about transparency, degree of response, accountability, and availability of information, among other issues. One • Increase links between NMRS and other NPD areas of the results that emerged suggests that nearly 47 percent • Increase follow-up of the development plan through of citizens regard public information as useful, which raises budgetary follow-up a challenge and establishes a commitment to guaranteeing • Link the design of follow-up indicators to the de- the availability of more and better information. velopment plan with a long-term development view Second, in evaluation issues, in recent years, NMRS has • Diversify the evaluation agenda, including sectors played the role of judge or third party in contracting eval- that did not traditionally participate uations (carried out by external consultants, as mentioned • Identify the evaluations that other government earlier). This is done to avoid the wrong sort of incentives, bodies have submitted to guarantee international because the ministry or executing entity that pays the eval- standards for evaluations (RADAR) uator is being evaluated as well. • Link the evaluations to broader analysis dimensions Thus, the selection of evaluating firms or the consul- than the program that has been evaluated tant is done through competitions, tenders, and public bid- • Promote policy evaluations in regional and local ding sessions that use rigorous selection schemes endorsed governments by the World Bank. Applicants usually include think tanks, • Define an inclusive policy for internal, external, universities, prestigious research centers, and consultancy and social accountability firms, with teams of well-known professionals. For academ- • Create spheres of exchange with the territorial level ics who wish to protect their reputations and studies, be- to continue expanding the culture of results-based ing hired by a government does not exempt them from the management (RAM). scrutiny of academia. This fact should guarantee the aca- • Promote training in follow-up and evaluation at demics’ objectivity. both the national and territorial level. Third, the Head Office of Public Policy Evaluation is currently working on the development of an M&E net- Diego Dorado work in Colombia. This development provides a master Director, Evaluation of Public Policies (DEPP), plan for the analysis of the achievements and difficulties DNP, Colombia found during the institutionalization of M&E systems. It is also a way to strengthen intergovernmental relations and provide civil society with information about public policy Special Session Notes 31 evaluation that is not necessarily carried out by the NPD or See the NPD‘s Web site, http://www.dnp.gov.co/PortalWeb/De- even the national government. fault.aspx. 32 On the NCESP, see http://www.dnp.gov.co/PortalWeb/tabid/55/ Default.aspx. Challenges for NMRS over the next few years 33 624 indicators are used. Cutoff date October 31 2009. Over the next few years and in the process of its strategic 34 This committee is composed of the presidency, the NPD, the alignment, it is essential for NMRS to focus its efforts on Ministry of Finance, and the respective ministries. 35 consolidating the work it has carried out with the remain- See the Web site, www.dnp.gov.co/sinergia. 36 ing entities in the public sector, particularly with the Min- Although NMRS did not take part in this exercise, sector committees do constitute a coordination and negotiation link istry of Finance. Likewise, the main challenge is to produce between the Ministries of Finance and Planning in the effort relevant, quality information for citizens. The following are to consolidate a results-based budget. 96 Session 4: Monitoring and Evaluation Systems: Capacity Building for Monitoring and Evaluation and Academia-Government-Civil Society Partnerships Conceptual Framework for Developing Evaluation range of stakeholders in the development and humanitari- Capacities Building on Good Practice in an communities. The chapter aims to provide these stake- Evaluation and Capacity Development holders with a conceptual framework to support their ef- forts. It is divided into three main sections: one to discuss Introduction evaluation capacities at three levels, which overlays the eval- This presentation contributes to a better understanding uation principles and capacities at three levels, one to dis- of the complexities of evaluation capacity development cuss the capabilities and processes to instill them, and fi- and suggests ways of approaching them, within an institu- nally one to discuss examples and draw out suggestions for tional context or at national level. It aims to make use of stakeholders working on evaluation capacity development. “good practice” in capacity development to ensure attain- ing “good practice” in evaluation. It builds on established Evaluation capacities at three levels “good practice” in these two professions and on the experi- ence of evaluating capacity development. Evaluation principles Both professions—evaluation and capacity develop- The evaluation principles date back to a meeting of evalua- ment—are multifaceted and evolving, which means this tion professionals in 1991 and probably to earlier meetings. chapter captures only a segment of the whole at a certain They have become integral to the evaluation profession at point in time. Capacity development practitioners contin- large.� For instance, independence has taken central stage ue to develop and refine concepts and practices. The Or- in the multilateral development banks when the evaluation ganisation for Economic Co-operation and Development’s function at the World Bank started reporting to the execu- Development Assistance Committee (OECD/DAC 2006) tive board rather than the president; a change that was rep- publication on capacity development recognizes the ongo- licated in regional development banks and that stimulated ing challenges. In a similar way, the evaluation profession discussions in, for instance, the United Nations. In 2008, continues to grow. the framework for peer reviews raised three principles to be The demand for evaluation is high, but increasingly central to evaluation: independence, credibility, and utility critical of evaluation practices that are not demonstrating (DAC/UNEG 2007). independence, credibility, and utility. In addition, follow- Independence forms the bedrock of good evaluation prac- ing the Paris Declaration and the Accra Agenda for Action, tice (Chelimskey 2008). It is fundamental to attaining credi- demand is increasing for evaluation capacities in partner bility and utility of evaluation. It should lead to impartiality countries, and the evaluations of the “Delivering as One” reflected in, for instance, the choice of subjects of evalua- initiative of the United Nations will be country-led. There- tion or the selected evaluation method. It is achieved through fore, developing evaluation capacities is important for a means at structural, institutional, and individual levels. 97 M&E SYSTEMS AND M&E TOOLS Credibility of evaluation is enhanced with greater inde- function, which reduces its usefulness. Or, some stakehold- pendence but needs to be accompanied by competence of ers might perceive evaluations to be useful solely when they the evaluators, transparency of the evaluation process, and report only successes (and omit information about fail- impartiality of evaluators and process. ures), or when it serves fund-raising or publicity purposes. Utility of evaluation is not guaranteed with indepen- Or, policy makers may want to pursue a certain course of dence and credibility, but requires that commissioners and action that they do not like to see derailed by evidence that evaluators undertake the evaluation with the intention to would advise otherwise. Such evaluations have little credi- use its result, and undertake it at a time when the results bility with other, well-informed stakeholders and are of lit- can meaningfully inform decision-making processes and tle, if any, use for resolving problems, as they fail to dis- that evaluations are accessible. This principle exists to en- cuss problems. In other examples, stakeholders believe that sure evaluations are undertaken to influence change. credibility can derive exclusively from close familiarity with Together these principles are markers of quality evalu- the subject under evaluation, but they ignore the possibil- ation and ensure “good practice” in evaluation. Indepen- ity that such a close link eliminates independence and re- dence, credibility, and utility are three equal sides of the tri- duces the chance that the evaluation will produce an im- angle at the center of which the quality of evaluation rests. partial and credible evaluation that provides new insights. �� Their interrelationship, including inherent complementa- Each evaluator must act in line with these evaluation rities and tensions, is illustrated in Figure 21. Each of the principles. Measures are needed to institute the indepen- central evaluation principles has other, supporting ones. dence, credibility, and utility of evaluation to ensure that Complementarities exist among these principles: an adherence to these principles is less dependent on individ- evaluation that is biased toward the perspective of one uals. Evaluators may be subjective. They may lack an un- stakeholder (meaning not independent or impartial) tends derstanding that independence is granted to ensure impar- not to be credible with others. If an evaluation lacks credi- tiality and not for other reasons, and they are at risk to be bility it is unlikely that stakeholders will pay attention to it, put under pressure from stakeholders who want to influ- let alone use it. Ignoring the stakeholders, including their ence evaluation findings. Therefore, it is important to have interest or timelines, will reduce the utility of an evalua- an institutional framework that holds evaluators account- tion but also the credibility of its process. By maintaining a able and protects them from undue influence, and an en- good balance among independence, credibility and utility, abling environment that supports the notion of evaluation, these three principles will reinforce each other and enhance learning, and accountability. the quality of the evaluation. But the evaluation principles also have tensions. For in- Capacity development: working at three levels stance, independence can lead to isolation of the evaluation Capacity development is recognized by many as central to development and progress. The Accra Agenda for Action is FIGURE 21: the latest such commitment to strengthening capacities in Evaluation Principles support of ownership of development processes. For many Competence Intentionality years, investments into developing capacities have been manifold and have been supported by many different ac- Transparency Credibility Utility Timeliness tors, many of whom struggle with similar challenges. The concept of capacity development evolved from a focus on Quality training of individuals, through institutional development, Impartiality Accessibility when it was recognized that individuals worked within the context of their organizations and therefore more than Independence training was needed to be successful (UNIDO 1990).� Ca- pacity development evolved to acknowledge that organi- Strcutural Institutional Individual zations do not work in isolation but rather require an en- Source: Author. abling environment that consists of, among other things, 98 S ession 4 : M onitoring and E valuation S ystems policies, networks, and an attitude of engagement. Capac- FIGURE 22: ity therefore goes beyond an individual or an organization, Levels of Capacity Development but includes all three levels, as illustrated in Figure 22. The enabling environment provides a context that fosters (or hinders) the performance and results of individuals and organizations Evaluation capacities at three levels Capacity development practitioners highlight the impor- The institutional framework in which individuals tance of working at three levels, which—when applied to work needs to provide a system and structures the world of evaluation—illustrate the need to institution- in which individuals can perform and attain results individually as well as collectively as an alize the evaluation principles with measures that go be- organization yond the individual, and span the institutional framework The individual whose knowledge, skills and and the enabling environment for evaluation. competences are essential to perform task and manage processes and relationships Enabling environment Source: Author. The enabling environment for evaluation is determined by a culture of learning and accountability, meaning the degree to which information is sought about past performance, the risks: (1) that declared commitments to independence are extent to which there is a drive to continuously improve, revoked through personal actions—systems can be ob- and the need to be responsible or accountable for actions served and, once established, more difficult to reverse with- taken, resources spent, and results achieved. Such culture is out agreement of all stakeholders—and (2) that inherent in embedded in tacit norms of behavior, the understanding of the dependence on the individuals, who need a system of what can and should—or should not—be done, in many checks and balances to ensure accountability and protec- cases behaviors being role-modeled by leaders. They are, or tion. Such an institutional framework should be, codified in government legislation or an evalu- ation policy that expresses the leadership’s or an organiza- • Includes a system of peer review or assurance that tion’s commitment to learning, accountability, and the eval- the evaluation function is set up to safeguard and uation principles. An enabling environment is supported implement the principles of independence, credi- or created through governance structures that demand in- bility, and utility. dependent evaluation, whether through parliaments or gov- • Establishes safeguards to protect individual evalu- erning bodies, and is further enhanced through professional ators—evaluators, evaluation managers, and heads associations and networks that set standards and strive to- of evaluation functions—when exercising their in- ward greater professionalism in evaluation. Finally, the struc- dependence, including transparent and credible tural independence of an evaluation function is important processes for the selection, appointment, renewal to setting an enabling environment. Placing the evaluation (if applicable), and termination of contract of the function in such a way that it does not report to the person head of evaluation and assurances that evaluation or function responsible for the policies, strategies, or opera- staff do not suffer in promotion exercises. tions that are evaluated creates an environment that enables • Ensures that a multidisciplinary team exists in the greater independence, credibility, and utility. Ideally, the en- evaluation function or on an evaluation team that abling environment is such that decision makers proactive- ensures credibility of evaluation by understand- ly demand impartial evaluations to inform their debates and ing multiple dimensions of evaluation subjects and choices, which increases the usefulness of evaluations. combining the necessary technical competence. • Secures the independence of funding of evalua- Institutional framework tions, at an adequate level, to ensure that necessary The institutional framework for evaluation ensures that evaluations are carried out and that budget hold- a system exists to implement and safeguard the indepen- ers do not exercise influence or control over what dence, credibility, and utility of evaluation. It reduces two is evaluated and how. Therefore, funding should be 99 M&E SYSTEMS AND M&E TOOLS BOX 2: Structural independence Structural independence requires that an evaluation function or office does not report to the same individual who is responsible for the subject of evaluation. Otherwise, the evaluation function might be under political or organization- al pressure that interferes with the independent and impartial planning, conduct, and reporting of evaluation findings. For example, • National systems evaluators report directly to parliament to ensure the highest degree of independence and ensure this decision body benefits from impartial evaluation insights; • Many of the development banks, starting with the World Bank, this structural independence translated into eval- uation functions reporting directly to the executive boards of these banks; • The UN system for which the Evaluation Norms suggest that a reporting line to the governing body and/or the head of the organization were adequate to ensure structural independence; and • The practices of the Food and Agriculture Organization or the United Nations Development Programme, which introduced dual reporting lines: one functional reporting line to the governing body and an administrative line to the head of the organization. Structural independence does not guarantee impartiality. It requires the personal or behavioral independence of evaluators, and institutional measures that help safeguard independence and ensure it leads to impartiality. under the direct control of the head of the evalu- when choices are made for site visits, the selec- ation function, and should be adequate for a rea- tion of projects or programs for inclusion in an sonable work program. The adequacy of funding evaluation work plan of an organization or of for evaluation can be determined through a ratio initiatives of a national program. As a result, of evaluation resources (human and financial) over the findings of the evaluation(s) will be as rep- the total program size (some organizations aim to resentative as possible, and therefore indicate allocate 1 to 0.8 percent of their total budget for strengths and weaknesses that are likely to ap- evaluation) or the coverage rate of evaluations (for ply to the project, program or portfolio as a instance, evaluating 25 percent of the overall pro- whole. There is no bias to select only the good gram, or to evaluate a minimum of 30 projects per performers or only the problem cases. year to have a valid sample size). • To ensure utility, for instance, of strategic or • Combines measures for impartial or purposive se- thematic evaluations, these may be chosen lection of evaluation subjects to ensure impartially purposively (rather than to establish a repre- on the one hand and increased utility on the oth- sentative sample) to link the conduct of the er by making deliberate choices linked to decision- evaluation to the information needs and de- making processes. cision-making processes of those using evalu- • To avoid biases in the selection of subjects for ation findings. In these cases, a consultation evaluation, objective selection criteria are used process to determine what are the most im- to ensure the evaluation sample is representa- portant and/or strategic topics for evaluation is tive of the whole. This principle would apply important to ensure various stakeholder needs to the sample selection in a project evaluation are considered. 100 S ession 4 : M onitoring and E valuation S ystems • Sets out a system to plan, undertake, and report nal evaluators are believed to exercise greater independence evaluation findings in an independent, credible, than those who work in an organization, because they are and useful way. To increase objectivity in the plan- less exposed to institutional or peer pressures or have not ning and conduct of evaluation, systems are needed absorbed the corporate culture in an unquestioning way. that increase the rigor, transparency, and predict- However, individual or intellectual independence is depen- ability of evaluation processes and products. Such dent on the individual who demonstrates in his or her be- systems can include more or less detailed process havior the adherence to and practice of the evaluation prin- descriptions or guidelines for the design of evalu- ciples: avoiding conflict of interest, acting with integrity and ations, for preparatory work that needs to be con- with an independence of mind, engaging in evaluations for ducted and reported on, and for reporting find- which they are competent, acting impartially, and undertak- ings. The processes should have built-in steps for ing an evaluation with a clear understanding of the clients communication, consultation and quality assur- or decision-making process and how these need to be in- ance, and be communicated to stakeholders to en- formed. Many evaluation associations and evaluation func- hance transparency and to secure their willingness tions of national and international organizations adopted to share information. codes of conduct for evaluators, and the United Nations • Institute measures that increase the usefulness of Evaluation Group developed addition ethical guidelines evaluations, including the sharing of findings and (UNEG 2007). Table 11 provides an overview of the points lessons that can be applied to other subjects. Eval- discussed above and illustrates the intersections among the uations should be undertaken with the intention— three levels of capacities and the three evaluation principles.� of stakeholders and evaluators—to use their results. For projects or programs, this intention can derive From capacities to capabilities: from plans to use evaluation findings in the design The process matters in a future operation. Collectively the findings of The above discussion centers on individuals and enti- these evaluations can be used to understand the per- ties, such as governing bodies, evaluation units, or evalu- formance of an organization with the intention to ators and how they collectively form an evaluation capaci- replicate well-performing aspects and rectify system- ty of mutually reinforcing ingredients. To be effective, these ic problems. The timeliness of planning and con- evaluation capacities need to be capable of delivering evalu- ducting evaluations and presenting their findings is ation services—producing independent, credible, and use- equally important to ensure utility. Finally, accessibil- ful evaluations—and much more. ity means that evaluations are available to the public, can be retrieved (for instance, through a user-friend- Five capabilities ly Web site), are written in ways that are understand- The five-year project on capacity development (ECDPM able (clear language and limited jargon), and are dis- 2008) concluded that capacities, at whichever level they ex- tributed to a wide group of relevant stakeholders. ist, needed to be able to achieve the following: Individuals • Commit and engage: volition, empowerment, mo- Debates about professional standards and accreditation tivation, attitude, confidence of evaluators and evaluation managers have been ongoing • Carry out technical, service delivery, and logistical within professional forums for years. Even with structures tasks: core functions directed at the implementa- and systems in place, the independence and impartiality tion of mandated goals of evaluation depends on the integrity and professionalism • Relate and attract resources and support: managed of individuals as evaluators, evaluation managers, and the relationships, resource mobilization, networking, head of evaluation. legitimacy building, protecting space The profession and its ethics require limiting personal • Adapt and self-renew: learning, strategizing, adap- biases to the extent possible. In some circumstances, exter- tation, repositioning, managing change 101 M&E SYSTEMS AND M&E TOOLS TABLE 11: Evaluation Principles and Levels of Capacity Independence Credibility Utility Enabling • Culture of Accountability and Learning Environment • Government Legislation or Evaluation Policy • Community of Practice, Networks, Associations for Evaluation • Governing or over- • An attitude of provid- • An attitude that entails the sight body that seeks ing access to informa- intention to use evaluation find- independent credible tion that allows cred- ings and recommendations advice ible evaluation • Willingness to change • Structural indepen- dence Institutional • Institutionalized process of “peer review” to assess the independence, credibility, and utility of the Framework evaluation function • Independent budget • Evaluation quality • Understanding of the use of • Impartial selection of assurance system to evaluation evaluation subjects ensure credibility, • Consultation processes • Independent planning transparency, and im- • Timeliness and conduct partiality • Accessibility of evaluations • Noninterference in • Multidisciplinary evalu- • Active sharing of lessons from reporting of findings ation team that works evaluation • Measures to protect well together • Utility demonstrated by the evaluators from reper- actual implementation of evalua- cussions tion recommendations • Ethics Guidelines • Code of Conduct Individual • Avoidance of conflict • Competence (technical • Client orientation of interest knowledge and evalua- • Communication • Behavioral indepen- tion skills) • Coordination with different dence • Impartiality stakeholders and with ongoing • Integrity processes Source: Author. • Balance coherence and diversity: encourage inno- ing by evaluation principles, and for the evaluator entails an vation and stability, control fragmentation, manage attitude of behavioral independence, among other things. complexity, balance capability mix The application of these capabilities to the three lev- els at which evaluation capacities need to be established is Evaluation capabilities summarized in Table 12. This recognition is important for moving from having, for �The enabling environment for evaluation needs to be instance, a written evaluation policy that means little, how- committed to a culture of learning and accountability. It ever, because it is not backed by capabilities to commit and should adopt an evaluation policy in line with the evalua- engage. An evaluation unit may exist and carry out evalua- tion principles and use evaluation findings and insights in tions, but may lack the capability to relate to stakeholders its policy making, performance improvements, and organi- and attract resources, or to adapt and self-renew by learn- zational renewal. It accepts that the independence of eval- ing to provide new evaluation types in response to chang- uation needs to be safeguarded, including its funding. In a ing needs. national context, this could entail embedding evaluation In addition, these capabilities need to be interpreted dif- into legislation and government policy. In an organization, ferently depending on the stakeholders concerned. For in- the corporate culture, evaluation policy, and resources for stance, the commitment and engagement of stakeholders evaluation form the context in which an evaluation func- in the enabling environment will set the culture for learn- tion operates. ing and accountability, whereas the capability to commit The evaluation function should be committed to the and engage for an evaluation unit entails setting and abid- evaluation principles, protect these and the evaluators from 102 S ession 4 : M onitoring and E valuation S ystems TABLE 12: Evaluation Capabilities Enabling Environment Institutional Framework Evaluators, Evaluation Managers, Capabilities for Evaluation for Evaluation Heads of Evaluations Units To commit and A culture of learning and ac- The commitment to evaluation Commitment to evaluation principles, engage countability that empowers principles, and the motivation ethical guidelines, and code of con- individuals and organizations and attitude to follow them duct. Behavioral attitude of indepen- to reflect on their practice, through in daily practice. dence and impartiality. Motivation to take stock of what works and conduct evaluations in a credible and what does not, and take neces- useful way. sary action. To carry out techni- An evaluation policy that codi- Ability to conduct I/C/U evalua- Competence to conduct evaluations, cal, service delivery, fies evaluation principles and tions to support organizational both from an evaluation and a techni- and logistical tasks good practice. adaptation and renewal. cal point of view. To relate and attract To legitimize evaluation The ability to demonstrate Capability to conduct evaluations resources and through the evaluation policy legitimacy based on impartial- transparently and credibly, and to support and actions, to recognize and ity and credibility, to protect communicate and relate with stake- ensure the need for adequate evaluation principles and stan- holders. and independent funding. dards and the evaluators, and to generate adequate funding. To adapt and Recognition of evaluation The ability to make strategic Learn new evaluation skills. self-renew playing a role in adaptation choices about evaluation’s and self-renewal. engagement in organizational renew and to relate evalua- tion findings to stakeholders to ensure learning. Frequently adapt and update evaluation methods and approaches. To balance Manage change, using evalua- The ability to ensure evalua- Find the right balance between coherence and tion evidence and recommen- tion processes and approaches systematically applying evaluation diversity dations when available. Avoid are systematic but flexible, guidelines and seeking opportunities fragmentation of systems for balance capabilities on the for innovation. learning and accountability. team, and employ a mix of approaches to develop an evaluation culture. Source: Author. pressures, and have the motivation to implement them in tion, as evaluators, evaluation managers of heads of evalua- their daily practice. It should have a system that safeguards tion units—have to be committed to the evaluation princi- and institutionalizes independence, credibility, and utility. ples, the ethical guidelines, and the code of conduct. He or In its relationships with others, it needs to demonstrate she has to demonstrate independence and impartiality, and legitimacy based on impartiality and credibility (includ- conduct evaluations in a credible and useful way. Evalua- ing technical competence) and it needs to contribute to tors are expected to be technically competent as evaluators the adaptation and renewal of the context within which or competent in the field that is being evaluated, or both, it works—an organization or a wider, say, public sector re- and keep updated in his or her knowledge of evaluation form beyond just one organization. It has to have the ca- methods and techniques. He or she should have the abili- pability to work systematically (to ensure transparency) but ty to follow guidelines, but seek possibilities for innovation. flexibly (to ensure credibility and utility), use a mix of tools to foster an evaluation culture, and keep its methods and The process matters approaches updated. The evaluation unit needs to have a The process of capacity development is, in itself, an impor- mix of capabilities on its teams and has to have the capabil- tant part of the outcome, because capacities and capabilities ity to determine and negotiate adequate funding. are developed through a process of interaction and dialogue, The individual evaluators—as staff of an evaluation unit mixed with specific initiatives to set up systems, install hard- or as consultants contracted to conduct a particular evalua- ware, and train people. The process combines parts that are 103 M&E SYSTEMS AND M&E TOOLS well planned, based on participatory diagnostics of capaci- development process. Developing a common understand- ties and their weaknesses, and others that are flexible and op- ing of capacity weaknesses through a participatory, struc- portunistic. Just the number of stakeholders and the range tured diagnostic—using tools like the conceptual frame- of capacities and capabilities discussed above illustrate the work in Figure 22 or the five capabilities in Table 12 can complexities of capacity development and flag the need for serve as an analytical framework to move attention from re- planning and implementation tools that are able to address source gaps to recognize broader issues—and a shared vision complexity, something that blueprints tend not to be good of the needed capacities are important steps in channeling at. Instead, space needs to be created for analyzing capaci- resources toward clear objectives. Very often the many ac- ty gaps in a participatory way, agree on common goals and tors in capacity development have tacit understandings of translate them into a joint strategy, while at the same time what capacity is and how its development should happen. maintaining the flexibility and capability to take up oppor- Few of them have explicit strategies, which makes it more tunities as they arise, learn from experience, and change tac- difficult to find a common ground and work in the same di- tics and work on various levels of capacity at the same time. rection. A diagnosis of existing capacities is necessary to find Capacity development practitioners arrived at the con- entry points to start capacity development. This diagnosis clusion that 10 process matters are important for greater benefits when it is developed in a participatory way that de- success in capacity development. In this chapter, these pro- velops ownership at the same time. cess issues have been regrouped around (1) the drive from The “ordered chaos” or combining high degrees of flex- within, (2) development and agreement of clear expecta- ibility with systematic approaches entails a recognition that tions, and (3) “ordered chaos” or combining high degrees capacity development, while systematic, does not (always) of flexibility with systematic approaches. follow an ordered process. Instead it includes planned as The drive from within has to do with the realization well as incremental and emergent approaches. For instance, that capacity development needs to come from within or when developing specific skills, a well-structured process have strong internal champions, whether it is when devel- would be appropriate. On the other hand, when trying to oping a national system or a function in an organization. influence culture to become more supportive of learning If an idea is imposed from the outside, chances are that ca- and accountability, opportunities might arise at short no- pacity development is not likely to succeed. The drive from tice, in various settings, and without prior notice, and in- within relates to the capability “to commit and engage” and clude an informal conversation with a key stakeholder. In includes the following: these contexts, capacity development initiatives would be incremental and emergent in response to opportunities as • Ownership, which manifest itself in having local or they arise. Capacity development also requires recognizing internal champions, resource allocations, and en- the less tangible aspects of capacities and combining small gagement. It is, however, difficult to measure, not and large initiatives, depending on context and opportuni- homogenous across internal stakeholders, and not ties. It requires finding the right balance between an operat- constant over time ing space that allows capacities to evolve and accountability • Leadership, which is important, especially for set- for capacity development results; a balance that is hard to ting the culture of learning and accountability, but attain. And, finally, capacity development takes time, so it also must fit with the context and its culture is necessary to stay the course (even through adverse times), • Collective action, motivation. and commitment, but build quick wins into the process (to keep up motiva- meaning that capacity development will not occur tion and build on success) and allow time to reflect and if it is one person writing documents, but requires evaluate whether progress is being made. engaging various stakeholders whose capacities and capabilities will be developed Where to start? Illustrative examples: What does it mean for stakeholders? The process of developing and agreeing on clear expecta- There is no right answer to the question “Where to start?” tions in capacity development is in itself part of the capacity Depending on the situation, a diagnostic could be the best 104 S ession 4 : M onitoring and E valuation S ystems entry point to understand where the strengths, weakness- tain standards to develop a corresponding and relevant ca- es, and gaps are. Sometimes, however, it might be neces- pacity development strategy. sary to first have a champion within who provides own- ership and leadership of the process and generates interest Placing evaluation capacities in the context of in developing evaluation capacities. In other situations, it good governance might be clear that the quality and credibility of evalu- Governance and evaluation are interdependent. Good ation has to be improved before anything else can hap- governance creates an enabling environment for evalu- pen. Or, in other cases, the incentives for accountabili- ation, while evaluation reinforces good governance. Sri ty and learning need to be improved before anything else Lanka provides a good example of how the commitment can happen. And, in many cases, it requires a mix of all of from the highest political level—the prime minister—cre- these things to develop capacities, but these choices are sit- ated an enabling environment for evaluation. By demand- uation specific. ing reports on results, stakeholders in ministries were en- An entry point for developing institutional capacities abled, even required to record and critically reflect on is a diagnostic, which can be done using the conceptu- performance. This strong leadership was combined with al framework suggested here, and through peer reviews in a strong champion to translate political commitment in- the bilateral system, the UNEG and Evaluation Cooper- to practical action. ation Group of the multilateral development banks. This Ideally, governments or chief executives of organiza- expert review can support a diagnostic process and lend tions seek independent, credible, and useful advice through it further credibility, but the evaluation unit concerned an evaluation function. Demand for evaluation creates a needs to be involved in and own the diagnostic process. strong enabling environment for evaluation. Decision- The diagnosis could lead to a revision of policy, as has makers demand feedback on the use of resources and re- happened in the case of the United Nations Development sults achieved with them and set, through their demand Programme, or to the formulation of a strategy to develop for such information, an environment that enables evalua- evaluation capacities. tive thinking and practice. However, political and institu- Few organizations have articulated an evaluation strate- tional power structures can limit the environment in which gy, even less so one that explains how evaluation capacities evaluations are conducted, especially if vested interests re- will be developed. Yet, good practice in capacity develop- sist evidence that might demonstrate the weaknesses of po- ment argues for articulating such strategy to have a com- litical choices. In these cases, it is important to counterbal- mon vision and bring stakeholders together to work toward ance political power structures with other stakeholders and a common goal. Such strategy would employ linear plan- to establish legislation or an evaluation policy that legiti- ning approaches for some activities, for instance when im- mizes and commits the country or organization to the eval- plementing a training incremental approach, for instance, uation principles. An increasing number of United Nations when gradually expanding the evaluation requirements in a organizations are adopting evaluation policies. In the case national system as discussed in the Sri Lanka case study to of the World Food Programme, the evaluation policy estab- emergent approaches, when a number of flexible, informal lishes the commitment to the evaluation principles and ex- initiatives are needed, for instance, to respond to opportu- plains measures to safeguard them. nities when possible and necessary. An environment that is unresponsive to or even fearful The following paragraphs highlight examples to draw of evaluation might originate in a lack of understanding of out suggestions for stakeholders and how they might work the value of evaluation to make more informed decisions. toward developing evaluation capacities. It is structured In these cases, it is for the evaluation function to deliver around the three levels—the enabling environment, the in- credible, high-quality evaluations to demonstrate the val- stitutional framework, and the individual—as these three ue added of evaluation. In addition, it will be necessary to levels associate well with identifiable stakeholders. At any explain the role and usefulness of evaluation in the con- of these levels, it is advisable to understand the existing ca- text of governance and performance of the organization as pacities, the capabilities, and reluctances to adopting cer- a whole. Table 13 provides tips for stakeholders. 105 M&E SYSTEMS AND M&E TOOLS TABLE 13: Tips to Reinforce an Enabling Environment for Evaluation Governing Bodies (Parliaments, Executive Boards, and so on) • Be aware of the threats to independence, credibility and utility of evaluation and demand for measures to safeguard these principles • Adopt and oversee the implementation of legislation or policies that institutionalize the independence, credibility, and util- ity of evaluation • Exercise oversight over the quality of evaluation • Demand for evaluation and demonstrated use of evaluation findings and recommendations Head of State, Minister, Policy Makers, Chief Executives • Demonstrate leadership in setting a culture of learning and accountability • Seek and use evidence from evaluations to validate the attainment of goals and objectives and to improve performance whenever possible • Understand evaluation as part of good governance that aims to use public resources effectively and efficiently Evaluation Function • Demonstrate value added of independence, credible, and useful evaluations • Raise awareness of stakeholders about the role and importance of evaluation and of the evaluation principles • Contribute to evaluative thinking through awareness building, dialogue, and training Professional Evaluation Networks, Associations • Set standards as benchmarks that can be used to convince other stakeholders about the importance of the evaluation prin- ciples and measures to safeguard them Source: Author. Delivering evaluation services independently, ful evaluations. The process should assure governing or credibly, and usefully oversight bodies and chief executives alike that an impar- The importance of developing an institutional framework tial candidate has been chosen, and the selected candidate for evaluation, rather than using ad hoc arrangements, lies should have contractual arrangements that protect him or in the safeguarding of the evaluation principles, the provi- her from repercussions, should critical evaluations be un- sion of a framework that protects and holds evaluators ac- welcome. In addition, it is important that a system is set countable, and the predictability and transparency of this up by which evaluators are held accountable for their ac- framework for all stakeholders. Such a system needs to be tions through a code of conduct or ethics guidelines, such built in ways that it has the capability to adapt and self-re- as those established by the UNEG. new, one of the five capabilities, to respond to new challeng- �To set up a planning system that is representative for es with, for instance, a focus on new issues or by introduc- evaluations of operations, the Office of Evaluation at the ing new types of evaluations that address information gaps. United Nations World Food Programme (2008b) analyzed Generally, the evaluation functions need to lead efforts the portfolio to identify factors that should be used to en- to develop the institutional framework for evaluation, set- sure a selected sample of operations is representative of the ting up systems, processes, and guidelines that are in line total. This resulted in a system that combines the size and with the evaluation principles and fit with the work of their number of operations to reflect the considerable variation in organization or the national context. these factors: a relatively small number of operations repre- These systems may include human resource manage- sent a large proportion of the financial value of the portfo- ment measures that ensure the selection of an indepen- lio, while the rest is spread over a large number of small op- dent and qualified head of the office of evaluation. The erations. Treating each operation, regardless of size, equally best systems will not—and should not—eclipse the human would have overemphasized the smaller operations in the factor, but this makes it important to select and appoint sample; selecting the sample by size would have underrep- heads of evaluation functions that are capable of safeguard- resented the smaller operations in the overall sample. The ing the evaluation principles and managing the evaluation weighted numbers are applied to determine how many op- function so that it delivers independent, credible, and use- erations in each geographic region should be selected for 106 S ession 4 : M onitoring and E valuation S ystems BOX 3: Selecting the head of evaluation Selecting the Head of Evaluation • The process should be agreed upon between key stakeholders. • The process should be documented and publicized for reasons of transparency. • The composition of the panel needs to ensure credibility in the eyes of key stakeholders, requiring the repre- sentation of stakeholder groups and the profession. • The panel should have clear terms of reference that specify whether the panel “chooses” or “suggests”, and if it is the latter who has the final decision-making power. • The panel should be involved in the entire process from agreeing on the vacancy announcement, the selection criteria, agreeing on interview questions and criteria for assessing answers, interviews, and so on. • Appointment should be made for a fixed term period, including a contract for the same duration. • Contract renewal should follow a process that involves the panel to ensure the renewal is based on perfor- mance and not dependent on producing biased evaluations. • The termination process, while following standard human resource procedures, should again involve the pan- el to ensure termination of contract is based on performance issues and not a repercussion for critical evalu- ations. evaluation. By contrast, a different system was needed for An example of a system that ensures complete indepen- the strategic evaluations to ensure that they address issues dence of finance is that of the International Fund for Ag- that are important to key decision makers in the governing riculture Development (IFAD). Its evaluation policy de- body and management, and revolves around a consultation termines that the budget is prepared independently of that process with these stakeholders. The utility of all evaluations of the fund’s secretariat. The process is managed by eval- is enhanced by linking them to decision-making processes. uation office and involves discussions and approval of the TABLE 14: Tips to Develop an Institutional Framework for Evaluation Governing Bodies (Parliaments, Executive Boards, and so on) • Get briefed about the evaluation system to understand whether the institutional framework includes adequate checks and balances and to become a discerning reader of evaluation reports. Head of State, Minister, Policy Makers, Chief Executives • Get briefed about the evaluation system to understand whether the institutional framework includes adequate checks and balances and to become a discerning reader of evaluation reports. • Introduce processes for the selection of the head of evaluation that ensures his/her independence. • Provide secure, separate, and adequate funding for evaluation. Evaluation Function • Develop and document systems for the selection, design, conduct, and reporting on evaluations. • Provide briefings on these standards to increase transparency and confidence in the process and products, which enhances credibility. • Develop mechanisms to ensure lessons from evaluation are systematically shared and integrated into debates and decision- making processes. Professional Evaluation Networks, Associations • Set professional standards and good practice standards. • Develop and implement professionalization, accreditation, and credentialing systems. 107 M&E SYSTEMS AND M&E TOOLS evaluation committee. The work program and budget of the profession. They determine whether an evaluation is credi- evaluation office appears as a separate, clearly identifiable ble and useful. Their professional competences—as evalua- section in the fund’s overall work program and budget.39 tors and in the technical area that they evaluate—and their A further system that is important to develop is one that ability to listen, understand, and weigh the different views governs evaluations processes and sets standards for evalu- of stakeholders are central to the quality of evaluation. ation products. They often take the form of guidelines or The importance of training as a tool for one part of handbooks. Placing information about the process into the the capacity development process is well understood. The public domain increases transparency of evaluation pro- chapter illustrates the importance of a structured training cesses. The production of guidance materials often needs program that is based on specific training needs assessments to go hand in hand with briefings: between the evaluation and training goals. In addition, it is important to coach manager and the evaluation team, and between the eval- and mentor evaluation managers and evaluators, especially uators and the managers of operations to clarify expecta- when they are working for the first time in this field. tions in evaluation quality and process. The process needs to include formal feedback processes and quality standards Conclusion and assurance mechanisms. If evaluations are conducted in This chapter demonstrates the synergies that exist between a decentralized way—that is, not managed by the central the evaluation profession and that of capacity development evaluation office—training will be needed to ensure that practitioners. A combination of good practice from both standards are understood and implemented. sides promises results in greater effectiveness to develop Evaluation functions need to have capacities to incorpo- evaluation capacities that will safeguard and use the eval- rate findings and recommendations into decision-making uation principles of independence, credibility, and utility processes as well as to share lessons from evaluations. These and also promises higher returns on investments in eval- systems can include search engines on Web sites of reposi- uation of capacity development. It comes at an important tories of lessons from evaluations, incorporating evaluation time when evaluation needs to professionalize, and when lessons into guidance materials that program managers use, demand for evaluation and for developing such capacities or proactive efforts to share evaluation lessons throughout is high. the evaluation process. The consideration of three levels of capacity (the en- abling environment, the institutional framework, and the Impartiality, knowledge, and skills of individuals individual) are important to institutionalize evaluation ca- The best enabling environment and greatest institutional pacities in ways that it supports and protects individual arrangements will not replace the individuals involved in evaluators, evaluation managers, and heads of evaluation evaluation. Their integrity to plan, undertake, and report functions, while holding them accountable. These three evaluation findings in an impartial way is essential for the levels, together with the capabilities that are needed, pro- vide a conceptual framework that moves the debate from TABLE 15: training individuals for improved skills or calls for resourc- Tips for Stakeholders to Develop Skills and Knowledge of Evaluators es to a more sophisticated set of ingredients that promise to be mutually reinforcing capacities. Applying the concep- Evaluation Function tual framework to specific examples provides a number of • Invest in training • Build teams that are multi-disciplinary pointers for stakeholders who want to work on evaluation • Encourage team work and cross-fertilization capacity development. Providers of Evaluation Training • Continue to develop and offer training courses Caroline Heider • Increasingly move towards university courses to build a professional tradition Director, Evaluation Office, World Food Program, Professional Evaluation Networks, Associations, etc. United Nations • Provide opportunities for professional exchanges and networking 108 S ession 4 : M onitoring and E valuation S ystems BOX 4: Definitions Definitions Accessibility means that stakeholders have unrestricted access to evaluations that can be retrieved easily from their stor- age place. It also requires that these evaluations are written in ways that are easily understandable and that findings are shared with stakeholders and interested parties. Accountability is the obligation to account for (and report on) work carried out and results achieved. Competence of evaluators means that they have and can demonstrate a good understanding of the subject under eval- uation and of the evaluation principles, rigorous data collection and analysis, including ethical principles for evaluation. Credibility is the extent to which evaluation findings and conclusions are believable and trustworthy. Credibility is determined by objective factors, such as the transparency of the evaluation process and accuracy of the evaluation re- port, and subjective factors such as the perceived or demonstrated impartiality and competence of the evaluators. Learning means that lessons are drawn from experience and, accepted and internalized in practices, thereby build- ing on success and avoiding past mistakes. Impartiality is the absence of bias; it entails due process, methodological rigor, and the consideration and presenta- tion of achievements and challenges, success and failures. Independence means that the evaluation is free from influences—political or organizational pressures or person- al preferences—that would bias its conduct, findings, conclusions or recommendations. It implies that evaluations are typically carried out or managed by entities and persons free of the control of those responsible for the design and im- plementation of the subject of evaluation. Intentionality is a clear intent to use evaluation findings. In the context of limited resources, the planning and se- lection of evaluation work has to be carefully done. Transparency means that consultation with the major stakeholders is an essential feature in all stages of the evalua- tion process. This improves the credibility and quality of the evaluation. It can facilitate consensus building and own- ership of the findings, conclusions and recommendations Timeliness means that evaluations are chosen and undertaken in a manner that their findings and recommenda- tions are available in time to inform decision-making processes. Utility of evaluations means that they aim to and do affect decision making. Therefore, evaluations must be per- ceived as relevant and useful and be presented in a clear and concise way. Evaluations are valuable to the extent to which they serve the information and decision-making needs of intended users, including answering the questions posed of the evaluation by its commissioners. Sources: EC Evaluation Glossary (online), OECD/DAC 2002, DAC/UNEG 2007, UNEG 2005, UNEG 2007, United Nations World Food Programme 2008a. The Experience of the Latin American and ribe de Planificación Económica y Social, ILPES) is a perma- Caribbean Institute for Economic and Social nent body with its own identity, forming part of the Eco- Planning in Evaluation Training nomic Commission for Latin America and the Caribbean The Latin American and Caribbean Institute for Economic (ECLAC). Created in 1962, its mission is to support gov- and Social Planning (Institututo Latinoamericano y del Ca- ernments in the region in the area of governance, through 109 M&E SYSTEMS AND M&E TOOLS research, consultancy, and, in particular, training. The in- in Latin America and the Caribbean including people from stitute has worked since late 2002 as a central training body private sector institutions and civil society. of ECLAC and, in that role, it helps to disseminate the The training courses offered by ILPES/ECLAC revolve work of the ECLAC system, particularly that of an insti- around specialized topics not usually addressed by academ- tutional nature. ic programs in Latin America and the Caribbean. They To perform its role as the central organ of ECLAC train- combine theory with practice and systematize the experi- ing, ILPES has at the United Nations headquarters in San- ences of countries in the region. Moreover, the internation- tiago, Chile, three fully equipped classrooms and 10 work- al nature of many of the courses and teamwork offer par- shop rooms with 22 computers and six printers available to ticipants an interesting exchange of views and problems students. In addition, the auditorium, with a seating capac- connected with Latin American economies. ity of 55, is equipped with facilities for simultaneous inter- Each year several courses are open to international par- pretation. ILPES has all the necessary audiovisual equip- ticipants, as well as national and subnational courses. The ment, such as video projectors and facilities, to conduct language of instruction is Spanish and the courses normal- video conferences. ECLAC has three large-capacity confer- ly last between one and five weeks. ence rooms that can be used particularly for the opening The teaching staff consists mainly of ILPES experts and and closing sessions of a course. To perform its job, ILPES technicians from various substantive divisions of ECLAC. is structured around two thematic areas and a training unit. The ILPES takes pains, in turn, to recruit senior staff from The thematic areas are Budgetary Policies and Governance governments, public agencies, universities, academia, and the and Local and Regional Management. The main topics of private sector to teach or give lectures. In 2009, ILPES orga- each of these areas are as follows: nized and conducted international courses on the following: Budgetary Policies and Governance • Tax estimates • Fiscal and budgetary policy • Evaluation of government management and pro- • Socioeconomic evaluation and project manage- grams ment • Evaluation of impact of programs to overcome pov- • Evaluation of public programs and management erty Local and Regional Management • Management of project implementation • Regional development and territorial competitive- • Strategic management of local and regional devel- ness opment • Decentralization • Strategic planning and construction of perfor- • Local and regional management mance indicators • Planning and prospects. • Macroeconomic policies and public finance • Budgetary policies and public management by re- The Training Unit provides all the logistical and ad- sults ministrative support necessary to conduct the various • Preparation and evaluation of public investment courses organized by the institute at its Santiago head- projects quarters and in cities across the continent, either in part- nership with other agencies or national, regional, or local As for national courses within the framework of agree- governments. ments with national or subnational governments of mem- To illustrate the efforts made in training by the insti- ber countries, ILPES has carried out such activities in al- tute in its 45 years of existence, it has taught various types most all countries of the region. In some cases, it was a of courses that have involved approximately 24,000 partic- single course or a few courses, but in others, it has devel- ipants mainly from the public sector in all the countries in oped extensive training programs for public sector officials, the region. It also has had the benefit of significant partic- sometimes with more than 100 courses. This is the case ipation by teachers from universities and academic centers of training programs held in Chile (1996–2000), Colom- 110 S ession 4 : M onitoring and E valuation S ystems bia (1989–1990), and Mexico (ongoing since 2007). This gram in evaluation to have the desired impact. These are year, courses have been conducted within the framework of as follows: agreements with national governments in Argentina, Bra- zil, Costa Rica, Mexico, and Peru. • Training should be theoretical and practical. For example, in Mexico’s case, the latest agreement is in- • There should be courses at different levels of spe- tended to strengthen the technical capabilities of those re- cialization. sponsible both at the federal and state levels for the design, • Training should be inserted in a process. implementation, monitoring, and evaluation of the matri- • Training should include managers. ces for performance indicators to ensure that these matri- • Training should be ongoing. ces are consistent, that they do not show any incongruence in their logic, and that their indicators are constructed with Theoretical-practical training the highest possible technical rigor and supported by quality Students quickly forget what they hear, but learn from what information. The products of this agreement are as follows: they do. All the training activities undertaken by the ILPES therefore combine theory and practice. Generally, morn- • Workshop courses for improvement in construc- ings are devoted to presentation of the theoretical aspects of tion of matrices for indicators of federal programs. the issue being addressed in the course, either by ILPES or Designed for staff of federal government programs ECLAC experts, public sector professionals, or consultants. and agencies, and at officials of state programs. In the afternoon, students work in groups to apply to real • Courses on awareness of the results-indicator ma- cases what they have been taught in the morning. trix for senior officials of federal programs and fed- In the formation of these working groups, the great- eral entities. est possible diversity of professions, diversity of nationali- • Technical assistance for improvement of indicator ties in international courses, and gender balance should be matrices and their indicators. Aimed at federal and sought. This greatly enriches the learning process for the state programs. important exchange of personal experiences that occurs. • Workshop courses for improving the matrix in- It helps to achieve better results in workshop activities be- dicators of federal programs. Designed for feder- cause participants with different educations provide differ- al government officials responsible for the design, ent perspectives on the problem to be solved or issues to monitoring, and evaluation (or related areas) of evaluate. In general, working groups should have no more performance indicators of budgetary programs. than five or six members, because when a group is larger, some members probably will not keep pace with the rest The ILPES long-distance education program, provides (social pressure decreases when responsibility in a more nu- online training to students who cannot go to training ven- merous group is diluted). ues in person. Since 2005, more than 1,500 people have Finally, practical experience has taught us that it is bet- taken the various types of courses offered, including par- ter for the learning process to work with real projects or ticipants from Latin America, India, South Africa, Bosnia, programs than with cases prefabricated for the course. Pre- and Haiti, among others. fabricated cases facilitate the workshop, but they cannot This approach uses the Internet, and the course design is prepare students for the reality to be faced on returning to based on the “learning by doing” approach, which consid- their jobs and having to evaluate a project. Instead, work- ers the practical application of content delivered. Current- ing with real projects or programs teaches students how to ly, the two courses are “Basic elements of local and regional solve the difficulties they will face, such as incomplete or development” and “Logic for the formulation of develop- incorrect information. ment projects.” Preparations are under way for other cours- es that refer to specific ILPES-related topics. The institute’s Courses with different levels of expertise extensive experience in training courses and programs al- Countries that have been most successful in the creation lows us to give some recommendations for a training pro- and operation of national public investment systems have 111 M&E SYSTEMS AND M&E TOOLS developed large-scale training programs in preparation and In short, an evaluation-training program is justified on- evaluation of projects, procedures, and the system’s use of ly if inserted into a process of modernization of public ad- information about projects. This normally will include a ministration in terms of investment management. It should series of courses at basic, intermediate, and advanced levels, include methodological developments and procedural and requiring a large number of basic courses, fewer intermedi- even legal and institutional reforms. ate courses, and a few advanced courses. The goal of basic-level courses is to provide trainees with Training authorities and managers a culture of project evaluation. This culture aims to establish Having spent many hours carrying out a socioeconomic a common language, to know the objectives of project eval- evaluation of a project or obtaining the results of a pro- uation and their value, and to create the skills to implement gram, only to find that whoever runs the institution does simple methods of formulating and evaluating projects. not use the evaluation results, or worse, demands a dif- The intermediate-level courses continue to train train- ferent result, can be extremely frustrating for any public ers in the development and evaluation of projects, as well official. So if you want to create a culture of evaluation as technical experts in project evaluation. Their role is to in the public sector, it is essential to consider factoring provide basic-level training and technical advice within the into the design of the training program lectures for se- public sector, thus contributing to the sustainability of the nior directors and authorities who will receive the evalu- SNIP and quality of the results obtained. ation results. Finally, advanced training, often delivered through part- For example, when Mexico began to implement re- nerships with universities, aims to form the top layer of sys- sults-based budgeting and the performance evaluation tem professionals, who will take on the responsibility of de- system and decided to use the logical framework meth- veloping the investment system and fulfill duties as trainers odology to evaluate the results of federal programs, talks or experts in evaluation. were scheduled in the conference room of the Ministry Given the high staff turnover that often occurs in the of Finance for the directors of programs and government public sector, these training programs should be perma- ministries in charge. More than 800 directors attended nent. To this end, the entity responsible for organizing these two-hour lectures. This allowed them to have a basic the investment system should be concerned with organiz- knowledge of the tool to use, thereby facilitating dialogue ing courses at the different levels, either directly or through with the technicians who would later prepare the matrices agreements with national and international entities. Un- for the programs. doubtedly, it will require a large number of basic courses, some intermediate, and a few advanced level courses. Ongoing training In most countries in the region, there is a high turnover Training inserted in a process of public officials, due to their seeking better wages in the To cause a significant and lasting impact, training in eval- private sector, movements between government ministries, uation should be inserted in a process of reform and mod- or changes that occur with a new government administra- ernization of the state. If the student returns to the job and tion. This results in a significant drain of personnel trained is not faced with the need to apply the acquired knowledge, in evaluating projects and programs. To compensate for he or she is likely to forget the material. In other words, this drain, training programs should be permanent. In fact, having an evaluation training program that is not accompa- the most successful experiences in relation to public in- nied by reforms in resource allocation procedures to make vestment systems have been accompanied by major pro- evaluations compulsory makes no sense. In addition, to en- grams and training that has been sustained for decades. sure that courses taught in the training program meet the Such is the case, for example, of the National Investment requirements stipulated, it is essential to have an adequate System of Chile. The Ministry of Planning, responsible for methodological backup. This must be found in manuals its operation has been concerned about training techni- that serve as teaching material in courses and as guides for cians with various levels of expertise in evaluation. So it has practical evaluation work. held courses on its own account, had an agreement with 112 S ession 4 : M onitoring and E valuation S ystems ILPES for several years under which more than 20 cours- Strengthening Monitoring and Evaluation es were held annually, and has signed agreements with var- Capacities through the Design and ious universities to train public officials at intermediate or Implementation of National Results-Based advanced levels. Monitoring and Evaluation Systems: Lessons These training programs, which usually begin under Learned in Africa, Asia, and Latin America agreements with international agencies that finance the creation of the system or support its operation, cannot Abstract continue to rely on external support always, either finan- Strengthening monitoring and evaluation (M&E) capac- cially or academically. It therefore is essential to involve ities is more effective and efficient if a midterm training universities in the process, so that they can offer courses plan is designed to meet internal monitoring needs for pi- on evaluation, which are conducted in partnership with loting public programs and the objective evaluation of re- the government or included in the curricula of the cours- sults and performance of the public sector. This plan should es they teach. include various components such as more decentralized, ef- Universities are a key source of knowledge and experts fective, and diverse training; complementary technical as- in the field of evaluation. Whenever ILPES has organized sistance; support for information systems; sensitization large-scale training programs in different countries, univer- discussions; policy discussion and exchange platforms be- sity professors have been part of teams of consultants who tween professionals; institutional support for national, sus- have been allowed to develop them. Universities also are a tainable M&E units; development of external evaluation source for creating studies and knowledge on evaluation of capacity; and a culture of accountability. It is a road map projects and programs. defined in a coordinated fashion at the national, sectoral, The evaluation courses that universities offer tradi- and local level to progress in a timely, modular fashion in tionally have been limited to diplomas or master’s degrees various M&E systems and to contribute to the profession- lasting one or more years. Conversely, a massive training alization of both M&E. In a mature national M&E system program aimed at public officials requires the enactment with information-rich and learning organizations, the bor- of numerous short courses (one to four weeks). This re- ders between M&E persist but in a complementary fashion quires the public sector to engage with universities, sign- with overlapping areas. ing deals to offer the courses that government needs. Ex- amples of this type of agreement can be found in Chile Context and Colombia. M&E is one of the pillars in the implementation of Man- An advantage of universities in the organization of agement for Results (MR), also known as Management for courses by international bodies is that they can award di- Development Results (MfDR), in developing countries’ plomas that are valid for the purposes of government career public sectors. The 2005 Paris Declaration on the effective- development. The courses improve résumés and enable at- ness of aid highlighted the importance of improving the tendees to apply for other positions in the public sector. In- M&E of interventions for development (High Level Fo- ternational bodies such as ILPES cannot provide such di- rum 2005). The Accra Action Agenda of 2008 reinforced plomas, but have the advantage of providing a broader view the commitment of developing countries and donors to of the subject and the experience gained in other countries. demonstrating results through increased accountability Therefore a significant synergy exists in partnerships be- and transparency (Third High Level Forum 2008). De- tween international agencies and academia in conducting veloping countries pledged to improve “the quality of training programs in evaluation. policy design, its implementation and evaluation by im- proving information systems.” Developing countries and Eduardo Aldunate donors have agreed to work to develop cost-effective in- Expert, Budgeting Policies and struments for MR to evaluate the impact of development Public Management Area CEPAL, Chile policies and adjust them if necessary. The MfDR team has assembled an MfDR Capacity Scan tool kit to help assess 113 M&E SYSTEMS AND M&E TOOLS progress in national MfDR capacity, including M&E ca- • Improve quantity and quality in the supply of pacity (OECD/DAC 2009). Strengthening evaluation ca- M&E products pacity building (ECB) and improving the capacity for in- • Ensure the cost-efficiency relationship of M&E ternal monitoring of public programs has become a priority products on the development agenda. In a context of limited public • Promote the sustainability of M&E systems and in- resources exacerbated by the financial crisis, the importance stitutional mechanisms of improving national M&E capacities with suitable, inex- pensive, sustainable approaches cannot be overemphasized. Establishing ECB as part of the implementation of We should recall two basic considerations. First, the do- the M&E System main of M&E covers a variety of subareas. Second, ECB M&E training must not be given simply for the sake of covers far more than just training. ECB entails reinforcing training but rather should be considered a catalyst for im- or constructing M&E systems, particularly national sys- plementing M&E systems for results. Incorporating ECB tems, so that M&E is regularly conducted and used by the into the current implementation and improvement of the countries and organizations (OECD 2004). M&E system for results gives meaning to ECB’s final pur- Several ECB initiatives have been implemented as a pose, which is contributing to better public policies and component of development projects or by specialized in- programs. Conversely, making progress in the implemen- stitutions (World Bank Institute and Institute for Develop- tation of the M&E system for results provides an opportu- ment in Economics and Administration, IDEA)� and spe- nity to learn by doing, thereby strengthening and building cific programs (International Programme for Development capacities within a given organization. Evaluation Training, IPDET, Paris 21). Unfortunately, few The design and implementation of an M&E system ECB initiatives have been carefully evaluated. The few sig- should be progressive as part of various M&E compo- nificant evaluations carried out, such as the self-evaluation nents that are gradually implemented. There are no “off- of ECB, conducted by the World Bank Operations Eval- the-shelf ” solutions and the approach should be adapted uation Department (OED) in 2004 (World Bank 2004) to the needs and particular situation of the existing M&E. and the evaluation of training in World Bank Projects and The now-standard reference in this field by Kusek and Rist the World Bank Institute by the World Bank Independent (2004) is showing the way. Evaluation Group (IEG) in 2007 have seriously questioned Tailor-made by taking into account the institution- the effectiveness and impact of existing ECB initiatives, al and political context, needs analysis, existing M&E sys- particularly those based exclusively on training. tems, and available resources, this plan has been used in This document is based on the experience of several ex- various assignments conducted by experts from the IDEA perts in the IDEA network in supporting institutions in Institute and tested to provide a relevant, effective frame- developing countries, programs, and projects in the im- work for ECB. A good example of this is the design and plementation of results-based management (RBM) in Af- implementation of the Results-Based Monitoring System rica, Asia, and Latin America. It outlines lessons learned (RBMS) at the Secretariat of Social Development (SEDE- and identifies some of the best practices in capacity build- SOL). The good practices drawn from this experience in- ing in M&E. cluded the following: Developing ECB strategies based on lessons • Close involvement by the organization’s person- learned and good practices nel in the implementation of the M&E system that The following ECB strategies should be regarded as com- boosts know-how within the organization and en- plementary basic components. They pursue four basic ob- sures the sustainability of the system. Staff involve- jectives to obtain better results. ment in the process of implementing RBMS at SEDESOL not only served as a group of resources • Highlight the importance of M&E for decision mak- for other programs in the secretariat but also for de- ers to encourage their demand for M&E products veloping the National M&E System, currently be- 114 S ession 4 : M onitoring and E valuation S ystems ing designed and implemented by the Civil Service semination of the information. For example, the di- Secretariat with the support of the National Eval- agnosis conducted by SEDESOL in 2004 identified uation Council for Social Development Policy and 33 information systems in the organization. These the Secretariat of Finance and Public Credit. systems did not always provide relevant information • Participation by all the actors at all stages of the for strategic decision making, or they were difficult design and implementation process of the M&E to interpret because they did not always meet the system to guarantee their support and ensure the needs of the users of this information. success of the project, thereby achieving greater sus- • A diagnosis should try to evaluate the organization’s tainability. M&E capacity and build an M&E training plan • Designing and implementing the M&E system as a central component of the project for imple- through a pilot phase in 4 of the 27 programs to menting the M&E system to ensure the personnel’s learn from the experience of boosting the organiza- participation in the entire process, maximizing the tion’s capacity to expand the system, which would number of “champions,” and limiting the number therefore result in less dependence on external re- of potential “opponents.” sources. • Program personnel involved in the design and im- • An ECB approach based on MR principles, includ- plementation of an M&E system sometimes may ing focusing on results and a process that involves see the new system as a source of additional respon- evaluating needs, formulating strategies to boost sibilities and work that may cause frustration or in- M&E capacities, designing and implementing ac- difference. They therefore should offer incentives tion plans to boost capacities in M&E, and M&E to guarantee the full participation of those involved of the implementation of strategies and plans of ac- and of the personnel throughout the process. tion to boost M&E capacities and enhance train- • Boosting M&E capacity requires bearing in mind ing effectiveness. the capacity of the other pillars of MR41. Problems found at the M&E level often result from a limit- The lessons learned include the following: ed design and implementation in other pillars of MR. At SEDESOL, the logic of the logical frame- • It is important to clearly understand the needs of works was reviewed; performance indicators were se- all those involved in terms of data, information lected on the basis Clear, Relevant, Economic, Ad- systems, reports, and the like. Often the clients equate, and Monitorable (CREAM+) criteria; and themselves do not know exactly what their specif- goals were based on Specific, Measurable, Achiev- ic needs are. They regard information as a free as- able, Relevant, and Time-bound (SMART) criteria. set and have infinite needs. Or else they have a nar- In many countries, M&E is not seriously considered row perspective based on specific short-term needs by many actors until the results have consequences linked to their role in the organization, with little for future budget assignments, which means linking concern for the results. It is the advisor’s responsi- the M&E process to the budget preparation process. bility to help clients understand their specific needs • M&E goes beyond technical skills. To serve as and, where necessary, suggest different options as a agents of change, M&E officers should possess lead- starting point. ership and communication skills to convince their • Acquiring a good knowledge of the existing M&E colleagues in technical departments as well as cabi- system can be a challenge, particularly in middle- net ministers of their own interest in implementing income countries, where the ministries have devel- RBM and insist on carrying out the reforms. oped a variety of M&E mechanisms and informa- tion systems over the years. These systems are spread Clearly identify the expected results of ECB over various levels of the organization, without The expected results of ECB should be clearly identified and much coordination, harmonization, or even dis- a consensus should be reached between interested parties. 115 M&E SYSTEMS AND M&E TOOLS Performance indicators obviously have to go beyond results training. Based on the experience of the IDEA Institute, we to include effects and, whenever they can be measured, im- agree with IEG that pacts. What matters in the end is that the M&E reports and control boards are accessible to strategic (cabinet) and op- One of the most powerful determinants of train- erational (program and project) decision makers and used ing success is the organizational context in which for policy design, program management, and determining training is carried out. For training to be success- future budget assignments. Decision makers often are con- ful, participants must have the resources and in- fused about results. In Senegal, several training workshops centives to implement the skills and knowledge clarified what they meant by results at various levels. For ex- they have acquired. When these resources and in- ample, the director of primary and secondary education re- centives are not present before training, training alized that the classrooms built and the teachers trained were must be accompanied by suitably designed inter- the result of building and teacher training projects, respec- ventions in order to take organizational and insti- tively, but that they also were input for the most strategic tutional limitations into account. (IEG 2008) level. But, above all, they should be interested in graduation, dropout rates, and double costs, which were effect indica- Experience has shown that demand-driven capacity-build- tors, as well as in the success of their former students at the ing activities are usually more sustainable and effective. subsequent education level, graduate placement rates, and They tend to respond to more specific felt needs, and the the development of self-employment on the labor market. participants in these training activities tend to be more mo- Key results must be accessible to other interested par- tivated to learn and then to apply what they have learned. ties for the sake of accountability. In Senegal, a great deal of A demand-oriented approach obviously entails designing a M&E information was available, but with little coordina- training program on the basis of the training needs iden- tion and validation from the Head Office of Statistics. This tified in a participatory fashion during the diagnosis and led to frustration among government employees and civil preparation evaluation. Table 16 suggests a structured ap- society, which had limited access to these results and the re- proach for measuring performance gaps and evaluating ports. A new National Agency for Statistics and Demogra- training needs that has been successfully used by the IDEA phy was set up, with greater autonomy and power, and with institute on several occasions. an information system on national statistics in a Web en- ��This also involves the careful selection of participants vironment. Using the results chain as an analytical frame- for each type of training activity. An ECB plan should con- work helped identify four levels of training evaluation with sider various types of training activities designed to match the corresponding means of verification. various categories of participants, objectives, and limita- tions. Strategic-level participants want to focus on the over- Key factors for effective training view and have little time. M&E professionals want a deeper Since training is an important component of an ECB strat- understanding of concepts and a better command of the in- egy, attention should be focused on key factors for effective struments and techniques and have more time. The IDEA TABLE 16: Training Evaluation Levels Level Measures Common means of verification Level 1 Participants’ satisfaction End of course questionnaires for participants. Level 2 Learning products Post tests, compared with pretests. Level 3 Performance of change of Multiple, including observation, and interviews with participants, colleagues, and effects supervisors. Level 4 Organizational impact/ Multiple, including comparisons with organizational baseline, performance measure, results surveys, and interviews with key informants. Source: IEG 2007. 116 S ession 4 : M onitoring and E valuation S ystems Institute has therefore organized one- or two-day sensitiza- cusing on practical training. This does not mean, however, tion and high-level discussion seminars with a small num- that training is transformed into a collection of anecdotes. ber of cabinet members, delegates, and high-ranking do- There must be a solid analytical framework, like the 10-step nors. For professionals with sufficient time and money who guide. Participants must learn how to use evaluation instru- require official recognition of their skills in the form of an ments professionally to establish their relevance and credi- U.S. university diploma, Laval University has organized a bility. However, practical training methods use many exam- Diploma Course in Management for Results, which in- ples that are presented and discussed through debates with cludes an M&E certificate. To make them more accessi- professionals with experience. Success stories are discussed ble, these courses are offered in the three languages (Eng- in working sessions, which helps participants develop their lish, French, and Spanish) and in various locations (Dakar, own M&E systems and tools during training. Some groups Dar es Salaam, and Panama City). For professionals with take study trips. For example, the Vietnam delegation trav- limited time and money, the institute has organized sever- eled to Canada to learn more about measuring munici- al tailor-made technical workshops at its regional centers or pal performance; and the Benin delegation visited Panama in the country interested in specific M&E issues based on to learn more about large project M&E. Complementa- expressed demand. What is astonishing is that most of the ry training strategies include participation in practice com- participants in these workshops secure their own financing munities and meetings with international associations such and demand is growing, which proves that a market-orient- as the Global IDEAS Conference. ed approach can work. Selecting instructors is another key factor in effective Training a critical mass of government officials in M&E training. Any training process is based on the instructor’s has several advantages for a government. It offsets staff ro- ability to elicit the participants’ interest and to encourage tation and burnout by maintaining a minimum of ECB in them to keep listening and adopt the right mental struc- M&E units. This creates a common approach among par- ture to accept new concepts, tools, and approaches, which ticipants who share the same vision, language, and tools will effectively transfer their know-how to learners. This re- and who therefore can communicate better with each oth- quires a rare species of instructor who combines a high level er in the community of practices. In several countries, the of education with an analytical framework and related con- government asked each ministry to assign part of the bud- cepts and has a good grasp of evaluation instruments. This get to building capacities in MR, particularly in M&E and person has extensive field experience in a variety of institu- to send their executives on the same training course to cre- tional and cultural contexts so that he or she understands ate this critical mass and to create momentum in terms of the content, has the relevant teaching skills, and, lastly, de- mobilization toward reforms for MR. velops the human skills required to be able to handle high- A key factor in effective training is choosing a teaching level participants with a variety of characteristics. Academ- strategy that has been adapted to the context, the partici- ics tend to be, well, academics. Researchers tend to focus pants, and the expected objectives. The institute’s experi- on advanced technical subtleties of their own interest and ence suggests that participants appreciate several teaching the few peers they do have are too advanced to be relevant approaches, a minimum of readings, and a maximum of to most M&E professionals working in the context of a de- participatory methods. Instructors, therefore, not only re- veloping country. Long-term consultants and practitioners quire M&E skills and field experience, but also teaching tend to talk excitedly about their own valuable experienc- skills. The institute, with Laval University, uses a skills ap- es but may be out of date as regards methodologies and in- proach whereby a group of skills are identified for each tar- formation and communication technologies (ICT). Civil get training group, together with the current and target lev- servants and national and regional advisors (1) may have el of command for each skill. A training plan subsequently problems distancing themselves from the specificities of the is designed to establish links between the training modules context and the tools they know, (2) are not necessarily ex- and their contribution to improving skills. perts in a variety of modern M&E instruments and ap- In terms of training methods, executive training re- proaches, or(3) do not necessarily have the teaching skills quires distancing oneself from theoretical training and fo- required of a professional instructor. So, the second option 117 M&E SYSTEMS AND M&E TOOLS is sometimes to use a team of complementary instructors years) ECB in which the institute uses a combination of and ensure that they work together so that training is not training, technical assistance, and support to help a nation- just an unsystematic collection of uncoordinated, individ- al team improve data quality and information systems. Ac- ual contributions. tivities are organized according to a flexible road map that A key challenge to the M&E community is how to in- includes the production of intermediate products required crease access to quality M&E training. For example, in by the national team, such as progress in policy matrix ob- Canada, the International Programme for Development jectives to be prepared for the World Bank on a six-month Education Training (IPDET) annually trains approximate- basis; review missions, national development plans, or en- ly 275 participants in English. This institute trains M&E terprise research planning evaluation; impact evaluation professionals who will act as champions of change when studies of major programs; annual performance reviews they return to their jobs. Most M&E practitioners are un- aligned with the next budget preparation; and processes to able to receive IPDET training in Canada. An initial solu- update the midterm fiscal framework. The program usual- tion would be to reproduce IPDET in various localities and ly begins with training, followed by technical support mis- languages. Two similar initiatives have been carried out: sions and subsequently in-service training programs. SHIPDET in Shanghai, offered twice a year, and Czechdet in Prague, offered on an occasional basis, both in English. Improving M&E information and knowledge IDEAS and the IDEAS Institute are discussing the possibil- management systems ity of organizing a DIPDET in Dakar in French for Fran- Information systems can be an excellent entry point for cophone Africa and a PIPDET in Panama in Spanish for ECB. They focus on data-gathering, processing and anal- Latin America. ysis and the dissemination and use of information as well Another solution would be to design a Trainer Train- as data quality. All these elements have enormous implica- ing Program. The institute recently used this formula in tions to permit full evaluations and their credibility. the Democratic Republic of Congo. During the first two- A good practice is a series of new applications that have week phase, IDEA experts trained 50 participants. The best been developed by various organizations and firms such as were selected for a second phase of Trainer Training, during automated control panels. The IDEA Institute has devel- which they expanded their knowledge and developed their oped control panels at the sectoral level (such as the Sec- teaching skills. The recently trained instructors then gave retariat of Social Development in Mexico) and at the pro- the following training courses with the technical support gram level (such as the National Land Administration and advice of the IDEA experts. The result was that in six Program in Panama). It has developed an attractive com- weeks, 100 participants and five instructors were trained, puter solution called e@satisfaction that provides a reliable who are now qualified to repeat the training at both a cen- means of measuring, processing, and analyzing the quality tral and a decentralized level. of the services provided on the basis of online surveys and A third solution is to build alliances with regional and of obtaining rapid feedback on the Web in a user-friendly national training institutions to develop executive training way to enable decision makers to conduct their programs programs leading to certification. The IDEA institute has or projects better. developed this kind of alliance with the eight countries of Another good practice is to establish knowledge man- the West African Economic and Monetary Union (Sene- agement and learning units in the organizations. Many na- gal, Guinea Bissau, Cote d’Ivoire, Mali, Burkina Faso, To- tional and international organizations have realized that go, Benin, and Niger) to train public finance officers at the experiences, lessons learned, and good practices were not École Nationale des Régies Financières, in Ougadougou. It al- shared among members of the same organization and were so provides training in project management at the Tanzania lost once these members retired, were transferred, or left the Public Service College in Dar es Salaam organization. These units play a crucial role in the preserva- Another factor in effective training is follow-up. In tion and dissemination of institutional memory by gather- IDEA’s experience, the most effective, sustainable strategy ing available information on M&E, correctly documenting is to design and implement a medium-term (two to three metadata, storing data safely, and providing selective access 118 S ession 4 : M onitoring and E valuation S ystems to authorized users. They can make it easier to share expe- telligence tools to design a user-friendly M&E in- riences among internal and external experts, participate in formation system. Nevertheless, M&E specialists impact evaluation studies on large programs to learn about must focus on the objective, namely, that the sys- failures and success, orient future policies and programs, tem should work, and be robust, profitable, and and train the organization’s personnel in M&E methods regularly provide a minimum amount of the infor- and their results. In their new mandate, these units can take mation requested by decision makers. There are too advantage of advances in ICT and electronic documenta- many horror stories and the world is full of inef- tion management systems (EDMS). The IDEA Institute fective, costly white elephants and database cem- has helped implement these systems in various institutions eteries. It is useless to have a complex information to protect an enormous amount of administrative and tech- system with the latest technology if it is not used nical documentation with user-friendly retrieval methods. for the program’s everyday activities or in decision The following are lessons drawn from various experiences making. Computerizing a poor information system at the IDEA Institute: only achieves faster GIGO (Garbage In, Garbage Out). It is more useful to begin on a small scale but • In several large public bureaucracies, information slowly and surely update and improve the M&E in- is scattered, meaning that it is not always easy to formation system. know where to find the information, who has it, or how it is used. It is important to identify all infor- Sensitization and dialogue forums on policies and mation sources, whether they include sophisticated professional associations information systems, Excel spreadsheets, or hand- The IDEA Institute has been associated with the establish- written reports. This information can be identified ment, participation, and strengthening of various types of through interviews but also may appear as summa- platforms and mechanisms for sharing experience that con- ries in the design and implementation process of tribute to ECB. At the country level, the use of the Joint the M&E system. Progress Review (JPR) has increased between the govern- • The point should not only be to try to determine ment and donors, sectoral thematic groups, and forums where the information is found but also how this to create a common strategic vision. In Vietnam, the JPR information was gathered or calculated. This is ex- mechanism implemented for the discussion of the 135 Pro- tremely important in ensuring the quality and in- gram, a large-scale program to alleviate poverty, has con- tegrity of the information to be incorporated in- tributed to a better policy dialogue between government, to the system. Otherwise, the information might making the latter more effective and efficient. In Cambo- be outdated or, worse, incorrect. The Millennium dia, the Poverty Forum proposed by the institute helped Challenge Account (a U.S. initiative administered share a great deal of scattered information on poverty and by the Millennium Challenge Corporation) spon- contributed to a policy dialogue between public organiza- sored data quality evaluations in various countries. tions and a vibrant NGO community. Sensitization ini- This initiative has been welcome, as have the efforts tiatives such as brief seminars or cabinet retreats may in- to revitalize and improve national statistical systems crease acceptance at a high level and create a momentum within the framework of the 21 initiatives of the for the implementation of M&E-MR under national lead- Paris declaration (the Paris 21). ership. The organization of top-level conferences at both • An information system for monitoring and evalu- the national and regional level, involving a combination ation–managing for results (M&E-MR) should be of ministers, legislators, high-level technical personnel, designed by computer experts under the supervi- and consultants may help to compare national experienc- sion of M&E specialists. M&E specialists should es, carry out benchmarking, and orient participants toward listen to computer specialists who are aware of the a common understanding and strategy. Several communi- latest technological developments particularly as re- ties of M&E practices have arisen at the national, region- gards database management terms and business in- al, and international level, with varying degrees of success 119 M&E SYSTEMS AND M&E TOOLS and sustainability, which, if properly handled, can certain- Ministry, established with approximately 20 local young ly contribute, in a profitable way, to sharing lessons learned professionals, each acting as a focal point with a sectoral and best practices. Last but not least, professional evalua- ministry. This arrangement achieved a reasonable balance tion associations have a significant role to play in ECB, and between the need to produce M&E products in the short IDEAS (as the only international association for develop- term and the desire to increase capacity. ment evaluation) has a unique niche and a contribution to make to sharing knowledge through geographic regions External evaluation and accountability and linguistic and cultural zones. In addition to the key role of M&E units within the sectoral ministries, the institutional device should include Building sustainable M&E units other public institutions as well as others from the private Beyond more effective training, improvements to the in- sector and civil society. In addition to organizations for ex formation system, sensitization, policy dialogue forums, ante and ex post internal control, an external public evalu- and professional associations, ECB seeks to build sustain- ation organization should have the legal independence and able M&E units. The term “unit” is used in a generic sense the legal power to obtain needed information as well as the and may cover a variety of international systems according human and financial resources to produce quality work. to the country and organization involved. The experience of several industrial countries such as Can- To improve the performance and sustainability of M&E ada has proved its significant contribution to the efficiency units, the IDEA Institute has encouraged management ap- and effectiveness of public spending. Behind the organiza- proaches, all of which are designed to improve institution- tion of powerful auditing, the culture of results develops on al performance and results. The National Institute of Stan- both the demand and the supply side through the imple- dards and Technology (NIST) Balridge National Quality mentation of a management-by-results system in the public Program (National Institute of Standards and Technolo- sector with performance contracts, and through the follow- gy 2009; Balridge National Quality Program 2009), the ing: (1) the public accountability of the results of each actor equivalent in Quebec, called the Qualimetre (Mouvement in comparison with his or her goals and commitments, (2) Québécois de la Qualité 2009), and the Balanced Scorecard, the dissemination through the media and members of civil initially developed for the private sector, were all adapted to society to the population of the follow-up provided by jus- the public sector (Kaplan 1999). tice when necessary, and (3) the development of evaluation When applied to a public organization as an M&E capacity by the private sector and universities, and the like. unit, institutional support focuses on the mission and tar- The demand from both the private sector and civil society get groups of the M&E unit and its balanced consideration for accountability from the public sector and the demon- of human resources, financial resources, and processes of stration of the public value created goes hand in hand with the unit. Its links have proved to be extremely useful if the improving the quantity and quality of accountability. aim is to develop the institutional sustainability. In a mature M&E system, the limits of the two pillars In fragile states, an important consideration for institu- evolve. This means, in other words, internal monitoring for tional sustainability involves whether the M&E unit is in- piloting programs and external evaluation for accountabili- side or outside the public sector. If incentives and morale ty. The incorporation of a results culture into public organi- are low, for example, implementing the unit within minis- zations leads to the development of internal evaluation per- tries will be counterproductive. Implementing M&E units ceived as a tool for improvement and one that will develop within external project implementation units (PIUs) is not an information-rich and learning organization. In addition a sustainable approach and could seriously undermine sus- to the classic monitoring reports on physical and financial tainable national institutions by attracting the few compe- performance and base, intermediate, and final evaluations, tent human resources available in the public sector through other tools have been developed such as results monitoring, higher salaries and better working conditions. One ap- control panels and rapid warning systems, annual perfor- proach used by the Institute in Cambodia has been to sup- mance reports, performance auditorships, qualitative and port a poverty monitoring technical unit in the Planning quantitative or combined evaluations (Q2 approach), and 120 S ession 4 : M onitoring and E valuation S ystems the like. The natural framework for progressively includ- ing all these components, modules, and tools is the nation- Session 4 Notes 37 al M&E system that is constructed as a result of a coordi- The meeting was sponsored by the evaluation network of the OECD-DAC. The evaluation principles were included in nated, midterm M&E plan. DAC Principles for Effective Aid (published in 1992). Many of the principles discussed then also had been discussed at and Conclusion were later integrated into international and regional evalua- Building M&E capacities involves several challenges. How- tion associations, such as the American and African Evalua- ever, experience based on lessons learned and good practic- tion Associations (AEA and AFREA, respectively), the Eval- es shows that it is possible to achieve significant progress by uation Cooperation Group of the multilateral development banks, or norms of the United Nations Evaluation Group.The using various mechanisms and tools in a complementary underlying principles are the same, although the terms used fashion. Training should be decentralized and made more may not be identical. effective and diversified. Investing in complementary tech- 38 The UNIDO report defined “institutional capability” to include nical assistance, support for information systems, and cer- services to be performed, demand for these services, capabilities tain sensitization and policy discussion forums and profes- (skills in necessary quantity and quality) to meet demand, hard- sional exchange platforms as part of a strategy and a road ware (including facilities, premises, and instruments), method- ology, legislation, and management and coordination functions. map defined at the national, sectoral, and local level can These features, considered necessary for an institution to func- help the M&E system to make a great deal of progress to- tion, were integrated into project design training and the annu- ward effectiveness and sustainability and contribute to pro- al evaluation reports of UNIDO at the time. fessionalizing the evaluator’s work. This process of continu- 39 The work program and budget can be found on http://www.if- ous improvement must be perceived as a key element in the ad.org/evaluation/whatwedo/wp/index.htm. 40 overall reform of the public sector toward MfDR. This re- The authors thank their colleagues Juan Abreu and Sylvain quires strong leadership and greater commitment from all Lariviere for their suggestions and comments on this docu- ment. the actors in developing countries and donors to achieve a 41 IDB/PRODEV considers the following five pillars of MR/Mf- culture of results. DR: Strategic planning,Budgeting for Results, Public Finance Management, Program and Project Management, and M&E Gilles Clotteau (Garcia López 2008). The OECD/DAC/MfDR considers the Marie-Helene Boily following five pillars of MfDR: Leadership; Evaluation and Sana Darboe monitoring; Accountability and Association; Planning and Budgeting and Statistics (MfDR 2009) Frederic Martin42 42 For more information, see IDEA‘s Web site, www.idea-inter- national.org. 121 �Session 5: Institutional Arrangements and Related Policies to Ensure the Quality, Access, and Use of Performance Information Challenges in Monitoring and Evaluation: Essentially, these concepts provide a theoretical frame- An Opportunity to Consolidate the M&E Systems work within which the quality of an indicator can be veri- The litmus test of the success of a monitoring and evalua- fied against the given criteria. But the mechanism of how to tion (M&E) system lies in the quality of indicators that are do it remains a challenge. In the context of public admin- used to capture a dimension or an attribute to show the re- istration, the main actors associated with this mechanism, sults in the form of an assessment of the performance in a very broadly, are the state, market, and civil society. The particular aspect of governance or public service delivery. state provides a political, legal, and economic framework, The classical view of the quality of indicators in M&E sys- whereas the market creates opportunities for people, and tems is to ensure that they are objective and adequate to re- the civil society mobilizes people’s participation. Therefore, liably measure the impact of an input or intervention. The these three groups of stakeholders need to be brought in- indicators are supposed to measure and reflect change over to the process of feedback and consultation on the quali- a time and hence conform to the criteria represented by the ty of indicators. This arises out of the need to move away acronym CREAM, which refers to the following: from the M&E of mere processes, inputs, and outputs to an emphasis on measuring results, outcomes, and impact • Clear: Precise and unambiguous and that is the goal that should determine the quality of in- • Relevant: Appropriate to the set goal dicators. • Economic: Available or computable with reason- able cost Case study • Adequate: Provides sufficient basis to access perfor- When a recent study was undertaken to prepare a tool to mance monitor the state of governance in India,43 the Centre for • Measurable: Quantifiable Good Governance initially took 271 indicators to shape up and describe five dimensions of governance (see Figure 23). The well-known SMART test uses a quick reference to These indicators, then, were put to a rigorous quality test determine the quality of indicators by ascertaining wheth- based on the SMART criteria. The study conducted exhaus- er they are tive stakeholder consultations at a number of levels. In the end, the number of indicators was reduced to 164, and were • Specific modified substantially, based on information from the stake- • Measurable holders on availability and reliability of the data, frequen- • Attainable cy and dates of collection, the agency’s background, and the • Realistic like. The stakeholder consultations included politicians, bu- • Time-bound reaucrats, experts in the field, and ordinary citizens. 123 M&E SYSTEMS AND M&E TOOLS FIGURE 23: Five Dimensions of Governance The nature of political contestation, conduct of political players & Institutions, use & abuse of political authority and citizen’s faith in the political system Political Ability of the state to manage its human, financial and technological resources to Ability of the state to maintain law & deliver basic services to public order, safeguard human rights, enable access & delivery of justice Administrative Legal & Judicial Economic Social Ability of the state to create a conducive Ability of the state to take care of the vulnerable climate for business in all sectors and ensure sections of the society; role & quality of the civil macro-economic stability society and environmental governance Source: Author. The mechanism adopted was to validate the robustness number of indicators that were considered to arrive at the of indicators on a laid-out criteria by obtaining the opin- composite score on the five dimensions mentioned above. ions of the concerned stakeholders, including the judg- The approach adopted to validate the quality of indica- ment of experts on whether or not a given indicator would tors was based on the findings from field-testing and inputs measure up to the criteria of quality. This qualitative anal- from various stakeholders, including common people, ex- ysis may not completely eliminate the bias in such assess- perts, and professionals. Field-testing gave a lot of insight ment, but it certainly improves the validation process due into the quality of indicators as quite a few things could to stakeholder involvement. be placed in perspective only after this “ground-truthing.” The methodology to identify the indicators included a Some illustrations of the reasons for modifying or deleting review of available literature, different approaches adopt- an indicator are given in Table 18. ed by different organizations to define the basic issues, and The quality, reliability, access, and use of information then a broad list of indicators that were put through a pro- produced and used by M&E systems can be optimized by cess of scrutiny and validation. Table 17 gives an idea of the introducing a Citizen-Centric System of Social Account- TABLE 17: Governance Indicator Number of Indicators In the Initial Indicator Retained after Deleted after Modified/ merged Newly added In the final Governance dimension Framework testing testing after testing after testing framework Political 46 18 19 9 — 24 Legal/Judicial 53 23 22 8 4 33 Administrative 88 29 35 24 11 52 Economic 39 19 19 1 5 25 Social 45 28 14 3 0 30 Total 271 117 109 45 20 164 Source: Author. 124 �SESSION 5: INSTITUTIONAL ARRANGEMENTS AND REL ATED POLICIES TO ENSURE THE QUALIT Y, ACCESS, AND USE OF PERFORMANCE INFORMATION TABLE 18: Modification of Indicator Dimension of Source Action Governance Indicator under validation of data taken Reason Political Proportion of bogus voters in the Secondary Deleted Found difficult to ascertain. electoral list Percent of women candidates fielded Secondary Deleted Outcome—the proportion of women in election elected is not captured by this. Administrative Proportion of women in government Secondary Modified Modified as “Aggregate proportion service in senior cadres of women in government service.” Percentage of the government budget Secondary Deleted Difficult to collect the information as earmarked for women gender budgeting is not done. Social Percent of habitations not having Secondary Modified Percent of parents who say that the primary school within one kilometer primary school is not easily accessible. Source: Author. ability in which, both in collection of information and its • Preparatory groundwork dissemination, the citizen is intensely involved. Some dis- • Input tracking, sharing information, verifying re- tinct tools have proven reliability to improve the effective- cord, comparing supply-side data, and finalizing ness of M&E systems by actively involving the people. the indicators Some of these tools are as follows: • Performance scorecard by the community—the scoring is done by community in the given format • Community scorecards • Self-evaluation by service provider—this involves • Citizen report cards giving score • Social audits • Interface meeting and action planning—feedback • Participatory expenditure tracking, planning, and is given on evaluation of the scorecards budgeting • Institutionalization—following up a repeating scorecard with benchmark data Community scorecards • Consensus—finally, the community action plan in- The community scorecard is a tool to monitor and evaluate volves both indicators given by the service providers the services provided by the state or public service providers as well as community to arrive at a consensus with in a joint and participatory manner. This tool is powerful mutual agreement enough to influence the quality, efficiency, and effectiveness of these services, particularly, at the local level. Community Citizen report cards scorecards recently were developed by the Centre for Good Whereas the community scorecards involve the communi- Governance in health service provision and school educa- ty as a unit, citizen scorecards are administered through a tion.44 These are used by the community as a basic unit survey. They can be used for local, provincial, or nation- of analysis. They focus on monitoring public service deliv- al assessment of services particularly in services like health, ery at the grass-root level and generate relevant information water supply, and utility. Citizen surveys help in bench- through focus group interaction in a participatory manner marking and comparing the level of services or activities that provides feedback and facilitates a joint decision on for the government or unique service provider at a rating the quality of service between the recipients and the service scale and help engage citizens and empower the monitor- providers. This technique usually brings about a clear em- ing process. powerment of community and promotes good governance. Such scorecards have been tried successfully in Ghana, Ma- Social audits lawi, and Zambia. Various features in the construct of com- This is a unique mechanism of creating social accounts in- munity scorecards are as follows: volving the community and to audit a particular parameter 125 M&E SYSTEMS AND M&E TOOLS or activity. The use of social audits in the M&E mechanism The entire exercise has taken the shape of a massive so- can be effective in program implementation. This practice cial movement and, at the same time, all the components is being encouraged in important pro-poor programs, such of the M&E system are standardized with well-designed as the National Rural Employment Guarantee Scheme and indicators and a replicable process. The system has a Web- National Rural Health Mission. The essential steps in so- enabled support through a well-managed portal that has cial audit involve training of social auditors, preferably re- brought transparency and improved accountability. This cruited from the community, creating an environment of has brought to light a good number of cases of irregulari- acceptability by the establishment for social auditors to op- ties in funds flow, and corrective action was taken. Anoth- erate, and a system of fixing responsibility and award of er important aspect is the empowerment of the poor and penalty or punishment for any irregularity. The key to suc- enabling their participation and ownership in the develop- cessful implementation lies in bringing quality to the tools mental process. of measurement and design of indicators, so that only rele- vant information is captured. Participatory expenditure tracking, participatory planning, and budgeting Case study Participatory expenditure tracking, participatory planning, In India, the social audit is ingrained in the design of the and budgeting involve creating people-oriented accounts National Rural Employment Guarantee Scheme (NREGS) of public expenditure in schemes and programs and using by a statutory provision. Therefore, all state governments participatory techniques in budget preparation. This is an- are required to facilitate this while implementing the flag- other example of involving people to bring useful infor- ship program to provide guaranteed wage employment to mation as input in decision making. This can serve as a the rural poor. Various models can conduct social audits. monitoring tool, which is the basic function of a budget. The state of Andhra Pradesh adopted a new approach and The pioneering examples of Porto Alegre in participatory trained a large number of young volunteers from the wage budgeting and Uganda in expenditure tracking are the role earners’ families and empowered them to conduct social models in this. audit. The government provided an enabling environment to facilitate the following processes. What succeeds? Intense civic engagement is the key to success in M&E as it • Both proactive and on-demand disclosure of infor- improves the accountability system. It is important to cre- mation to the wage seekers under the Right to In- ate an overall atmosphere of mutual trust in government to formation Act to give them access to muster rolls, citizen dialogue. An environment of social accountability pay orders, measurement sheets of civil works, en- and specific voice mechanisms for people to air their views gineering cost estimates, and sanction orders. needs to be incorporated. • Creation of requisite infrastructure for a two-day Different networking arrangements, systems of stan- training for at least five volunteers constituting a dardization, and quality assurance in public service deliv- social audit team in each village by providing venue ery such as the instrument of citizen’s charter, simplifica- and space for the training, training material, food, tion and demystification of processes and procedures, and transport, and other logistics. unfettered access to information are necessary ingredients • Facilitating the verification on ground and com- of success. parison with official records to conduct social audit Social accountability is a nonelectoral system that still and summarize the findings over a period of four provides a straight control to bring sufficient account- days. ability to the actions of government. Such mechanism • Hosting the public hearings in the presence of all can be supported by the state, or the citizen, or both and officials and stakeholders to discuss the findings of usually are bottom-up in nature. The use of a robust e- social audit and to take immediate remedial action governance system and grievance redress mechanism to in case of irregularities. improve the interface between the citizen and the govern- 126 �SESSION 5: INSTITUTIONAL ARRANGEMENTS AND REL ATED POLICIES TO ENSURE THE QUALIT Y, ACCESS, AND USE OF PERFORMANCE INFORMATION ment and to bring them close to each other is necessary to the open unless such evaluation is done by a third-par- in this context. ty public agency like the comptroller and auditor general, Poor quality of information for decision making in Planning Commission, or watchdog bodies of legislature public policy results in negative fallouts. These fallouts in- or executive. clude poor targeting of expenditures in various schemes The national system of M&E in India is multilayered, and plans, corruption and leakages in the system, appro- as indicated in Figure 24, and the information to civil so- priation of the subsidies and government assistance by the ciety is given in the form of periodic and annual reports undeserving, wasteful expenditure, and monitoring un- of various committees and ministries, national sample sur- important items at the cost of important ones. The core veys, and the like. of decision making gets obscured if adequate information The spread of information in the public domain is in- is not available. In such cases, the decisions are taken on creasing, with easy access due to use of information tech- the information brought by a close group of people who nology and Web sites. A lot still needs to be done to create may be from a political or bureaucratic or business inter- a baseline and present analysis over a period of time rath- est group. er than giving barebones data. The Planning Commission Many countries have implemented policies and laws of India and the Ministry of Statistics and Programme Im- on “Access to Information.” The difficulties and challenges plementation collect regular data on the government’s var- they each face differ from country to country. The challenge ious programs and schemes. Most of these systems are at is to draw a line between individual privacy and freedom of the macro level and there is a glaring gap between what is information. A brief mention of the challenges faced by In- available at the national level and what the people at com- dia includes a lack of adequate and effective record man- munity level possibly would need. But this dissemination agement systems, resistance of bureaucracy to part with the has to go far and beyond the federal and state levels up information, ignorance of people of the procedures to ac- to the last villages and municipalities. This is possible only cess information, and lack of training to make use of this with intense engagement of community through social ac- law, including government officials and civil society. The countability mechanisms at the local level. The key to suc- implementation aspect of the Right to Information Act has cess, therefore, lies in a bottom-up approach in M&E, rath- provided a strong system of eliciting compliance in India er than a top-heavy macro management system. It may be by the government officials and public authorities, because said in the Indian context that the direction is positive and hiding information attracts severe fines and reprimand. the atmosphere of transparency with citizen involvement is Similar laws exist in a large number of countries now. gradually finding a base at the grass-roots level. Australia has an Ombudsman; Japan has a system of ap- peal to an “Information disclosure Board”; the United State Rajiv Sharma has a system of appeals in case of denial of information to Director, General and Project Adviser, the concerned agencies, with a provision of judicial review; National Implementing Agency, Mexico has a sound information regime; and Nepal has the Centre for Good Governance (CGG), India Whistle Blower Protection provision. In India, the regulatory framework for access to infor- mation generated by M&E systems is good enough in the Performance Information: How Web 2.0 Can Help sense that a citizen can ask for any information. But its Measurement and evaluation (M&E) is challenged by a effectiveness lies in the voluntary and proactive disclosure paradox: the ability to measure the effectiveness of a gov- of information by the public authorities. The government ernment program is negatively correlated with the ease of should help citizens learn where the information resides measurement. Inputs (what government consumes) and and what he or she should request. outputs (what government produces) are relatively straight- The voluntary and proactive mechanisms of disclosure forward to measure, but they do not give an indication of have not fully matured in the government departments. the effectiveness of the intervention. Measurement of im- The data in respect of such evaluation are not brought in- pacts and outcomes is much more challenging—the tech- 127 M&E SYSTEMS AND M&E TOOLS FIGURE 24: Structure of the Government M&E System in India GOVERNMENT OF INDIA Central System Planning Audit & Ministry of Statistics & IT Research Civil Society Commission Accounts System Programme Implementation (MoSPI) System Institutes Organizations PA&M Div. Controller Statistics Program NICNET IIPA NGOs General of Acc Wing Implementation PEO NIRD Media Comptroller CSO 20 Points & Auditor General Program PO&RM Div. ICSSR Individuals NSSO Infrastructure CGOOM Others Monitoring DEAC Project Monitoring Plan Speed Group MPLADS State System State Planning Boards Dept of Stats & Econ Planning Statistical Bureau of Economics Directorates/ SEOs Department Bureaus & Statistics Boards of Statistics CGOOM – Core Group on Overview of Outcome Monitoring DEAC – Development Evaluation Advisory Committee IIPA -- Indian Institute of Public Administration PA&M – Project Appraisal & Management Division PEO – Programme Evaluation Organization PO&RM – Programme Outcome & Response Monitoring Division SEO – State Evaluation Organization NIRD is supported by a system of state level institutes of rural development (SIRDs) Source: Author. niques to measure are more complex and take longer, the integrating a top-down centrally driven approach to perfor- costs of measurement are higher and the challenges of attri- mance management with bottom-up community-based re- bution of outcome to intervention are well understood. As porting of citizen experience. a consequence, evaluation of a specific intervention can re- ly on indicators that are readily measureable, although they What is performance information and how can it may not reflect the outcome, or the underlying policy in- best be measured?45 tent of the intervention. This section describes the approach that New Zealand has Recent developments in information communication taken to measure government performance. Performance technology (ICT), in particular the phenomenon of Web in the public sector can be assessed at four levels: the con- 2.0, may provide a response to this paradox and improve sumption of resources by a government agency, the outputs the quality of M&E. This paper describes a framework for (products) produced by the agency, the impact of those 128 �SESSION 5: INSTITUTIONAL ARRANGEMENTS AND REL ATED POLICIES TO ENSURE THE QUALIT Y, ACCESS, AND USE OF PERFORMANCE INFORMATION FIGURE 25: Linking Outputs, Impacts, and Outcomes What are our goals Outcomes Less deaths on the road for New Zealanders? PLANNING What difference are Reduced Reduced Increased use Impacts we making? Speed drink-driving of safety belts DELIVERY What activities are Outputs Engineering Enforcement Education we undertaking? How are we using Resources Workforce Funding Assets our resources? Source: Author. outputs, and the resulting outcomes. The distinction be- Development (OECD) in their ambitious measurement tween these levels and examples at each level are shown in and evaluation program. The initial report, published in Figure 25. October 2009, did not attempt to measure the higher lev- �Measurement at each level requires different tech- els of outputs and outcomes.48 While this proved successful niques, both in gathering the data that can be used to assess in focusing attention on improving individual agency per- performance, and in the attribution of results to specific ac- formance, the bigger challenge—evaluation of the impacts tions, or programs. The characteristics of measurement at and outcomes—has proved more elusive. each level are shown in Table 19. �In general, the lower levels are easier to measure, and Trends in ICT the higher levels are more meaningful for evaluation pur- poses. The accountability regime for delivering outputs Web 2.0 involves a range of legislative and assessment levers.46 For The widespread use of the Internet has resulted in the de- lower level measurement, the New Zealand public man- velopment of software that enables cooperative work- agement system has adopted a devolved approach, hold- ing, and creation of new tools and approaches that enable ing agency heads completely accountable for delivery of groups to manage and share information. These tools have outputs. been given a generic title of “Web 2.0” or “social media”; A number of countries use a scorecard or dashboard ap- examples include the following: proach to assess the performance of an agency. Three exam- ples are as follows:47 • Collectively authored information sources, wikis such as Wikipedia New Zealand: Performance Improvement Framework • Social network sites such, as Facebook and LinkedIn Canada: Management Accountability Framework • User generated content sites such as Blogs, You- United States: The e-government scorecard from the Tube, and Flickr previous administration has been super- seded by the Performance Assessment The magnitude of these changes and the effect on coop- Rating Tool erative working is set out in The Wealth of Networks,49 which describes how social production is reshaping markets, while A similar framework, Government at a Glance, was ad- at the same time offering new opportunities for innovation opted by the Organisation for Economic Co-operation and and creativity. Clay Shirky (2008) describes how technology 129 M&E SYSTEMS AND M&E TOOLS TABLE 19: Measurement Techniques Type of Result Level Focus On Examples of Common Measures END or FINAL Impact or Outcome Aggregate improvement Greater health/wealth/happiness More equity Less difference across deciles/areas Cost effectiveness Fewer deaths/accidents/kids in care Unintended effects Cost per unit of improvement in outcome Increased welfare dependency, risk, and so on INTERMEDIATE Impact or Outcome Cognitive change Percent aware of risks/able to use new idea Behavior change Percent investing/saving/quitting/ working Risk reduction Fewer drunken drivers/“bad” incidents Lifestyle change Percent in jobs/new career/crime free Survival Percent alive after 30 days/time event-free Unintended effects Graduates migrating or excessive uptake NEAR TERM Impact or Outcome Completion rate Percent finishing/getting qualified/in service Knowledge retained Percent core messages remembered Reduction in queue Average wait time/number in queue Receipt of benefits Percent impoverished with more money Incentives changed Percent believing regulatory change matters Unintended effects Higher incident or reduced survival rates COVERAGE Output Coverage Percent population in need receiving output (or Reach) Targeting efficiency Percent in “treated” group who met entry criteria Access Percent targets who did not access/use service Transit time (or other “big” barrier to use) QUALITY Output Quality of delivery Percent output fully meeting specification Timeliness Percent ministerial/passports/etc. on time Acceptability Percent who would use again/recommend use QUANTITY Output Volume produced People receiving training/rehabilitation Cases/complaints processed EFFICIENCY OF Input Utilization Efficiency Real output price trend (inflation adjusted) PROCESS Economy Price per unit, versus benchmarks Percent prison beds full/maximum capacity used Real input cost trend (for example, per policeman or nurse) Source: Author. enables group collaboration in a way that was never previous- The last two years have seen the meteoric rise of social ly possible. Another example of Web 2.0 is the widespread media on a global basis, and a significant effect within the use of aggregated rating as a method of assessment, combing political domain, most notably in the election of President the individual views of any product or service consumed by Obama on the back of an Internet-savvy campaign, and a a wide range of people—news stories, books, buyers and sell- raft of open government initiatives from the administration ers on eBay, hotels, music, and even American Idol contes- in Washington, D.C. tants—to create a collective “wisdom of the crowds.” At its Technology-based consultation is starting to be adopt- heart, the change is a shift from centralized production and ed more widely in the policy creation process, using social control of information products to a distributed cocreation media tools to enable the collective development of pol- resulting in a richer and higher quality result. icy options for consideration by lawmakers. One of the earliest examples of this was the use of a wiki to enable Government 2.0 community input to a new Police Act in New Zealand in The term Web 2.0 was initially popularized by Tim 2007.51 O’Reilly in 2004. Tara Hunt took the O’Reilly tenets for In 2009, the use of these tools increased significantly, Web 2.0 and applied them to government, and came up for example, the development of the open government pol- with the nine concepts in Box 5.50 icy in the United States.52 A similar approach was taken in 130 �SESSION 5: INSTITUTIONAL ARRANGEMENTS AND REL ATED POLICIES TO ENSURE THE QUALIT Y, ACCESS, AND USE OF PERFORMANCE INFORMATION BOX 5: Comparison of Web 2.0 and Government 2.0 Web 2.0 Government 2.0 The Web as a platform The government is my springboard The long tail The long diverse tail of citizens Data: the next Intel inside Data are tricky Users add value Going to the edges for feedback Network effects by default Citizen community is relationships Some rights reserved Some rights reserved The perpetual beta Evolution is an ongoing process Cooperate, do not control Trust builds empowerment Software above a single device Government on the go Source: Presentation by Tara Hunt, May 2007. Australia to create Public Sphere—Government 2.0 Policy that the priority for government should be to produce ma- and Practice.53 chine-readable data (wholesale). Governments have a sec- Beth Noveck, director of the White House Open Gov- ondary role in interpretation (retail), which many others ernment Initiative, has described the three values of open also can do—any interpretation of data is only one view. government—transparency, participation, and collabora- Government will need to shift from being a retailer of da- tion. Each federal government agency must have a road ta to providing wholesale data—focusing on building an map for open government, which not only covers technol- information infrastructure with a focus on the quality of ogy, but also includes people and operations; agencies are data. This becomes increasingly important in the mea- required to develop a schedule for publishing data online in surement of the impacts and outcomes of programs—gov- raw structured machine-readable formats.54 ernment can provide the platform for civil society to oper- This approach of sharing data and engaging in open ate as watchdog and innovator. collaborative dialogue enables governments to work with Open government data have been defined as data that are the citizens and the community to tap into wisdom and reviewable against eight principles: complete, primary, time- expertise outside government, strengthen trust in govern- ly, accessible, machine readable, nondiscriminatory, non- ment, and reduce costs. proprietary, and with an open re-use license. Open data can deliver better citizen engagement, and strengthen trust in Open government, open data government. The Gapminder tool illustrates how official da- One significant and relevant trend is the opening up of ta can be used by the community in a way that never could government data for use by commercial and nonprofit or- be achieved within government.55 Other publicly available ganizations. Most jurisdictions have laws that provide for free tools include ManyEyes, Swivel, and DataMasher. The public access to information, with exceptions to be sub- supply of government data is increasingly being cataloged ei- ject to specific justification; although this legal position is ther by governments themselves or by civil society.56 clear, operational practice is often far from compliant with the law. What are the implications of Web 2.0 for M&E? Government information is public data. While the gov- As illustrated in Figure 26, measuring the effect of gov- ernment collects and stores a lot of information, it rarely ernment interventions is likely to be more informal, qual- has the time or resources to fully use the data. This means itative, collective, and community based, than measuring 131 M&E SYSTEMS AND M&E TOOLS FIGURE 26: Different Measurement Techniques Informal Qualitative What are our goals Outcomes Collective for New Zealanders? Community MEASURE EFFECT PLANNING What difference are Impacts we making? What activities are Outputs we undertaking? Formal MEASURE AGENCY Quantitative PERFORMANCE How are we using Resources Analytic our resources? Administration Source: Author. inputs and outputs of individual government agencies. The ERO’s findings inform decisions and choices made by par- characteristics of measurement of the effect of government ents, teachers, managers, trustees, and others, at the indi- programs are well aligned with the Web 2.0 collective as- vidual school and early childhood level and at the national sessment tools deployed in the commercial world. It is clear level by government policy makers. that government does not have the resources to effective- Based on the findings from individual school reviews, ly measure the breadth of impact of programs, and tapping ERO also prepares national reports that evaluate differ- into the wisdom of the crowds is a potentially powerful tool ent aspects of the education outcomes sought by govern- to achieve better measurability. ment. Recent reports include Progress in Pacific Student Achievement, Teaching of Health and Physical Educa- Examples from the New Zealand education sector tion, and curriculum materials to support teaching and This section contains three specific examples from the New learning of Te Reo Maori. This approach can be de- Zealand education system, ranging from formal evaluation scribed as systemwide performance evaluation by profes- based on legislation to crowd-sourced assessment of teach- sionals giving broad performance information on educa- er performance. tion outcomes.59 Formal assessment of schools Publication of data on student and school The Education Review Office (ERO) was established in performance 1989 to evaluate and report publicly on the education and The use of the data on individual student performance is care of students in schools and early childhood services. currently an issue of significant debate in New Zealand. In ERO investigates and reports to boards of trustees, manag- 2010, the government will introduce literacy and numera- ers of early childhood education services, and the govern- cy testing of individual students in year one through eight ment on the quality of education provided for children and against national standards, a move that is opposed by some students in individual centers and schools. professionals within the education sector. Schools and early childhood services are reviewed on av- One area of concern is publication of such data, when erage once every three years. Reviews are undertaken more combined with emerging ICT tools, means that the na- frequently in cases in which the performance of a school tional standards can be used to compare the performance or center is poor and there are risks to the education and of schools through “league tables.” The subject is contro- safety of the students. Reports on individual schools and versial in other countries such as Australia, Japan, and the early childhood services are freely available to the public. United Kingdom.60 This approach can be described as ag- 132 �SESSION 5: INSTITUTIONAL ARRANGEMENTS AND REL ATED POLICIES TO ENSURE THE QUALIT Y, ACCESS, AND USE OF PERFORMANCE INFORMATION gregating individual professional assessments of achieve- government working with the community to share data ment, to provide comparative performance information. and information on the impacts and outcomes of individ- ual policy initiatives. This option, of using a wider range of Community-based assessment of teachers sources and tools, gives policy makers a richer palette from Rate My Teacher is a Web service that provides students which to paint the evaluation picture. with an opportunity to rate individual teachers. While there are drawbacks to such a service—harsh and unfair com- Laurence Millar ments often cam emerge from anonymous assessments—it Managing Director, Global Village Governance Limited and provides a new and unique insight into the effectiveness of Former New Zealand Government Chief Information Officer teachers from the perspective of the “customer” that previ- and Deputy Commissioner, State Services Commission, ously was not available. This approach can be described as New Zealand collective and subjective performance information of indi- viduals and institutions. Institutional Arrangements and Policies of Conclusion and implications for M&E Monitoring and Evaluation Systems to Ensure Evaluation is not an exact science; the challenges of mea- Quality, Access, and Use of Information surement and attribution mean that effective measurement always will be a composite of a number of indicators. In Introduction 2005, at the first of these meetings, the challenges of M&E Within the framework of state modernization, the govern- were laid out.� ment’s coordination task is a critical, central point. The Technical developments in social media, initially ap- government must seek to improve efficiency of state ac- plied in the commercial and personal arena, provide an ex- tivities on a continual basis, and this has, in turn, made it citing opportunity to add an additional lens to our evalu- necessary to seek new forms of exercising adequate strate- ation tool kit—one that enrolls stakeholders more actively gic leadership and control mechanisms for all government in the evaluation process. This has the potential to ad- activities. dress three of the challenges—ensuring relevance, involv- Thus, interest in the allocation and use of resources has ing stakeholders, and communicating the results to stake- led to concern over results and, especially, impacts. It has holders. generated a tendency to consider the budget as one of the The use of public choice continues to evolve as a pol- governments’ primary operational systems for coordina- icy mechanism to improve performance—the contestable tion, monitoring, and evaluation. As a result, the financial supply of government-funded services through vouchers or ministries have taken on an increasingly central role. other citizen-controlled selection of services. Previously, as- In Chile, the Treasury Ministry’s mission is defined as sessment and evaluation of the results were blunt instru- “maximizing the economy’s long-term growth potential and ments. They amounted to the quantitative assessment of promoting the best use of the country’s productive resources how citizens exercise choice. This resulted in simplified per- in order to achieve sustainable economic growth that trans- formance information on the outcomes of a program. lates into a better quality of life for all Chileans and espe- Broadly based personalized capture of qualitative infor- cially for the most vulnerable, left-behind sectors.”� Espe- mation means that evaluation can now can be a lot more cially noteworthy among its strategic objectives are (1) to nuanced, reflecting the aggregation of individual citizen promote efficiency in the use of public resources, develop- evaluations. There are many examples of civil society mak- ing management control techniques and instruments linked ing use of government information to enable more effec- to the budget; (2) to direct the state’s financial administra- tive evaluation. As government information becomes more tion with the objective of fulfilling its institutional mission, widely published as raw data, this trend will become more and (3) to develop the budget proposal for the public sector, widespread. As a result, the evaluation of the effectiveness and establish the norms for its implementation. Outstand- of government programs becomes a cooperative effort— ing among the main products obtained are the following: 133 M&E SYSTEMS AND M&E TOOLS • Development of the nation’s budget and its negoti- Presidential priorities ation in parliament Based on the government program, as well as the commit- • Control over budget implementation and the cor- ments in presidential messages and addresses, a set of goals responding achievement of expected results and projects in which the president has particular interest • Evaluation of government programs are presented. These may be legislative initiatives or matters • Definition, implementation, and monitoring of related to government management. economic policy • Participation in international economic negotia- Government programming tions This system, operated by the Inter-Ministerial Coordina- • Advancement in the process of perfecting the cap- tion Office of the General Presidential Ministry Secretariat, itals market, and the country’s international finan- makes it possible to maintain constant, total monitoring of cial integration all the institutions in the executive branch of government, with regard to progress made and fulfillment of their pri- Mechanisms for ensuring the quality of M&E mary annual commitments. It assumes that, for each year, indicators with the Government Program in mind, the various insti- To ensure the quality of the indicators in the M&E sys- tutions establish their primary commitments in relation to tems, governments must consider and understand the func- the goals for their actions. This programming task is im- tion of strategic planning and work planning. In this way, portant for the development of the budget proposal for it is possible to develop better M&E indicators that will al- each institution and should be subsequently adjusted to low for more precise, assertive evaluations. In Chile, the el- what is established in budget legislation. ements considered include the government program and The task of monitoring allows the government to obtain presidential addresses, messages and priorities, and govern- an overall perspective of its priorities within each part of the ment programming. Based on these instruments, indicators administration. It makes it possible to determine the degree are added and defined to monitor the government’s oper- to which each concrete plan is being carried out; to what de- ation and to better evaluate the progress made in fulfilling gree the results meet the objectives; and the commitments as- what has been promised. sumed by the president. The primary objective of this mon- itoring process, in addition to organizing the government’s Government program activities, is to anticipate and contribute toward resolving the This is the starting point for defining the government’s obstacles and difficulties that the state apparatus may be con- tasks, because it contains the proposals and issues that were fronting in achieving the objectives. These difficulties may revealed during the presidential campaigns and that the refer to matters of a legal, budgetary, or management nature, government has committed to achieving. or to insufficiencies in human resources aspects. In particular, government programming involves an on- Presidential messages going process that is initiated with the definition of annual In these messages, presidents give an administrative ac- priority tasks by each ministry. These tasks must be in ac- count of what has been achieved, present their perspectives cordance with the definition of policies in each institution. on the situations in their countries, declare new commit- The process later incorporates evaluation stages, with mon- ments, and specify the issues proposed in the government itoring and evaluation (M&E) at the end of each quarter. program or define new policies and projects. The information generated in this entire process is turned over to the president, ministries, and assistant minister’s of- Presidential discourses fices. Along with this quantitative evaluation, the Inter- These are the presidents’ public interventions that refer to Ministerial Coordination Office drafts a quarterly report of specific issues or particular realities, and that define public a qualitative nature for each ministry, and this report em- policies, proposals for action, or commitments for govern- phasizes the main achievements or delays in the quarter and ment actions. suggests some lines of action. 134 �SESSION 5: INSTITUTIONAL ARRANGEMENTS AND REL ATED POLICIES TO ENSURE THE QUALIT Y, ACCESS, AND USE OF PERFORMANCE INFORMATION Tools for optimizing quality, reliability, access, and General Presidential Ministry Secretariat and the Bud- use of M&E information get Office. The tools for optimizing quality, reliability, access, and use of information produced and used by M&E systems are The Law on Access to Public Information important instruments for verifying progress of planned ac- (No. 20.285, dated April 20, 2009) tivities, and if corrective measures are needed. The follow- This law recognizes the right of individuals to access public ing tools are available: information, with the appropriate legal exemptions. This law also obliges state entities to maintain a body of relevant Governmental programming information for supervision over their performance perma- Governmental programming makes it possible to maintain nently available to the public through their respective Web constant, total monitoring of all the institutions in the exec- sites, and establishes sanctions for authorities who deny ac- utive branch of government, with regard to progress made cess to information without justification. and fulfillment of their primary annual commitments. Most of these instruments are characterized by their simplicity and ease of use, and contribute to the function- Monitoring of presidential priorities ing of the entire model and to the creation of synergies Every month a document is developed with an analysis of among them. The production and processing of associat- the current status for each one of the presidential priorities. ed indicators is assisted by available modern information The information is recorded in an ad hoc document. This technologies, particularly statistical programs, tools for da- monitoring process makes it possible to carry out appropri- tabases and treatment of information, and, above all, Web ate actions to fulfill the identified priorities. pages for disseminating associated information and the cor- responding access available to citizens. Control over budget use The main tool is the System of Management Control One of the elements in state modernization corresponds and Results-Based Budgeting. Through this tool, a link is to understanding the budget in relation to the desired re- provided between the use of allocated resources and the sults and impacts from the implementation of public poli- consequent decisions regarding allocation of funds, by cies—more than in terms of the use of inputs and control obtaining information on performance and introducing over such use. National budget development and budget- best practices for improving the quality of expenditures. ary control actions increasingly have been advancing to- The basis for the model is the traditional budgetary cy- ward a conceptualization of a results-based budget. Thus, cle: the development of the budget, its discussion and ap- the task of the Treasury Ministry’s Budget Office is funda- proval, its implementation, and its evaluation. During the mental and complementary to the work of monitoring gov- various stages of the cycle, various performance-linked in- ernmental programming and therefore to monitoring the struments are used, including performance indicators and achievement of the government’s tasks and objectives. In evaluation. addition a fundamental role is played by the Planning Min- Performance indicators and evaluations are instruments istry in the social evaluation of investment projects, a task for collecting information regarding performance, and even that facilitates and complements the monitoring of govern- when both have common elements and complement each mental programming. other; they are different in terms of methodological aspects and use. Performance indicators are measurements that Evaluation of government programs contain algorithms for measuring results. Evaluations, on Evaluation is directed by the Budget Office and focused on the other hand, are studies and, consequently, more com- verifying impacts from the various public programs. prehensive than measurements, although they include per- formance indicators. Analysis and monitoring Other instruments cover another part of the decision- Analysis and monitoring of public policies is carried making process and are included in the model. One of out by presidential advisors in collaboration with the these instruments is the presentation of budget initiatives, 135 M&E SYSTEMS AND M&E TOOLS which has its own methodological and conceptual ele- mation. According to this law, all information “prepared ments, and is focused on reviewing new initiatives or on with public funds and any other information possessed by expanding those already existing for a budget presenta- Administration entities, no matter what its format, medi- tion. In addition, the comprehensive management analyses um, date generated, origin, classification or processing” is (CMA), which are annual reports prepared by the institu- deemed public information.61 The only reasons for infor- tions and coordinated by the Treasury Ministry, cover a set mation to be reserved or maintained in secrecy, causing ac- of financial and nonfinancial performance topics. Finally, cess to be partially or totally denied, are the following: the management improvement programs (MIPs) have a less direct link with budget decision making and include mone- • When the public availability, communication, or tary incentives for the public functionaries involved, in ac- knowledge of such information may affect the nec- cordance with the fulfillment of certain objectives in carry- essary due performance of the entity’s functions. ing out the management system. • When the public availability, communication, or knowledge of such information may affect the Challenges in ensuring the quality of information rights of persons, particularly when their safe- for decision making ty, health, private life, or commercial or econom- One ongoing challenge to ensuring the quality of informa- ic rights are involved. tion for public policy decisions continues to be to identi- • When the public availability, communication, or fy the best size of the system for the best administration. knowledge of such information may affect the na- This objective is pursued by moving toward those indica- tion’s security, especially when national defense, tors identified as the most useful and the most relevant, maintaining public order, or public security are in- and with the least difficulty or time involved. This may re- volved. quire having to focus on intermediate indicators of results, • When the public availability, communication, or which are simpler to construct but are not less assertive, knowledge of such information may affect national and which may be evaluated in shorter time periods during interest, especially when public health, internation- the implementation of programs. al relations, or the country’s economic or commer- Another challenge is to move toward products indica- cial interests are involved. tors focused on aspects of quality, such as user satisfaction • When involving documents, data, or information with respect to the public services they receive, and the op- that a “qualified quorum” has declared reserved or portunity for attention, as well as other indicators linked to secret (in which case administration entities are re- quality, such as beneficiaries’ perceptions and the technical quired to make and maintain an updated index quality of goods and services delivered. public).62 With regard to evaluations, it is necessary to improve the use of the results in public institutions. Although sta- While the enactment of this law has meant a major step tistics on use of results point to some progress, they also forward in guaranteeing citizens’ access to information gen- show that much work is left to do to generalize their use, erated by M&E systems, its implementation has not been especially in the case of performance information generat- without difficulties. The main difficulty has involved the ed by M&E systems. Matters also are pending in the area delayed dissemination of information within the public of improving the information available within institutions sector regarding the new institutionalism, as well as the in- for better applying evaluation methodologies and thus pre- sufficient training of key public functionaries prior to the cisely improving evaluation processes. law entering into effect. This has hindered the law’s imple- mentation, and the necessary elements have not been in Difficulties, advances, and challenges in policies place since the beginning for the various public services to and laws for access to public information be prepared. Thus, they have not been able to provide an The recently enacted Law on Access to Public Information adequate response to citizens’ requests for information. Al- recognizes the right of individuals to access public infor- so, the limited time period established for its implemen- 136 �SESSION 5: INSTITUTIONAL ARRANGEMENTS AND REL ATED POLICIES TO ENSURE THE QUALIT Y, ACCESS, AND USE OF PERFORMANCE INFORMATION tation has not been sufficient for departments to improve these departments prepare adequately for making all nec- their archive systems and the organization of their infor- essary information available. This does not always happen, mation in a timely manner. The greatest challenge, howev- however, because although public entities have large vol- er, has involved replacing the tradition of secrecy—which umes of information, they do not necessarily administer or continues to prevail in the public sector—with a new tradi- organize it well. And the organization they have developed tion of transparency. is not necessarily useful for the purposes of internal admin- Nevertheless, despite all the difficulties, a preliminary istrative control. In addition, the application of these sys- evaluation gives reason for optimism, because significant tems has been advantageous: these entities now are con- advances have been noted, resulting from efforts to pro- cerned about and paying attention to the need for better mote the new legislation. This means it will be increasingly coordination. Lastly, the greater dissemination of M&E in- possible to count on more public functionaries to contrib- formation has led to a favorable public perception in terms ute to creating a public tradition of transparency. of greater transparency in public management. With the re- cently enacted Law on Access to Public Information, this Other considerations transparency undoubtedly will be increasingly recognized. These considerations clearly contribute to improving ac- cess to information generated by M&E systems, thus bal- Fernando Barraza ancing the technical and political difficulties involved in International Consultant, Former Subdirector of Information disseminating M&E information, given the right to access Technology for the Service of Internal Taxes, Chile to public information. With regard to the practices and mechanisms generated by the M&E systems for providing information to civil society, the first element that should be Session 5 Notes 43 emphasized is a self-imposed requirement that all informa- The Centre for Good Governance, Hyderabad, conducted tion produced by these systems is of a public nature. Thus, the study for the Department of Administrative Reforms and Public Grievances, Government of India, in 2008. such information is not only sent to Congress, but also di- 44 These tools were developed by the Centre for Good Gover- rectly reported to the entities involved and also published nance, Hyderabad, India in a World Bank Institute–funded in full on their respective Web sites, in addition to the Bud- initiative. get Office Web site,63 thereby made permanently available 45 See the New Zealand government Web site, www.ssc.govt.nz. 46 to all the country’s political and social actors having a need This approach is described at the State Services Commis- for such information. In addition, the information must sion Web sites, http://www.ssc.govt.nz/display/document.asp? DocID=7121, http://www.ssc.govt.nz/display/document.asp?Doc fulfill certain technical requirements and, to this end, there ID=3530, and http://www.ssc.govt.nz/display/document.asp?Doc is a set of support documents, methodologies, and proce- ID=3581. dures that also are available to the public. 47 See the Web sites, http://www.ssc.govt.nz/display/document. Producing timely results is an important requirement, asp?navid=346, http://www.tbs-sct.gc.ca/maf-crg/index-eng. since they should be used in analysis and decision-mak- asp, http://www.whitehouse.gov/omb/assets/egov_docs/2009_ ing processes. If information is late in becoming avail- Expanding_E-Gov_Report.pdf, and http://www.whitehouse.gov/ omb/expectmore/ able, it is no longer relevant and becomes obsolete. This 48 See the Web site, http://www.oecdbookshop.org/oecd/display.asp? makes it necessary, therefore, to have rigorous procedures, sf1=identifiers&st1=5ksm37l6svjk&CID=&LANG=EN to ensure that time limits are observed, and to guarantee 49 The entire book is available at http://yupnet.org/benkler/ both the quality and the timeliness of the corresponding 50 See the Web sites, http://www.slideshare.net/missrogue/govern- information. ment-20-architecting-for-collaboration and http://www.e.govt. With regard to the impacts from the dissemination of nz/resources/research/progress/transformation/chapter7.html. 51 and access to M&E information, specifically in terms of See the Web site, http://www.policeact.govt.nz/wiki/. 52 See the Web site, http://www.mixedink.com/opengov/. improvements in public management, the first point to un- 53 See the Web site, http://www.katelundy.com.au/2009/05/29/ derscore is the positive effect on the public departments public-sphere-2-open-government-policy-and-practice/. that have been subjected to M&E systems. This has helped 137 M&E SYSTEMS AND M&E TOOLS 54 See the Web site, http://www.whitehouse.gov/blog/2009/ Latest-News.aspx/Alliance-of-school-communities-unite-to-op- 12/09/why-open-government-matters. pose-school-league-tables.aspx. 59 55 See the Web site, http://www.youtube.com/v/BPt8ElTQMIgv. See World Bank 2005. Institutionalizing M&E systems in 56 Examples can be found at the Web sites, www.data.gov, www. Latin American and Caribbean Countries. PREM Notes 102, open.org.nz, www.opengov.se, and www.opengov.fi. available at the Web site, http://www1.worldbank.org/prem/ 57 For full details, see the Web site, http://www.ero.govt.nz/ero/ PREMNotes/premnote102.pdf/. 60 publishing.nsf/Content/ERO%27s+Role. See the Web site, www.hacienda.cl/ministerio. 61 58 See the Web sites, http://www.telegraph.co.uk/education/ Article 5. 62 6193787/Scrap-school-league-tables-say-experts.html, http:// Article 21. 63 www.yomiuri.co.jp/dy/editorial/20091018TDY04304.htm, and See the Web site, www.dipres.cl/fr_control.html. http://www.parra.catholic.edu.au/News---Events/Latest-News/ 138 Select Presentations from the Fourth Meeting of the Latin American and Caribbean Monitoring and Evaluation Network Belo Horizonte, Brazil, November 25, 2008 Institutionalizing Impact Evaluation: Lessons en place in the target population over the life of the proj- from Latin America and Other Regions66 ect. In all these cases, data are generated only on the target This paper describes the progress being made in the tran- population and no information is collected on sectors that sition from individual impact evaluation (IE) studies to do not benefit from the project or that, in some cases, may building a systematic approach to identifying, implement- even be worse off as a result of the project. ing, and using evaluations at sector and national levels. With all of these approaches, there is a tendency for the When this is achieved, the benefits of a regular program evaluation to overestimate the true benefits or effects pro- of IE as a tool for budgetary planning, policy formula- duced by the project (Bamberger 2009b). Typically, on- tion, management, and accountability will begin to be ap- ly project beneficiaries are interviewed. Families or com- preciated. To date, the institutionalization of IE has been munities that do not benefit are not interviewed and the achieved only in a few developing countries, mainly in Lat- evaluation does not present any information on the expe- in America, but others have started the process. This chap- riences or opinions of these nonbeneficiary groups. Many ter draws lessons on the benefits of an institutionalized ap- of these evaluations are methodologically weak and often proach to IE, the conditions that favor it, the challenges biased and consequently there is a serious risk that ODA limiting progress, and the steps in developing such an ap- agencies may continue to fund programs that are produc- proach. The institutionalization of IE can be achieved only ing smaller impacts than are reported and that may even be within the framework of a comprehensive monitoring and producing negative consequences for some sectors of the evaluation (M&E) system that provides a menu of evalua- target population. tions to cover all the information needs of managers, plan- Although no hard statistics are available, it is quite like- ners, and policy makers. ly that rigorous IE designs are used in only 10 percent of ODA impact evaluations. Given the widespread recogni- The importance of impact evaluation for official tion by ODA agencies of the importance of rigorous IEs, development assistance why are so few conducted? Reasons include the limited To assess the effectiveness of official development assistance evaluation budgets of many agencies. Most evaluations are (ODA) programs in contributing to poverty reduction, not commissioned until late in the project cycle and con- economic growth, and sustainable development, it is im- sultants are given a short time to conduct data collection. portant to conduct systematic analysis of development ef- Also, many government agencies see evaluation as a threat fectiveness. Many assessments of effectiveness are based ei- or something that will demand a lot of management time ther on data collected from project beneficiaries after the without producing useful findings. Because many evalua- project has been implemented, or on the use of M&E or re- tions produce a positive bias thatshow programs in a pos- sults-based management to measure changes that have tak- itive light, many agencies do not feel the need for more 139 M&E SYSTEMS AND M&E TOOLS rigorous (as well as more expensive and time-consuming) cies. Figure 27 identifies three alternative pathways to the evaluation methodologies. One of the challenges for the in- institutionalization of IE. stitutionalization of IE is to convince both ODA agencies and host country governments that rigorous and objective The first pathway impact evaluations can become valuable budgetary, policy, The ad hoc or opportunistic approach evolves from indi- and management tools. vidual evaluations that took advantage of available funds and the interest of a senior government official or a partic- Institutionalizing impact evaluation ular donor. The approaches were gradually systematized as Institutionalization of IE at the sector or national level oc- experience was gained in selection criteria, effective meth- curs when (1) the evaluation process is country led and odologies, and how to achieve both quality and utilization. managed by a central government ministry or sector agen- A central government agency, usually finance or planning, cy, (2) buy-in from key stakeholders is strong, (3) proce- either is involved from the beginning or becomes involved dures for selecting, implementing, and using IEs are well as the focus moves toward a national system. Colombia’s defined, (4) IE is integrated into national M&E systems national M&E system illustrates this pathway with the that generate much of the evaluation data, (5) IE is inte- Ministry of Planning responsible for managing the Na- grated into national budget formulation and development tional System for Evaluation of Public Sector Performance planning, and (6) the focus is on evaluation capacity de- (SINERGIA). Although selection of evaluations initially velopment (ECD). Institutionalization is a process, and at was somewhat ad hoc and strongly influenced by donor in- any given point, it will have advanced further in some sec- terests, as the program of IE evolved, the range of method- tors than others. How IE is institutionalized will also vary ologies was broadened and technical criteria in the selection from country to country, reflecting different political and of programs to be evaluated were formalized through poli- administrative systems and historical factors such as donor cy documents (with more demand-side involvement from support for particular sectors. the agencies managing the programs being evaluated) and Many IEs have not been able to achieve their poten- in how the findings are used. Most of the IEs use rigorous tial contributions to program management, budget plan- econometric evaluation techniques (Mackay 2007, 31–36; ning, and policy making because evaluations were selected Bamberger 2009a). and funded in an ad hoc and opportunistic way determined by the interests of donor agencies or individual ministries The second pathway rather than by national planning priorities. The value of The second pathway is where IE expertise is developed in a IE as a policy-making tool can be greatly enhanced once priority sector supported by a dynamic government agency selection, dissemination, and use of the evaluations be- and strong donor support, and for which there are impor- comes part of a national IE system. This requires an annual tant policy questions to be addressed. Once the operational plan for selection of the government’s priority programs on and policy value of these evaluations has been demonstrat- which decisions have to be made concerning continuation, ed, this becomes a catalyst for developing a national system. modification, or termination and for which the evaluation framework permits the comparison of alternative interven- Mexico example tions in terms of cost-effectiveness and contributions to na- The evaluations of the Education, Health, and Nutrition tional development goals. Program (Programa de Educación, Salud y Alimentación, PROGRESA) conditional cash transfer programs in Mex- The three pathways ico are an example of this approach (Mackay 2007, 56; No single strategy is always successful in the institutional- Bamberger 2008). A series of rigorous evaluations of the ization of IE. Countries that have made progress have built PROGRESA programs conducted over a number of years on existing evaluation experience, political and administra- convincingly demonstrated the effectiveness of condition- tive traditions, and the interest and capacity of individual al cash transfers as a way to improve the welfare (particular- ministries, national evaluation champions, or donor agen- ly health and education) of large numbers of low-income 140 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K FIGURE 27: Three Pathways for the Successful Institutionalization of IE First Second Third IE starts through IE starts in IE starts at whole- ad hoc studies particular sectors of-government level Ad hoc opportunistic studies often Sector management information Whole-of-government M&E system with strong donor input systems Increased involvement of national Large-scale, more systematic sector Incorporation of government-wide government evaluations performance indicators Focus on evaluation capacity National system with ad-hoc, supply- development and use of evaluations driven selection and design of IE Systematization of evaluation selection and design procedures Increased involvement of academic Standardized procedures for and civil society selection and implementation of IE Standardized procedures for dissemination, review and use of IE findings Whole of government standardized IE system Examples: Colombia–SINERGIA Examples: • Mexico–PROGRESA Examples: Chile–Ministry of Finance Ministry of Planning conditional cash transfers • Uganda–Education for All • China Rural-based poverty-reduction strategies Source: Author. families. The evaluations are considered to have been a ma- tive currently supports some 90 experimental and qua- jor contributing factor in convincing the new government si-experimental IEs in 20 African countries.� At least 40 that came to power in 2002 to continue these programs countries in Asia, Latin America, and the Middle East are started by the previous administration. The evaluations al- now implementing sectoral approaches to IE with World so convinced policy makers of the technical feasibility and Bank support. A number of international evaluation initia- policy value of rigorous IEs and contributed to the passage tives are being promoted through collaborative organiza- of a law by Congress in 2007 mandating the evaluation tions such as the Network of Networks for Impact Evalua- of all social programs. This law also created the National tion (NONIE)� and the International Initiative for Impact Commission for the Evaluation of Social Programs, which Evaluation (3IE).� was assigned the responsibility for regulating the develop- ment of M&E functions in the social sectors. The third pathway In the third pathway, a program of IEs was developed as Africa example one component of a whole-of-government system, man- The World Bank Africa Impact Evaluation Initiative aged and championed by a strong central government (AIM) is a regional initiative to help governments strength- agency, usually the ministry of finance or planning. Chile en their overall M&E capability and systems. The initia- is a good example of a national M&E system with clearly 141 M&E SYSTEMS AND M&E TOOLS defined criteria and guidelines for the selection of programs Although the creation of evaluation capacity to provide to be evaluated, their conduct and methodology, and how high-quality evaluation services is important, these supply- the findings will be used (Mackay 2007, 25–30; Bamberg- side interventions will have little effect unless there is suffi- er 2008). Starting in 1994, a system of performance indi- cient demand for quality IE. This requires that quality IEs cators was developed; rapid evaluations of government pro- are seen as important policy and management tools in at grams were incorporated in 1996; and, in 2001, a program least one of the following areas: of rigorous impact evaluations was introduced. There are two clearly defined IE products. The first are rapid ex post • Budget resource allocation evaluations that follow a clearly defined and rapid commis- • Policy formulation and analytical work sioning process, where the evaluations are used by the min- • Management and delivery of government services istry of finance as part of the annual budget process. The • Accountability second are more comprehensive evaluations that can take up to 18 months and cost $88,000 on average. The strength Creating demand requires powerful incentives within of the system is that it has defined and cost-effective proce- government to commission IE, create a good level of qual- dures for commissioning, conducting, and reporting of IEs; ity, and use IE information intensively. A key factor is a a defined audience (the Ministry of Finance); and a clear- public sector environment supportive of the use of evalu- ly understood use (the preparation of the annual budget). ation findings as a policy and management tool. To gener- The disadvantages are that the studies cover only issues of ate demand for IE among potential users, MacKay (2007, interest to the Ministry of Finance and the buy-in from the Table 11.1) suggests some such positive incentives as build- agencies being implemented is typically low. ing ownership and financing to build better IE systems; sanctions and threats, such as laws and decrees mandating Integrating IE into sector and national M&E and the planning, conduct, and reporting of IE; and positive other data-collection systems messages from key figures, such as high-level statements of The successful institutionalization of IE requires that selec- endorsement and conferences on good IE practice. tion, design, and use are integrated into national M&E sys- tems and data-collection programs. This is critical for sever- Capacity development for IE al reasons. First, much of the data required for an IE can be There has been an increasing awareness of the importance of obtained efficiently and economically from program M&E evaluation capacity development. This is largely due to the systems. Second, IE findings that are widely disseminated recognition that many past efforts were overambitious and provide an incentive for agencies to improve the quality of had disappointing results, often because they focused too M&E data they collect and report. For example, the Min- narrowly on technical issues without understanding the insti- istry of Education in Uganda reported that the wide dis- tutional and other resource constraints faced by many coun- semination of the Education for All evaluations contribut- tries. ODA agencies now recognize that institutionalization ed to improvements in the quality of monitoring reporting. of IE will require an evaluation capacity development plan to Third, national household survey programs provide valu- strengthen the capacity of key stakeholders to fund, commis- able sources of secondary data for strengthening method- sion, design, conduct, disseminate, and use IE. This involves ological rigor of IE design and analysis (for example, the strengthening the supply of resource persons and agencies able use of propensity score matching to reduce sample selec- to deliver high-quality and operationally relevant IEs and de- tion bias). Some IEs have cooperated with national statisti- veloping the infrastructure for generating secondary data to cal offices to piggyback information required for the IE on- complement or replace expensive primary data collection. to an ongoing household survey or to add a special module. An evaluation capacity development strategy must tar- Creating demand for IE get (1) agencies that commission, fund, and disseminate Efforts to strengthen IE and other kinds of M&E sys- IEs; (2) evaluation practitioners who design, implement, tems often are viewed as technical fixes—mainly involving and analyze IEs; (3) evaluation users and groups affect- better data systems and the conduct of quality evaluations. ed by the programs being evaluated; and (4) public opin- 142 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K ion (Bamberger 2006). The active involvement of leading to pressure agencies to improve services; (3) alerting agen- national universities and research institutions is critical as cies to the need to incorporate vulnerable groups and ad- these institutions can mobilize leading national research- dress community conflicts caused by limited access to ser- ers, and they have the resources and incentives to work on vices; (4) providing political cover to government to take a refining existing and developing new research methodol- politically sensitive decision, and showing how to mitigate ogies. Through teaching, publications, conferences, and negative consequences for influential “losers” from the pol- consulting, they can strengthen the capacity of policy mak- icy change; (5) developing methodologies to systematical- ers to identify the need for evaluation and to commission, ly document problems in the use of public funds that were disseminate, and use findings. widely suspected but that have not been possible to doc- An important role of evaluation capacity development ument; and (6) providing empirical evidence to convince is to help agencies design “evaluation-ready” programs and new administrations to continue major programs initiated policies. Many programs generate monitoring and other by previous administrations (Bamberger 2008). administrative data that could be used to complement or Bamberger (2008) suggests various ways to strengthen replace the collection of survey data, or to provide proxy the use of the findings of impact evaluations: baseline data in cases in which an evaluation started too late to have been able to conduct baseline studies. Often, • Understand the political context within which the however, the data are not collected or archived in a way evaluation is conducted. that makes it easy to use for evaluation purposes. Cooper- • Time the launch and completion of the evaluation ation between program staff and evaluators can greatly en- appropriately. hance the utility of project data for the evaluation. In oth- • Decide what to evaluate. A successful evaluation er cases, changes in project design or implementation could usually focuses on a limited number of critical is- strengthen the evaluation design. For example, in some cas- sues and hypotheses based on a clear understand- es, a randomized control trial could have been used, but ing of the clients’ information needs and of how the evaluators were not involved until it was too late.�� evaluation findings will be used. • Base the evaluation on a program theory model Promoting IE so that evaluators, clients, and stakeholders share There is widespread concern that even for evaluations that a common understanding of the problem the pro- are methodologically sound, the use of evaluation findings gram is addressing, the objectives, how they will be is disappointingly limited and the process of IE is not in- achieved, and what criteria the clients will use in as- stitutionalized (Bamberger, Mackay, and Ooi 2004). In sessing success. addition to factors that are external to the evaluation, the • Create ownership of the evaluation. reasons many include (1) lack of ownership, (2) lack of • Include an ECD strategy. understanding of the purpose and benefits of IE, (3) bad • Develop a follow-up action plan to monitor wheth- timing, (4) lack of flexibility and responsiveness to the in- er agreed decisions are implemented. formation needs of stakeholders, (5) wrong question and irrelevant findings, (6) weak methodology, (7) lack of local One of the factors contributing to evaluation underuti- expertise to conduct, review, and use evaluations, (8) com- lization may be client disagreement with, or lack of un- munication problems, and (9) lack of a supportive organi- derstanding of, the evaluation methodology. Therefore, IE zational environment. should be combined with process analysis and formative Program evaluations can be influential in many differ- evaluation. Even when the primary objective is to assess re- ent ways, not all of which were intended. Some of the dif- sults, it is important to help the client understand how out- ferent kinds of influence that IEs can include are (1) alert- comes and impacts are affected by what happens during ing public service agencies to problems of which they were program implementation. Process and formative evaluation not fully aware or had not considered important; (2) pro- also provide rapid feedback to the client, increase owner- viding objective, quantitative data that civil society can use ship, and improve perception that the evaluation is useful. 143 M&E SYSTEMS AND M&E TOOLS Conclusion ers and line ministries. This also requires a commitment to ODA agencies are facing increasing demands to account evaluate capacity development, and the development of a for the effectiveness of the resources they have invested in wider range of IE methodologies that are designed to re- development interventions. This has led to an increased in- spond more directly to country needs rather than to impose terest in more rigorous evaluations of the programs and methodologically rigorous evaluation designs that often are policies these agencies support, and an increasing num- of more interest to ODA research institutions than to de- ber of methodologically rigorous impact evaluations have veloping country governments. been conducted, particularly in Latin America. However, For their part, developing country governments must less progress has been made toward institutionalizing the invest the necessary time and resources to ensure that they selection, design, and use of IE. Consequently, many evalu- fully understand the potential benefits and limitations of IE ations have not achieved their full potential as instruments and the alternative methodologies that can be used. Usual- for budget planning and development policy formulation. ly one or more ODA agencies will be willing to assist gov- Successful institutionalization of IE requires that the ap- ernments wishing to acquire this understanding, but gov- proach be adapted to the political and administrative sys- ernments must seek their own independent advisors so that tems of each country. This paper identifies three quite dis- they do not depend exclusively on the advice of a particular tinct pathways toward institutionalization. Conducting a donor who may have an interest in promoting only certain number of rigorous IEs in a particular country does not types of IE methodologies. guarantee that agencies will automatically increase their de- mand for evaluations. As a result, a strategy is needed to cre- Michael Bamberger ate demand. While it is essential to strengthen the supply of Social Development and Program Evaluation Consultant evaluation specialists and agencies able to implement eval- uations, experience suggests that creating the demand for evaluations is equally if not more important. This requires a Building a Monitoring and Evaluation System combination of incentives, sanctions, and positive messages Based on Results for Social Development from key figures, and that IE be seen as an important policy and management tool in such areas as resource allocation, Introduction policy formulation, and analytical work, delivery of govern- Results-based management is designed to improve the per- ment services, and underpinning accountability. formance of public institutions seeking greater efficiency Evaluation capacity development is a critical compo- and effective use of resources. At least three objectives are nent of IE institutionalization. Some groups require the pursued through results-based management. The first is to capacity to design and implement IE; others need to un- direct government actions in the provision of goods and derstand when an evaluation is needed and how to com- services to achieve measurable goals efficiently. The second mission and manage it, whereas others must know how to is to promote evidence-based decision making. The third is disseminate and use the evaluation findings. Many well-de- to try to meet the demands of society regarding transparen- signed and potentially valuable IEs are underutilized be- cy and accountability. cause of the lack of ownership by stakeholders, bad timing, A number of member countries of the Organisation failure to address client information needs, lack of follow- for Economic Co-operation and Development (OECD) up on agreed actions, and poor communication and dis- have used this approach. Since 2000, more than 75 per- semination. A strategy to promote utilization is essential for cent of OECD member countries have reported using per- IE institutionalization. formance information in budget processes (ILPES-CE- ODA agencies will continue to play a major role in pro- PAL 2003). Countries with the most experience include moting the institutionalization of IE. This requires a com- the United Kingdom, with the introduction in 1982 of the mitment to move from ad hoc support of individual IEs Financial Management Initiative, and New Zealand, with to a genuine commitment to helping countries develop an the introduction in 1984 of a system of purchase agree- IE system that serves the interests of national policy mak- ments between ministers and autonomous executive bodies 144 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K that contain terms of reference expressed as results indica- the institutional and regulatory framework, (3) the new tors. Subsequently, in 1993, the United States approved the model for the expansion and consolidation phase and, fi- Government Performance and Results Act, which seeks to nally, (4) the conclusions. install a budget system based on performance and manage- rial flexibility, and which concludes with a programmed re- Building a results-based monitoring and porting process that is delivered to Congress to support the evaluation system, 2004–07 budget discussion (Shack Yalta 2003). For many years, social programs lacked the mechanisms to More recently, Argentina, Colombia, and Chile have ensure accountability and provide evidence of effectiveness. made similar efforts (World Bank and OECD 2006; May The first efforts, still few and far between, to quantify rig- et al. 2006). Chile’s effort is one of the most important orously the benefits of government intervention include Latin American examples. Its origins date from the early an impact assessment of the Education, Health, and Nu- 1990s, when a pilot scheme of the Plan for the Moderniza- trition Program (Programa de Educación, Salud y Alimen- tion of Public Management was launched to phase in a re- tación, PROGRESA) between 1997 and 2000. The obli- sults-based budgeting system. The Chilean model now has gation to evaluate programs annually subject to operating a management system linked to the budget with such tools rules and quarterly reporting of indicators subsequently has as performance indicators linked to targets, an evaluation been introduced. system for programs and institutions, a management im- SEDESOL asked the World Bank to undertake a diag- provement program linked to performance incentive pay- nosis of existing information systems and the ministry’s in- ments to public officials, and a competitive fund linked to stitutional capacity, and to develop an M&E system that the logical framework matrix, which, among other things, would facilitate progress toward results-based management. is used for ministries (and programs) to compete for fund- The World Bank’s diagnosis concluded that SEDESOL did ing (Guzmán 2004). indeed have the necessary foundation for moving toward In Mexico, in 1999, the Budget for Federal Spend- results-based management. However, it was necessary to ing established the obligation to perform annual exter- modify the following aspects: nal evaluations of all programs subject to operating rules. Since then, the Social Development Secretariat (Secretaría • Incipient linkage of delivery of program products de Desarrollo Social, SEDESOL) has striven to incorporate and services to the measurement of results a management approach geared toward achieving results. • Abundant production of operational information The approach emphasizes the development and strength- (products), not necessarily complemented by infor- ening of the functions and tools for M&E. Initially, this ef- mation on results (outcomes, impacts) fort focused on the annual external evaluation of the pro- • Multiplicity of administrative, operational, and grams subject to operating rules. In addition, since 2004, management data systems the secretariat has promoted the construction of a results- • Many reports on indicators to different external in- based M&E system. The system is designed to strength- stitutions without any clear use for managing the en the evaluation component with rigorous measurement programs of the impact of the programs and to develop a pilot mon- • Design of systems for reporting activities, but with itoring device based on results. SEDESOL is now ready limited information and usage for feedback to to begin a second phase of the system under a new model those responsible for decision making with greater scope. To that end, it has identified important • Disconnect between strategic, operational, and cross-sectional challenges in the institutional, technical-op- budgetary planning erational, and data-processing spheres. • Need to create incentives at midhierarchical levels This article describes SEDESOL’s efforts at implement- to promote results orientation ing results-based management. It is divided into four sec- tions that detail (1) the first phase of the results-based mon- Based on this diagnosis, an M&E system was designed itoring and evaluation (M&E) system, (2) the change in to support results-oriented management, considering three 145 M&E SYSTEMS AND M&E TOOLS main components: (1) evaluation of annual results, (2) im- FIGURE 28: pact assessment, and (3) pilot implementation of a system of Results-Based M&E System, 2004–07 indicators for monitoring focused on results. The conceptu- alization of the three components in a results-based moni- Improvement of living Impact conditions (long-term toring and evaluation system (RBMES) shifts the emphasis results) Impact of traditional management from input to achieving results. Evaluation Intermediate effects of Figure 28 shows the interaction of components and their products on beneficiaries (short and medium-term scope, all of which focus on results-based management. Purpose results) Management for Results Annual evaluation of results Goods and services In keeping with current regulations, an evaluation report produced and offered should be prepared each year by an external evaluator for Products by the program programs subject to operating rules (see Table 20). The Evaluation of evaluation approach was primarily for accountability in the Tasks and actions Results and Traditional monitoring undertaken in program Monitoring fiscal year concerned with meeting the program objectives, implementation of Indicators Activities operating rules, and physical and financial targets. Addi- tionally, an analysis of economic and social benefits was re- Financial, human and material resources of quested whose scope was limited by the schedule for the the program delivery of evaluation reports. A feedback of evaluation re- Inputs sults was promoted through the external evaluations track- ing system that had a work plan set up to address the rec- Source: Author. ommendations from the evaluations. Food Support Program, Fortified Liconsa Milk, Micro- Multiyear evaluation of impact regions Strategy, and Cement Floors. The results of these To complement these findings, various medium- and long- evaluations are public and have been used to inform deci- term evaluations were conducted to quantify the causal re- sions about adjustments to the design, expansion of cover- lationships between actions implemented by the programs age, and continuity of programs. and different dimensions in changes to recipients’ well-be- ing. The impact assessments attempted to provide robust Results-based monitoring evidence of the effectiveness of various interventions in Although periodic reports on indicators of the physical and child nutrition, school attendance, family income, access to financial progress of the programs were produced, they were basic services, and the like. These evaluations require more not systematic or focused on achieving results. Based on the investment in terms of budget, a higher level of expertise diagnosis undertaken by the World Bank, we began by im- for their analysis, and a longer maturation time for inter- plementing a pilot phase of the results-based monitoring ventions to observe the expected impacts. Consequently, component. The pilot phase included four programs (Pro- they were performed selectively for strategic-type programs ductive Options, Social Co-investment, Habitat, and Food and activities such as Opportunities, Habitat, Your Home, Support) with different degrees of institutional progress in TABLE 20: Annual External Evaluations, 2004–07 2004 2005 2006 2007 SEDESOL-sponsored programs 7 8 8 11 Programs operated by decentralized bodies or sector entities 6 8 9 10 Total 13 16 17 21 Source: Author. 146 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K the collection of indicators and data systems. This phase in- (Secretaría de la Función Pública, SFP), and CONEVAL cluded extensive training in the methodology of the logical was introduced to move toward the establishment of a link framework matrix (LFM), the construction of matrices, the (at national and sectoral levels) between planning, bud- conceptual design of the system, and the development of geting, and the implementation of federal programs using a computer platform. LFMs are public management tools M&E tools. In particular, this new approach is embodied for logically and systematically presenting the objectives in the federal administration through the creation of the of a program, the causal relationships of its components, performance evaluation system and based on the National and factors outside the program that can influence its ob- Development Plan (NDP) and sector programs. jectives. LFMs were developed through participatory work- The guidelines for evaluation and budgeting raise the shops with the leaders of the design, operation, and evalu- need for a framework of reference in management that con- ation of programs. These workshops (1) reached consensus siders the strategic objectives of the country and sectors and on the overall goal of each program and its correspond- their alignment and articulation. Likewise, the guidelines rec- ing logical model (string of results: activities, outputs, out- ommend that the agencies responsible for programming the comes, and impacts); (2) identified program indicators and budget use performance data, and promote institutional im- goals; and (3) determined the sources of information for provement based on performance incentives. Added to these monitoring indicators. The introduction of the LFMs led to guidelines is the agencies’ increased accountability through an improvement in the relevance and accuracy of the indi- more and better performance data for budget discussions. cators. Once the indicators had been identified and targets In this context, as of late 2007, the SHCP coordinated set, these were concentrated in a control panel. The panel an effort to implement results-based management from the showed data using traffic lights to indicate the progress of perspective of the performance evaluation system whose the indicators with regard to established targets. objectives include the following: Since 2007, SEDESOL has moved toward the defini- tion and use of LFMs in 21 programs subject to operating • Verify and monitor fulfillment of goals and objec- rules. This has meant a further step toward the improve- tives, based on strategic and management indica- ment and standardization of the formulation of indicators tors that show the results of programs and targets, and establishing a more solid foundation for • Link planning, programming, budgeting, monitor- performance measurement of its programs. ing, use of resources and evaluation of public poli- cies and budget programs Changes in the regulatory and institutional framework This resulted in the general guidelines for moving to- The two main reasons behind the need for a second phase ward a results-based budgeting and performance evaluation of the RBMES involve the reestablishment of the Minis- system. A matrix of indicators (or an LFM) is implanted in try’s results-oriented model. The first relates to the experi- this system as a tool for strategically planning budget pro- ence gained from pilot implementation of the monitoring grams. In addition, an exercise is carried out to link the ma- component. The second has to do with the need to adapt trix indicators of all programs subject to the federal bud- the RBMES to a new regulatory and institutional context get’s rules of operation. marked by the reforms of the national budget, the creation All the matrix indicators are concentrated through a of the National Social Policy Evaluation Council (Conse- computer platform, from the operating level to impact lev- jo Nacional de Evaluación de la Política de Desarrollo Social, els (and their connection with the strategic objectives of the CONEVAL), and the need to adopt the guidelines of federal agencies and the NDP), which are periodically monitored initiatives currently under development as the performance and reported to ascertain the degree of progress in meeting evaluation system and the Results-based Budget (RBB). associated goals. By mid-2007, clear guidance from the Ministry of Fi- To ensure the strategic management of information, nance and Public Credit (Secretaría de Hacienda y Crédi- the SHCP, in coordination with each agency, selected from to Público, SHCP), the Ministry of Public Administration the matrices a subset of key indicators to monitor program 147 M&E SYSTEMS AND M&E TOOLS budgets. The President’s Office also tracks the indicators analysis of causality between major programs operated and and targets of sector programs. This new institutional and coordinated by SEDESOL and its effects, as well as incor- regulatory framework established a different context for the porating new tools such as evaluations of consistency and development of the RBMES. This context is in stark con- results and specific performance details. trast to the previous one based on an individualized focus To improve the quality of spending, the results-based on the programs, particularly their operation. Previously, a budgeting component introduces mechanisms to ensure broader frame of reference to link strategic sector planning adequate planning and feedback of budget decisions with and the budget was lacking, as were strong mechanisms to performance data from M&E. The balance in the joint op- stimulate demand and the use of performance data by the eration of these components will facilitate and encourage presidency, the SHCP, SFP, and CONEVAL. the monitoring of the ministry’s aggregate results, by creat- ing incentives for continuous improvement. Additionally, A new RMBES model for the expansion and this approach strengthens strategic planning within social consolidation phase, 2008–12 policy management and improves accountability of results, In response to the institutional and regulatory changes, both internally and externally. there was a rethinking of the RMBES. The new model moves from an approach based essentially on the operation Objectives of RBMES of programs to a larger scheme focused on the aggregate The overall goal is to make operative within the social de- performance of the social development sector. This scheme velopment sector the RBB initiative and performance eval- is based on measuring performance through the strategic uation, under an approach that seeks the achievement of dimension of the Social Development Sector program and sector targets, through an RBMES. It aims to advance the its alignment with the NDP. Under this approach, the re- implementation and consolidation of the field as a tool of design of the RMBES is intended to operate the budgeting governance that will (1) monitor the integrated physical for results initiative and the performance evaluation system and financial progress of programs; (2) establish a unified within the SEDESOL, incorporating their guidelines and framework, based on priority criteria for the development purposes. This new approach aims to consolidate the prog- of annual and impact assessments to determine their effect ress achieved, correct any problems encountered, and com- on the improvement of the beneficiaries’ quality of life; (3) plement its scope by incorporating other areas of manage- provide feedback for decision making about the operation, ment evaluation (including the budget). resource allocation, and program design; and (4) facilitate The process of rethinking the system considered (1) re- and encourage the management of the secretariat in rela- sults-based monitoring, (2) results-based evaluation, and tion to sector and national planning. (3) a results-based budget (see Figure 29). The develop- The aim is to advance the implementation and consol- ment, strengthening, and integration of these three compo- idation of the RBMES as a tool of governance that will nents provide a better balance in the process of generating permit (1) integrated tracking of the physical and financial information about performance and facilitate its use with- progress of the programs; (2) the establishment of a uni- in the decision-making processes. The first step will bind fied framework, based on priority criteria for the develop- the budget to adequate planning of sector targets and out- ment of annual evaluations and impact assessments to de- comes. Monitoring will seek to be a continuous function to termine their effect on the improvement of the quality of strengthen the operation. And, evaluation will be a selec- life of beneficiaries; (3) feedback for decision making about tive activity based on the criteria of priority policies and a the operation, resource allocation, and design of programs; search for areas to improve. and (4) facilitating and stimulating the ministry’s manage- The monitoring component makes it easier to track the ment to plan at the sector and national levels. operation of programs and their results, a task that focuses Specific objectives include the following: mainly on the area of the Social Development Sector’s strat- egies and plans. With the evaluation component, progress • Establish a permanent, progressive link between is being made in the consolidation of methodologies for the planning, budgeting, implementation, and M&E 148 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K FIGURE 29: Results-Based M&E System, 2008–12 Results-based Monitoring and Evaluation System Sector Level Results-based Results-based Results-based Budget Monitoring Evaluation Priority Plan-Budget Studies and Indicators Integration Evaluations Budget by Impact Indicator Component Matrices Program Level Budgetary Specifics Advance Consistency and Results Others Information Agenda Capacity Development Institutional Organization Source: Author. processes, based on selected output, outcome, and various structures, as well as the coordination and impact indicators, and sector and social develop- decision mechanisms required. ment goals, to improve their alignment with the • Encourage a new organizational dynamic within the NDP and Sector Program. SHCP and coordinated entities, linking the areas re- • Improve the quality, availability, and timeliness of sponsible for planning, programming, and budgeting, M&E data that is useful for decision making at the as well as M&E, with the programs’ operating units. strategic and management levels. • Build the M&E capacity of the programs operated • Coordinate and balance the functions of monitor- and entities coordinated by SEDESOL to produce ing, evaluation, and accountability, to guide the reliable, timely information for decision making on management of the SHCP toward achieving re- programming and budgeting. sults, by improving the design and operation of so- cial programs and the quality of spending. Conceptual design • Strengthen the institutional quality of the system, The conceptual design of the RBMES is structured around by specifying the roles and responsibilities of the the first three pillars of (1) monitoring, (2) evaluation, and 149 M&E SYSTEMS AND M&E TOOLS (3) RBB as well as the system’s transversal aspects of (4) in- ferent types of evaluation, such as (1) evaluations of consis- stitutional structure required to support the RBMES, (5) in- tency and results or design, (2) specific performance eval- stitutional capacity building for M&E, and (6) the informa- uations, (3) impact evaluations for strategic programs, (4) tion agenda. evaluations of processes, and (5) specific evaluations of per- ception of beneficiaries and monitoring of public works. Results-based monitoring Areas for improvement that show up in the evaluation A set of 16 strategic indicators were selected to moni- results are tracked so that they effectively contribute to im- tor progress in meeting goals set by SEDESOL in the So- proving social programs. An institutional work plan and cial Development Sector program and the NDP. The selec- specific work plans for each program set out the measures, tion took into account the different levels of information persons and areas responsible, dates, and deliverables to ad- (monthly, quarterly, yearly, and five yearly) and objectives dress the areas for improvement identified in the external (results, components, and budget). These indicators were evaluations. selected strategically from the Social Development Sector program and the result-based matrix indicators to satisfy the Results-based budget demand for information that is relevant to the aggregate of The purpose of an RBB is for the definition of budget- individual decision makers in a timely manner. The main ary programs to emerge from a sequential planning pro- user of the indicators report at this level is the President’s cess: scheduling and setting goals, indicators, and targets Office, through its strategic indicators tracking system. geared toward the achievement of sector goals. This pur- At the program level, as in the rest of federal govern- pose requires the evidence provided by M&E to make the ment, the LFM provides a sound methodology for the se- RBMES consistent with the performance evaluation sys- lection of indicators for management oriented to program tem and RBB guidelines. This involves establishing con- results. The actor requesting monitoring and reporting of gruity between outputs and outcomes, on the one hand, indicators of the results base matrix indicators is the SHCP, with the budget to establish a link between the latter, sec- through its Web portal. tor program priorities, and evidence of results. The proce- Currently, there are matrices for 19 programs subject to dure is an attempt to identify where SEDESOL’s expendi- operating rules and about 500 indicators: 112 are linked to ture is directed and, based on M&E, to learn whether the results, 170 are related to the quantity, efficiency, and qual- expected results are being obtained with the budget alloca- ity of goods and services provided by programs, and the re- tions for the goals set. stare activities necessary for production. Of these indica- The initial phase of RBB is defined as presentation. At tors, 40 were identified as priorities for the development this stage, the budget relates to goods and services provid- of the 2010 draft expense budget and budget monitoring. ed by various social programs, allowing the alignment of These indicators are mainly at the component level and re- programs and their budgets for the objectives of the sector flect the priority goods and services provided by social pro- to be checked. Specifically, the RBB was linked to the oth- grams and directly related to the availability of funds for er two pillars of the RBMES through the draft 2010 bud- each program. get. The latter took into account the 40 priority indicators (linked to monitoring) and the evidence provided by exter- Results-based evaluation nal evaluations that the goods and services provided con- Considering the importance of the Sector Program and the tribute to the achievement of expected results, that is, to NDP, evaluations are planned on the basis of (1) the amount the purpose and end (linked to evaluation). of public funds involved, (2) the size and characteristics The connection with the budget must be contextual- of the target population, (3) the importance to the sector, ized by considering the difference between the time frame (4) the newness of the program or policy, and (5) the po- for the budget activity, on the one hand, and the length of tential for application and replication of the lessons learned. time required for the goods and services to materialize and This planning defines evaluation measures for social for the expected effects to mature and be measured, on the programs in the short and medium term, considering dif- other hand. While the budget period is almost instanta- 150 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K neous, building a water network, for example, can take sev- the first stage, a system with a greater emphasis on evalua- eral months, and measuring the change in the incidence of tion was conceptualized, with a focus on the management diarrheic diseases can take even longer. Additionally, con- of individual programs. sider that the budget-results relationship is neither direct In response to recent changes in the regulatory and in- nor automatic, so that the RBMES should be regarded as a stitutional framework and lessons learned in the first stage, tool that supports rather than determines decision making. the model was redesigned. This model seeks to incorpo- rate a vision of sector performance, linking M&E with the Information agenda planning and budgeting process and using explicit critical To stimulate continuous improvement, one of the prereq- transversal components, including an information agenda uisites of the RBMES is the strategic use of information on and institutional organization and capacity. spending, accountability, and management feedback. This Building a results-based management system is a con- implies the availability of quality, basic, and accessible in- tinuous medium- and long-term process that requires a formation for all programs, and it facilitates learning the combination of technical and political elements. It means management results and informed decision making. generating relevant, timely, and quality information to be It therefore is important to promote an information used in decision making. It requires shaping the institu- agenda that includes regulatory bases and standards to gen- tional bases to allow connections among planning, pro- erate, collect, and process performance data, as well as qual- gramming and budgeting, implementation, and evaluation ity-control mechanisms. The information agenda should activities. An incentive scheme aimed at achieving results seek to determine the parameters of interoperability of ex- and mechanisms is important to ensure the sustainability isting technology tools and the connectivity of the offices, of the initiative. entities, and delegations. Gerardo Franco Institutional organization Secretaría de Desarrollo Social, Sedesol, Mexico It is important to consider the organizational structure to strengthen the integration of planning and budgeting pro- cesses, and to establish coordination mechanisms for insti- Impact Assessment of Regional Development and tutional monitoring tasks. Likewise, it is essential to set up Peace Programs in Colombia an institutional framework that ensures that information on performance results provides feedback for the decision- Building an Evaluation Approach for Assessing making processes and ensures the sustainability of these ef- Change in Social Capital and Conflict forts in the long term. Management Institutional capacity development Introduction To consolidate a culture of results-based management at Policy makers increasingly are faced with the challenge of SEDESOL, it is necessary to build institutional capacity. evaluating initiatives that attempt to bring about social To achieve this goal, attempts will be made to (1) identify change in “intangible” aspects. Designing social interven- methodologies, including basic manuals for the construc- tions is complex, since their objectives often go beyond the tion of indicators and the application of assessment meth- classic concepts of poverty and well-being. To achieve true ods; (2) disseminate the conceptual underpinnings and change, social interventions must bear in mind and cope tools of the system; and (3) train public officials in results- with the complexity of the social context and the sociocul- based management and M&E. tural dimensions of the agents. Thus, in addition to reduc- ing poverty, social programs increasingly seek to produce Conclusion effects in values, aptitudes, perceptions, and motivations Since 2004, M&E has evolved into systematic activities of its beneficiaries to modify the structure of the social fab- that support a results-based management system. During ric and to improve relations within communities, between 151 M&E SYSTEMS AND M&E TOOLS the agents and the market and between the agents and the neous set of projects in the areas of (1) productive develop- state. To a certain extent, this requires assuming a (neo)in- ment, (2) institution building, (3) human rights, and (4) stitutional approach to the interpretation of development culture of peace. Using a bottom-up participatory model, processes, in which the modification of the formal and in- the projects are promoted and implemented by grass-roots formal rules of the game that define how agents interact social organizations and express the views, priorities, and with each other is as important as their objective condi- identities of local communities (“the life we want”). Bene- tions. The renewed popularity of concepts such as social ficiaries are usually members of the same grass-roots social capital, social networks, capabilities, and empowerment is organizations. In practice, the RDPPs promote the creation proof of the fact that these dimensions of development pro- of a coordinated structure of social networks among com- cesses play a key role in the political discourse and the de- munity members, among organizations, and between orga- sign of social programs, and should therefore also be in- nizations and local public institutions. cluded in their evaluation. The RDPPs’ work has been carried out with financ- In this paper, I will develop some ideas on the integra- ing from various sources. In particular, Peace and Develop- tion between quantitative and qualitative methods for eval- ment and Peace Laboratories represent joint a joint effort uation, on the basis of the experience of the impact assess- by Colombia’s national government and international co- ment of the Regional Development and Peace Programs operation to support the RDPPs. Peace and Development (RDPPs) in Colombia.69 The title refers to the urgent need functions as a result of a loan granted to Colombia by the to adjust our evaluation tools to record the effects of pro- World Bank, and the Peace Laboratories are a contribution grams on sociocultural dimensions, what in Colombia we by the European Union to these initiatives. The two cur- defined as relationship patterns or “modes of relationship.” rently support six RDPPs in Magdalena Medio, Norte de An evaluation approach based on classic multidimensional Santander, Oriente Antioqueño, Macizo Colombiano and indicators of poverty runs the risk of omitting these essen- Alto Patía, Meta, and Montes de María.70 In these regions, tial dimensions. It is here that the integration of evaluation the RDPPs finance nearly 1,400 projects, implemented by methods becomes essential, as complementing qualitative approximately 800 grass-roots organizations in 120 munic- and quantitative approaches enables one to reflect the com- ipalities, with an estimated population of 180,000 benefi- plexity of the processes of social change under way. ciaries. From a quantitative perspective, it is possible to ad- vance toward the development of new instruments to eval- How to evaluate the regional development and uate sociocultural dimensions, without diverging from the peace programs? classical framework of rigorous impact assessment. In this To encourage debate and reflection on the contributions and presentation, I will concentrate on certain aspects of the achievements of the RDPPs, in 2005 the Colombian gov- progress achieved precisely for this purpose in evaluating ernment, through the National Planning Department, be- the RDPPs in Colombia. gan implementing an ambitious evaluation plan.71 The eval- uation was intended to apply a rigorous methodology and Regional development and peace use quantitative indicators to assess the programs’ impact. programs in Colombia The initial challenge for evaluating the programs in- The RDPPs are networks of organizations that seek to cre- volved developing a suitable theoretical approximation for ate conditions of peace and development in the regions evaluating the expected changes. The complexity of the most affected by violence and poverty. They posit that task was due to the peculiarity of the intervention model. peace is the result of the consolidation of fair and inclu- The RDPPs have ambitious, long-term objectives, embed- sive development models, and they defend the principles ded in the concepts of integral development and sustain- of respect for life, human dignity, solidarity, and participa- able peace. Their impact strategy is both integral and mul- tory democracy. tifaceted. In particular, the RDPPs seek to have successive The RDPPs play a similar role to that of “social funds,” levels of incidence, since they focus on a limited number of whereby a stock of resources is used to finance a heteroge- direct beneficiaries, and expect to modify territorial con- 152 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K ditions (municipalities and regions) through propagation find answers to the following question: How are the RDPPs mechanisms. Finally, the programs promote intangible pro- modifying the living conditions of their direct beneficia- cesses that are difficult to measure. ries and their “modes of relationship” with the communi- The first step in undertaking the evaluation involved ty, toward the conflicts and toward public institutions? This drawing up and reaching agreement on a “change theory” question is thought to deal appropriately with the RDPPs’ model of the programs that would make it possible to out- transversal aims, regardless of the heterogeneity of the proj- line the main assumptions, logical stages, and channels of ects financed. action of the intervention (Figure 30). The next step in the construction of the “change theo- This conceptual reference framework posits that the ry” model involved designing an evaluation strategy and a conditions of violence and poverty characterizing the re- battery of instruments that would permit the measurement gions targeted by the RDPPs are directly linked to nega- of the changes achieved in the dimensions concerning con- tive “modes of relationship” (distrust, clientelism, and indi- ditions and “modes of relationship.” Whereas the first is- vidualism, among others), creating a vicious circle. Within sue can be dealt with a classic evaluation approach based this framework, the RDPPs’ proposal consists of support- on quantitative indicators, it is more difficult to evaluate ing specific projects that encourage new “modes of rela- the effects on relationship patterns from a rigorous per- tionship” based on positive values among direct beneficia- spective. To that end, two complementary methodological ries (for example, trust, reciprocity, and collective action, approaches were used. On the one hand, the traditional among others). It is expected that these new relationship framework of quantitative impact assessment was extended patterns, based on the dynamics of a virtuous circle, will by using a set of “novel” measurement instruments. On the lead to positive changes in the socioeconomic and security other hand, this main strategy was complemented by the conditions of the participants in the direction of building qualitative methods with which it was combined. peace and development. The model assumes that this vir- The quantitative strategy is based on a quasi-experi- tuous circle would extend slowly and sustainably, spreading mental design that permits a comparison of the outcomes from the projects’ beneficiaries to their communities and of interest between a group of households benefiting from local institutions, positively affecting an increasingly large the RDPPs (treatment group) and a group of nonbeneficia- population, and achieving a regional impact. ries (control group) in the same municipalities. The plan Based on this conceptual framework, the evaluation includes two data collection rounds: a base line and a fol- methodology focuses on establishing the programs’ contri- low-up. The first data collection occurred between Novem- bution at the level of direct beneficiaries, and attempts to ber 2006 and February 2007, while the follow-up will take place in 2010.� The design enables effects of the RDPPs to FIGURE 30: be estimated by using difference-in-difference models, and RDPP Change Theory by comparing beneficiaries among themselves through in- tensity models. In addition to a structured survey, the inno- vative element included two extra instruments used to mea- sure aspects associated with “modes of relationship:” a set of Initial conditions experimental games and a social dilemma. In what follows Violence and underdevelopment Traditional ways I describe these instruments and discuss how they can be- of relating come a tool to measure the impacts of social interventions on sociocultural dimensions. Social dilemma New conditions New ways Peace and of relating Social dilemmas are information-gathering exercises in development which participants are asked to describe options in a story narrated in the third person. They allow an exploration of Source: DNP 2009. the categories of right and wrong, or the processes of moral 153 M&E SYSTEMS AND M&E TOOLS deliberation on topics for which using direct survey ques- tions may yield biased answers. Within the RDPPs’ eval- BOX 7: uation framework, a social dilemma was used to measure Alternative responses to the social dilemma changes obtained in the modes of relationship toward con- flicts (conflict management). For this end, the story of Juana, 1. “Juana should meet the illegal group on her own, a housewife and farm worker was developed to illustrate a di- to find out for herself why they want to kill her.” lemma. Boxes 6 and 7 review the story and the dilemma. 2. “If the community speaks to those making the ��At the end of the story, Juana, faces a dilemma in threats, Juana can maintain her peace of mind and which she has to choose from a series of alternatives to deal continue her political activity.” with a direct threat to her life that she receives from an il- 3. “Juana should go to the authorities to accuse Pe- legal group. Participants listened to a recording of the sto- draza of having links with illegal groups.” ry and were asked to express their opinions on Juana’s con- 4. “The main result of Juana’s becoming a commu- text and situation and evaluate various choices of action for nity leader was the increase in the risk of her be- her (see Box 7). ing killed.” The purpose of the exercise is to help the participants identify with the situation, or at least relate the story to Source: Econometría, SEI–IFS. Informe Metodológico their individual context. When the participants express para la Evaluación de Laboratorios de Paz y Paz y Desar- their opinions on what Juana should do, they reveal their rollo (2006). processes of deliberation and unravel their views on what would be the fairest or best way to act in a situation such as the described. Threats by illegal armed groups (guerrillas communities where the RDPPs operate. Indeed, the pro- and paramilitary groups) are common in the regions and grams aim at modifying participants’ attitudes toward con- flict management by increasing their capacity to act collec- tively through social networks. BOX 6: Juana’s alternatives, from which participants are asked Juana’s story to choose, illustrate the main types of conflict management of (1) individual conflict management, (2) collective con- Juana was “a housewife, an ordinary farm worker, un- flict management, and (3) institutional conflict manage- til the day some violent men killed Jacinto, her hus- ment. It is also possible to elicit other features of partici- band. . .” pants’ aptitudes regarding sensitive issues, such as (4) the “the neighbor and her husband, who were vegetable risk associated with leadership in a context of violence (let- fiber growers, convinced Juana to start working with ters refer to the answer options in Box 7). them . . .” A preliminary analysis of the answers to “Juana’s Dilem- “her reputation grew until one day, Pedraza, a well- ma” based on intensity models shows that the RDPPs are known councilor and politician, suggested that she run encouraging community and institutional forms of con- as candidate for the Council.” flict management.72 In particular, it reveals an ideal conti- “however, before she took office, anonymous mes- nuity among the answers encouraged by the programs, be- sages began to arrive, addressed to Juana, with death cause initial community-based responses tend to lead in threats from the same armed group that had killed her the longer run to institutional conflict management op- husband . . . ” tions. This confirms the impact of the initiatives on the improvement of democratic governance. Moreover, partic- Source: Econometría, SEI–IFS. Informe Metodológico ipation in the RDPPs reduces the perception of risk asso- para la Evaluación de Laboratorios de Paz y Paz y Desar- ciated with leadership. rollo (2006). The experience of applying the social dilemma opens a wide-ranging scenario to measure aptitudes and capabilities 154 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K related to sociocultural dimensions of development pro- narios enables one to draw conclusions about participants’ cesses. Within this perspective, it is useful to consider some motivations, values, and attitudes. Two experimental exer- key elements of the debate about the scope and relevance of cises were carried out within the framework of the RDPP the instrument. The main issue concerns the existing rela- evaluation: a trust game and a public good game. tionship between the answers given regarding a hypothet- The public good game is carried out through a group ical scenario and the agents’ actual behavior. For instance, dynamic. Each participant receives an initial monetary en- participants tend to respond positively to choices involving dowment and has to decide whether to invest it in a pri- institutional conflict management, while conflicts actually vate or a public account. Every “peso” invested in the pub- may not be handled this way. It is possible that answers giv- lic account produces equal benefits for every member of en in a hypothetical scenario reflect views about socially ac- the group whereas every “peso” invested in the private ac- ceptable (correct) conflict management solutions and de- count only benefits the owner of the account. The dilemma pend partly on criteria of social desirability. This does not stems from the fact that there is a conflict between self-in- necessarily translate into real actions when people face the terest and optimum social outcome. If everyone invests in actual constraints, costs, and benefits of their decisions. In the public account, the total earnings are greater than they this respect, it is important to complement information on would be if everyone invested in his own account (Pareto the processes of moral deliberation with questions on actu- superior social outcome). However, the optimal strategy for al behavior. As a corollary to this general point, there is still the individual is always to invest in a private account (self- a need to understand the role played by certain features of regarding Nash equilibrium). the design of social dilemmas: the fact that the story is writ- Beneficiaries and nonbeneficiaries participate in the ten in the third person, the balance between generality and public good exercise in mixed groups of 40 people distrib- accuracy in the description of the context and the events, uted equally. The decision is taken in two rounds, first con- and the opportunity to use multiple choice answers. Les- fidentially and then after 10 minutes of free interaction be- sons can be drawn from the previous use of similar tools in tween the participants. This makes it possible to measure social psychology. the effect of communication. By sharing information and establishing agreements on the game strategy, participants Experimental exercises may be able to more effectively achieve the optimal social To measure attitudes and behaviors in the domain of the outcome or converge on the Nash equilibrium. “modes of relationship” toward the community, two exper- The decision dilemma raised in the public good ex- imental exercises were carried out as part of the evaluation ercise is particularly relevant to the RDPPs because they of the RDPPs. Experimental exercises constitute a possible precisely (re)create conditions that enable people to approximation to the topic of “social capital” from a mo- act collectively in focused regions by rebuilding bonds tivational (preference based) perspective and are based on of trust in the community. An analysis of the behavior the theoretical framework of experimental economics and observed indicates that the level of contribution to the game theory. public good of the municipalities where the RDPPs op- The instrument is based on the fact that people tend to erate is much higher than that recorded in other munic- react differently when it comes to answering direct ques- ipalities in the country with comparable characteristics. tions on issues such as trust or contribution to the pub- Moreover, the effects have a high degree of “propagation” lic good, compared with how they would act in actual sit- and spread within municipalities through spillover mech- uations. The tool therefore consists of simulating scenarios anisms, in line with the assumptions of the Programs’ in which participants face real dilemmas. Players’ decisions “change theory.”73 are turned into monetary payoffs according to the “rules Although experimental exercises traditionally have been of the game” of the exercise, and the exercises are carried used in a laboratory context, their use in the field has been out in a “controlled” fashion so that the decisions mainly spreading rapidly. The potential to evaluate the effect of involve consequences of an exclusively “economic” nature. social interventions on sociocultural domains is condi- An analysis of the decisions made in these simulated sce- tional to understanding the type of motivations identified 155 M&E SYSTEMS AND M&E TOOLS with the instrument. As for the social dilemma, this in turn Conclusion: The need to integrate methods points to the need to examine the relationship between the Social dilemmas and experimental exercises constitute two behavior adopted in the simulated scenario and the agents’ promising lines of work for applying a quantitative frame- actual decisions. work to the appraisal of the effects of complex programs In its more simplified version, economic theory assumes on the sociocultural dimensions of development process- that people act according to the calculation of the econom- es. However, o have a broader and deeper understanding of ic consequences of their actions, where the expected pay- the transformations under way, it is essential to ensure that offs of decisions only count insofar as they affect the de- they are complemented by qualitative methods. Using a se- cider (individualism). Experimental exercises emerge as an quenced combination of quantitative and qualitative ap- attempt to measures deviations from this theoretical ap- proaches, and ensuring effective feedback between the two, proach, testing the extent to which agents diverge from the will make it possible to develop and refine hypotheses on paradigm of homo economicus. To achieve this aim, the ex- the channels of impact of the interventions, adapt the de- ercises are designed in such a way that the consequences of sign of the instruments to the hypotheses of change, assess decisions are limited exclusively to the monetary spectrum the scope of instruments, contextualize the interpretation (through anonymity and the definition of the pay struc- of results, and triangulate information in depth. ture), and certain institutions (rules of the game) are estab- In the case of the evaluation of the RDPPs, developing lished, according to which there is a trade-off between indi- the change theory model has been the result of a participa- vidual and collective monetary payoffs. tory and essentially qualitative process. Moreover, the sec- If the observed results are different to what econom- ond phase of evaluation reveals the need to integrate the ic theory predicts, we can conclude that people value the principal strategy with qualitative methods to (1) validate monetary consequences of actions in a non individualis- hypotheses on the programs’ effect on relationship pat- tic manner (other regarding preferences). However, there are terns; (2) enhance understanding of the scope, potential, certain limitations to the interpretation of agents’ behav- and limitations of experimental exercises and the social di- ior in experimental exercises. First, the exercises purpose- lemma; (3) explore the negative and unexpected impacts; ly exclude noneconomic consequences of actions (such as and (4) analyze the effects of horizontal propagation of the the sociocultural payoff of cooperation), whereas the con- programs. The Colombian government, through the Na- sequences of actual decisions are not only economic and tional Planning Department, is committed to this task to may involve cultural or social dimensions (the effects of contribute to the search for, identification of, and dissemi- reputation, the esteem of peers, the importance of enjoy- nation of best practices in evaluation. ing these in the future, and so on). Second, unless their design improves, the exercises do not allow differentiating Luca Pellerano between extrinsic motivation (based on the consequences Departamento Nacional de Planeación-Colombia, Oxford of actions) and intrinsic motivation (based on obedience Policy Management and Institute for Fiscal Studies, England. to principles—one’s own criteria on what is fair or unfair, good or bad—regardless of the results of actions). Third, it is not clear whether the monetary payoffs used in the ex- Progress and Challenges of Monitoring ercises constitute a big enough incentive to mimic the eco- and Evaluation Systems in Latin America and nomic consequences of actual decisions. the Caribbean This discussion indicates the need to advance toward improving the design of the experimental exercises to cap- Introduction ture and differentiate between a broader spectrum of mo- This paper examines recent progress in the creation or tivations. At the same time, it complements experiments strengthening of monitoring and evaluation (M&E) sys- with other data collection tools that provide information tems in Latin America and the Caribbean. M&E systems on the occurrence of positive “modes of relationship,” such are a fundamental pillar of Management for Develop- as trust or cooperation in the actual context. ment Results (MfDR). This analysis uses two parameters: 156 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K (1) the work conducted by the Inter-American Develop- lic policies such as citizen security, agriculture, energy, em- ment Bank’s (IDB) Program to Implement the External Pil- ployment, and justice. All of these policy changes create lar of the Medium Term Action Plan for Development Ef- public value and induce social changes. fectiveness (PRODEV) and the PRODEV Evaluation Tool From the point of view of public value creation, market (PET), and (2) a survey by PRODEV conducted in late regulatory functions are in the same category as the provi- 2009.75 The PET provides a diagnosis of M&E systems sion of public goods. Thus, the establishment of competitive while the survey reveals the trend observed in late 2009. markets allowing for the provision of goods in a competi- tive scenario that makes those goods available at competi- MfDR and M&E systems tive prices also contributes to the creation of public value. Management for Results (MfR) is one of the instruments As a management strategy, MfDR requires decisions to and approaches that emerged from the new public man- be based on reliable information about the effects of gov- agement efforts to strengthen the government’s capacity to ernment actions on society. Its implementation will there- promote development. MR is defined here as a frame of fore depend on how the changes resulting from such ac- reference aimed to support public organizations to effec- tions are measured, that is, the existence of instruments to tively and comprehensively improve the creation of public capture these variations, systems to process the information value (results), ensuring maximum efficacy, efficiency, and and procedures to include data analysis in decision making. effectiveness in their performance, the achievement of gov- But these elements are not always found in government ad- ernment objectives, and the continuous improvement of ministration. their institutions (IDB and CLAD 2007). MfDR encourages a sense of responsibility of officers In the Organisation for Economic Co-operation and and institutions to achieve results and not for just perform- Development (OECD) countries, MR is used to guide gov- ing their duties. This promotes stronger and more substan- ernment actions toward maintaining the level of develop- tial verifiable evidence-based accountability. This is a major ment already attained. In contrast, in the countries of Latin change from traditional management and, consequently, American and the Caribbean, the emphasis is on accelerat- requires explicit political will from the highest government ing development, hence the term Management for Devel- level and a unit with enough power to promote change opment Results(MfDR). throughout the public sector. Although MfDR is primarily MfDR revolves around the creation of public value, a frame of reference for management, and its implementa- which includes observable and measurable social changes tion should reside with public authorities and officers, the the government introduces in response to social needs or gradual appropriation by mid-ranking officials and those demands that emerge through the democratic process. As a responsible for service provision is key. result, these changes “make sense” to the people. The ulti- Finally, results must come first. In other words, the pro- mate goal of MfDR is to endow its organizations with the cess states the results to be achieved and assigns resources capacity to manage public value creation and achieve the and processes accordingly. This is important and contrasts government program’s objectives. with the bureaucratic approach, which starts with available In MfDR, results relate to the social changes resulting inputs and known processes to identify potential results. from government actions and are not restricted to the activ- ities or outputs that contribute to such changes, which of- PET methodology ten taken as evaluating parameters for government actions. PET assesses the institutional capacity of countries in the For example, the results of the Ministry of Education will region to implement MfDR. The management cycle pil- not be measured by the number of schools built, teachers lars (see Figure 31) include the elements that must be part trained, or graduating students, but by the learning results of the cycle to align it with MfDR. These elements fall into actually achieved and, in the long run, the quality of the job three categories: (1) components (16), (2) indicators (37), offered based on the education received. and (3) minimum or variable requirements (141). Each el- Social changes do not only refer to changes in social ement is rated from zero to five. The average scores indicate policies like health and education, but include other pub- a country’s MfDR capacity. 157 M&E SYSTEMS AND M&E TOOLS FIGURE 31: ternational cooperation to improve public financial man- Management Cycle Pillars agement during the last decade. Initially, these efforts were part of the government reform measures of the 1990s that Strategic Planning aimed to overcome the economic crisis. These measures had a strong government bias and did little to improve managing capacity. Today, these efforts stem from the con- Monitoring and Budgeting viction that efficient management of public resources is es- Evaluation Management for Results sential to a country’s development. Cycle Pillars Strategic planning is also in a better position than the other pillars because most countries, after abandoning it in Financial Programs and the 1980s and 1990s, have once again embraced planning Management, Audit Projects as an instrument of development that targets strategic pri- and Procurement orities with greater emphasis on operational efficiency and Source: Author. participation. The programs and projects management pillar still This instrument analyzes the management cycle at na- shows poor performance, despite the fact that over the last tional central government level only, not at subnational two decades, countries in the region have implemented a level. Neither does it look at autonomous entities or pub- wide range of social programs to fight poverty and other lic companies. The sample included 25 countries from the social ills. These programs generally have been financed by Latin American and Caribbean region (see Table 21).� international organizations and were aimed at improving Results reveal great differences among the various coun- the quality of education and health services. Even though tries of the region, which were aggregated into three groups externally funded programs are backed by appropriate tech- according to their MfDR score. The first group includes nical, economic, and financial feasibility studies, it seems countries with a score of 3 or higher, the second includes that they have failed to change the management culture of countries with scores between 1.5 and 3, and the third is the line ministries that produce activities and outputs with- made up of countries with scores lower than 1.5. More than out much attention to results. half of the countries (16) are in the medium group. The av- The pillars with the lowest scores are budgeting for re- erage MfDR score for the region is 2 out of 5, indicating sults (BfR) and M&E. Both pillars are closely related, as that it is in the early stages of MfDR implementation. BfR is based on the information provided by M&E systems There are noticeable differences in the degree of prog- that follow up on government actions. Only those coun- ress between the different countries and also between Mf- tries in the advanced group show good capacity in these DR pillars. A comparison of the pillars shows that finan- pillars. cial management, auditing, and procurement (FMAP) is The high-scoring countries—Brazil, Chile, Colom- the most developed. This is a direct consequence of the ef- bia, and Mexico—have implemented innovative manage- forts deployed by governments in the region and the in- ment. They have a big lead over other countries in all pil- TABLE 21: Country Ranking by Degree of MfDR Development MfDR development Countries MfDR Score Advanced Brazil, Chile, Colombia, Mexico >3 Medium Argentina, Barbados, Bolivia, Costa Rica, Dominican Republic, Ecuador, El Salvador, >1.5 to <3 Guatemala Guyana, Honduras, Jamaica Nicaragua, Panama, Peru, Trinidad and Tobago, Uruguay Low Bahamas, Belize, Haiti, Paraguay, Suriname >1.5 Source: Author. 158 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K TABLE 22: MfDR Pillars by Country Group MfDR development Pillar Average Advanced Medium Low Strategic planning 2.4 3.5 2.3 1.9 Budgeting for results 1.5 3.1 1.3 0.5 Financial management, auditing and 2.6 4.0 2.5 1.3 procurement Programs and projects management 2.0 3.1 2.0 0.8 Monitoring and evaluation 1.7 3.8 1.3 0.8 MfDR Score 2.0 3.5 1.9 1.1 Source: Author. lars, particularly BfR and M&E, without which no MfDR whereas M&E and BfR are the least developed. Chile leads implementation can take place. In this group, pillars are al- the pack in the region thanks to its perseverance in pursu- so more balanced, thus confirming the integrality of pillars ing management innovation. as systems become more developed (Figure 32). As a whole, the region shows heterogeneous develop- Monitoring and evaluation systems ment of MfDR capacities. At one end a small group of Monitoring is the “continuing function that uses system- countries are implementing measures that gradually lead atic collection of data on specified indicators to provide them to systems for which results will be at the heart of management and the main stakeholders of an ongoing de- public sector management. At the other end are coun- velopment intervention with indicators of the extent and tries with basic systems, some even inserted in a tradition- achievement of objectives and progress in the use of allo- al, bureaucratic management framework. Most countries in cated funds” (OECD 2002). The goal is to ascertain prog- the region are between these two extremes, sharing what ress in the achievement of government objectives and goals may be called an incipient phase of MfDR development. as stated in its policies, programs, and projects. The mon- Their financial management systems show greater progress, itoring function is thus closely related to the functions of planning and budgeting. In MfDR, the monitoring system FIGURE 32: manages the public sector by means of indicators that verify MfDR Score by Pillar and Country Group the achievement of objectives and their quantitative goals. It analyzes both the achievement of objectives and the re- Strategic Planning sources used in achieving them. 5 A traditional follow-up system uses indicators to in- 4 form on inputs, activities, and outputs, without consider- 3 ing whether they contribute to fulfilling the objectives (re- Monitoring 2 Budgeting & for Results sults). In contrast, a results-based monitoring system uses Evaluation 1 indicators that measure the results obtained through pro- 0 grams and projects. Monitoring is an overarching function of the manage- ment cycle because each pillar has a role to play in its exe- Program and Financial cution. For example, the planning pillar seeks to establish Project Management, Management Audit and management objectives. BfR takes part in goal setting and Procurement public financial management provides information and da- Advanced Medium Low ta analyses. For its part, the monitoring system provides in- Source: Author. formation on the different levels of public policy execution, 159 M&E SYSTEMS AND M&E TOOLS such as services, programs, institutions, and policies. An completed the process or are still in the experimenting, de- overarching and integrated monitoring system imposes sig- signing, or implementing stages. This reflects the sparse in- nificant challenges to the articulation of the monitoring stitutional development of the monitoring function in the system, which may be overcome only through a continu- “medium” group of countries. Another significant short- ous improvement process. coming of such systems in these countries is the lack of re- Evaluation is the sults indicators. In the best of cases, there are output in- dicators, but this limits MfDR implementation because systematic and objective assessment of an on-go- it does not show whether these outputs have had the ex- ing or completed project, program or policy, its pected results. For example, building schools and deliver- design, implementation and results. The aim is to ing school books (outputs) do not necessarily lead to an determine the relevance and fulfillment of objec- improvement in children’s reading or writing skills (result) tives, development efficiency, effectiveness, impact measured through standard learning tests. and sustainability. An evaluation should provide The data collected reveal the wide gap in monitoring information that is credible and useful, enabling between industrial countries and the rest. In fact, the for- the incorporation of lessons learned into the de- mer have a much higher score than the latter and exhib- cision-making process of both recipients and do- it less of a difference between the institutional frame of the nors. (OECD 2002) monitoring function and the use of information, a clear in- dication of their advanced stage. Nevertheless, in all cases, The main difference between monitoring and evalua- the indicator for information use and dissemination had tion is that monitoring provides information on the state the lowest score, showing that the mere implementation of of achievement of the objectives and effects of a policy, pro- a monitoring system does not necessarily entail the use of gram, or project, while evaluation explains why these ob- information with management purposes. In this sense, it jectives or effects are being (or have been) fulfilled, or not. appears that several countries have stressed the preparation Evaluation uses systematic data collection and relevant in- of indicators and the construction of data applications but formation analysis processes to explain the causes and rea- have failed to create regulations and build institutional ca- sons for the results, examine unintended results, study the pacity to apply these tools in public management analysis process used to obtain them, and provide recommenda- and conduction. tions for future actions. With the exception of Chile, no other country has im- plemented institutional criteria and procedures to analyze Monitoring systems and manage government actions based on the information Of all the countries studied, only three do not have a dedi- provided by the monitoring system. This does not mean cated unit to follow up on government objectives and goals. that other countries ignore the information, but they use Monitoring units fall under the Presidency of the Repub- it in a noninstitutionalized manner, without formally inte- lic or the Ministry of Planning and, to a lesser degree, the grating it to the different stages of the management cycle. Ministry of Finance. The trend to place this function under For example, in those countries where the monitoring sys- the Presidency of the Republic is a reflection of the grow- tem falls under a unit of the Presidency of the Republic, se- ing interest of government officials to count on a set of ac- nior officials often base their decisions on the information countability and/or management instruments. The moni- provided. However, this use is discretionary and depends toring systems that report to the ministries of finance are exclusively on the will of the current administration. Sim- found in countries that have either consolidated (Chile) or ilarly, in countries that feature public investment systems, initiated (Peru and Mexico) their results-based budgeting the use of project monitoring information tends to be reg- systems. ulated, although these systems are restricted to investment Although most countries have an entity that is for- projects that are only a portion of the government plan. mally responsible for monitoring, the system is currently Finally, those countries that only recently began develop- operational in only 13. Over the past decade, most have ing a results-based budgeting framework still lack perfor- 160 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K mance indicator systems that are sufficiently robust to be norms on the evaluation of policy, program, and project used formally. results. Only Peru lacks specifically designed institutional- One important weakness present in most monitor- ized mechanisms because the regulations were established ing systems is the absence of vertical (government-orga- only recently and the system is still in the beginning stag- nization-operation unit) and horizontal (planning-bud- es. In the other four, evaluation mechanisms and method- geting-execution-M&E) articulations. Prior studies have ologies are formally operating together with an evaluation determined that most monitoring systems, especially those program with a budget of its own. This is a clear indication connected to the presidents’ offices, are used primarily by that systematic ex post evaluation of policies, programs, high-ranking authorities and only marginally by public of- and projects is a new phenomenon in the region’s public ficials to manage their organizations (Cunill and Ospina policy landscape. 2008). Monitoring systems are also weakly connected to Also worth mentioning is the fact that the four coun- the budget process and to program and project manage- tries have placed a high priority on the evaluation of so- ment. This suggests that many countries see their moni- cial programs and projects. Moreover, the evaluation sys- toring systems more as an accountability mechanism at the tems of Brazil and Mexico fall under the social sector and, top of the government hierarchy than as a management in- until now, only evaluate social actions targeting social de- strument that permeates the entire institutional tissue. velopment and poverty fighting. Even though the Colom- bian and Chilean systems reside in the National Planning Evaluation systems Department and the Ministry of Finance respectively, they Of all the countries studied, 13 have instituted laws that in also focus on social programs. one way or another make public management evaluation A number of differences also appear among these four mandatory. But only five (Brazil, Chile, Colombia, Mex- systems. In Chile, the articulation capacity of this system ico, and Peru) have a legal frame that specifically requires is both vertical (government, organization, personal) and policies, programs, and projects to be evaluated. The re- horizontal (planning, budget, execution, and M&E), but maining countries have evaluation regulations that do not the rest are still affected by bottlenecks. The Chilean sys- look at the results of government actions in the sense dis- tem is therefore capable of evaluating policies as well as cussed here. In most cases, these regulations refer to finan- programs and projects, and of systematically considering cial evaluation or analysis of achievement of the projects’ evaluation results in budget preparation and service man- physical goals. agement. Since 2003, Chile (2003), Colombia (2004), Bra- The M&E and the Budgeting for Results pillars are the zil (2006), and Mexico (2007) have introduced specific least developed in the Latin America and Caribbean region. TABLE 23: M&E Indicators by Country Groups MfDR development M&E indicators Average Advanced Medium Low Monitoring institutions 2.2 4.5 1.9 0.8 Extent of program and project monitoring 1.5 4.0 0.9 1.0 Use and dissemination of monitoring information 1.3 3.7 0.9 0.2 Statistical information systems 2.8 4.1 2.8 1.8 Legal and institutional frameworks for evaluation 1.7 3.9 1.4 0.8 Extent and articulation of the evaluation system 0.7 2.8 0.3 0.1 Actions derived from failure to achieve goals 0.9 3.1 0.4 0.3 Dissemination of evaluation results 1.4 4.6 0.8 0.4 Total 1.7 3.8 1.3 0.8 Source: Author. 161 M&E SYSTEMS AND M&E TOOLS Even though most countries have implemented systems to project management pillars. Without evaluation, it is not monitor the achievement of government objectives, these possible to guild planning and results-based budgeting or are still in the early stages of development. Their greatest to generate a continuous improvement culture in the pro- limitations to effectively contribute to MfDR are (1) the vision of public services. lack of formally established technical standards and work methodologies, (2) the lack of coordination between goal- Results of the PRODEV survey objective achievement information and resource invest- A recent survey by PRODEV describes a number of ini- ment information, and (3) the absence of result indicators. tiatives in the region with positive results in strengthening Interestingly, evaluation systems appear to be less devel- and consolidating M&E systems. The first concern was oped than their monitoring counterparts, for only a few to determine whether expenditure monitoring had in- countries have institutionalized ex post evaluation of pro- creased. Out of the sample, 71 percent of respondents be- grams and projects. With the exception of the four coun- lieve that expenditure monitoring had increased as a per- tries that feature advanced MfDR implementation, coun- centage of total expenditure and only 7 percent believed tries lack in evaluation capacity. This is a great challenge it had decreased. for MfDR capacity in the region because the absence of a The survey also asked whether the number of corrective system that evaluates government management results con- actions to address the failure to meet goals or achieve re- stitutes a major hurdle in the process of strengthening the sults has increased. This information is important because strategic planning, budgeting for results, and program and it provides information on how evaluations are used. In BOX 8: Information dissemination The Internet is an effective tool to disseminate the relevant information the government produces. This increases pub- lic management transparency and helps promote the involvement of civil society. Countries produce large amounts of information that they make available to the public through the portals of the various ministries. The most widely dis- seminated information has to do with the budget, financial statements, government procurement, audit reports, and the National Medium-Term Plan. There is less information on monitoring, management reports and evaluations part- ly because countries generate little information in this area. MfDR development Information available through the Internet Average Advanced Medium Low Information on the budget 3.3 5.0 3.4 1.3 Information on financial statements 3.3 4.5 3.2 2.3 Information on bids, proposals and awards 3.3 4.3 3.3 2.3 Audit reports 2.7 4.8 2.6 0.8 National Medium-term Plan 2.4 4.5 2.1 1.3 Monitoring system 1.5 4.0 1.1 0.0 Reports on sector management results 1.4 3.2 1.4 0.0 Reports on ex post evaluation 1.4 4.5 0.9 0.0 Reports on ex ante evaluation 1.0 2.0 0.9 0.0 Total 2.3 4.1 2.1 0.9 Source: Author. 162 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K this case, 55 percent of respondents believed that correc- As to whether access to evaluation reports by the legis- tive actions had increased, whereas 35 percent perceived no lature or civil society had improved, 73 percent of the sam- changes, and 10 percent thought they had decreased. ple believed that use by legislators and activists had grown, BOX 9: PRODEV survey results PRODEV Survey 2009 PRODEV Survey 2009 The percentage of expenditure monitored in relation to total The technical, administrative and financial actions introduced to correct expenditure the failure to meet goals and objectives detected during the evaluations Increased 70.9% Increased 55.0% Decreased 6.8% Decreased 10.0% Remained steady 22.4% Remained steady 35.1% The percentage of expenditure monitored in relation to total The technical, administrative and financial actions introduced expenditure to correct the failure to meet goals and objectives detected during the evaluations 80.0% 70.0% 60.0% 60.0% 50.0% 50.0% 40.0% 40.0% Series 1 30.0% 30.0% Series 1 20.0% 20.0% 10.0% 10.0% 0.0% 0.0% Increased Decreased Remained Increased Decreased Remained steady steady PRODEV Survey 2009 PRODEV Survey 2009 Do the Legislature and Civil Society have greater access to Human resources availability and capacity in evaluation reports? Monitoring and Evaluation Yes 73.5% Increased 43.8% No 26.5% Decreased 8.0% Remained steady 48.2% Do the Legislature and Civil Society have greater access to evaluation reports? Human resources availability and capacity in Monitoring and Evaluation 80.0% 70.0% 60.0% 60.0% 50.0% 50.0% 40.0% 40.0% 30.0% Series 1 Series 1 30.0% 20.0% 20.0% 10.0% 10.0% 0.0% 0.0% Increased Decreased Remained Yes No steady Survey: Author. 163 M&E SYSTEMS AND M&E TOOLS while 26.5 percent believed it had not. In addition, human ent. Training for human resources assigned to M&E tasks resources are not being assigned in sufficient numbers and and the financial resources of agencies responsible for these quality to perform M&E functions properly. Among the functions should be reinforced. respondents, 48.2 percent felt that the assignment of pub- lic sector human resources to M&E tasks had not changed Roberto García López and 8 percent felt that the number had declined. Prodev Coordinator, Inter-American Development Bank Finally, the survey showed improved coordination be- tween executing agencies and evaluation institutions: 66 percent of respondents believe articulation between these The Brazilian Monitoring and Evaluation institutions has increased. Network: A Report on the Creation and Development Process77 Conclusion The Latin America and Caribbean region is experiencing This paper discusses the background that led to the creation some good momentum for the institutionalization and of the Brazilian Monitoring and Evaluation Network and strengthening of M&E systems. The region must use this some of its recent advances and future challenges. The pub- momentum to good advantage and continue facing, with lic management evaluation process began in Brazil in the perseverance and continuity, the many challenges ahead. To 1990s. The Master Plan of the State Apparatus Reform, is- that end, the following are some conclusions that may be sued in September 1995, described management as a core drawn from this analysis. tool to consolidate economic stability and ensure sustain- First, M&E systems remain one of the weakest pillars able economic development. It reinforced the concepts and of MfDR, therefore, strengthening and consolidating them reforms needed for a more efficient government. Results- is crucial to the process of MfDR implementation. In this oriented management emerged in Brazil soon after the de- sense, there is a growing interest in building and developing velopment of the Master Plan and at the same time as efforts M&E systems in the region. Within these systems, moni- to control inflation and stabilize the economy (which im- toring is at a more advanced stage than evaluation and the proved the market’s capacity to plan its investments and fu- proportion of expenditure monitored has grown. ture projects). This movement resulted in important effects Second, in those countries that have implemented on the Brazilian federal government. It influenced the work M&E systems, the information generated is barely used for of the core and sector-oriented areas, and of internal and ex- decision-making purposes in the public sector, although ternal controls, in addition to the activities performed by the tendency is to increase its use. Efforts should be made the international organizations and civil society. The adop- to further involve political actors to expand the use of the tion of these new public management procedures was an- information generated by M&E systems. At the same time, chored in the 1988 Constitution’s provisions (Art. 165) it is important that nonexperts can access this information which set forth the rules to govern the federal budgeting in a friendly, accessible, and timely manner. process by introducing (1) the Multiyear Plan (PPA), (2) Third, dissemination of M&E results to congresses the Budgeting Guidelines Law (LDO), and (3) the Annu- and civil society will improve coordination among execut- al Budgeting Law (LOA). The state constitutions and mu- ing bodies and agencies and the institutions responsible for nicipal organic laws reproduced the Federal Constitution’s M&E. Dissemination of information to congresses and civ- provisions. As an input to new public management meth- il society must be continuous and permanent. Also impor- ods, the PPA represents a coming together of planning and tant is to disseminate information in a timely manner to budgeting by (1) implementing the medium- and long-term make it an input for analysis and decision making as well as government planning, (2) coordinating government actions, for the generation of proposals that may replace or supple- and (3) setting guidelines, objectives, and goals to the public ment current public policies and programs. administration to guide the allocation of public resources. Finally, improvements in the number and quality of The Ministry of Planning, Budgeting, and Manage- human resources assigned to M&E tasks are not appar- ment (MPOG) has been the main actor in this effort. For 164 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K the last 10 years, the MPOG has worked to incorporate by lottery, (2) annual financial auditing, (3) investigation of a results-oriented focus into sector-oriented public pol- cases, and the (4) federal government general balance. Fer- icies. The effort started with the 2000–03 PPA through raz and Finan (2008) have shown that Brazilian electors, three major innovations. The first was to make investments when well informed, reject those politicians who have been based on the land planning studies, resulting in a portfo- charged of corruption in favor of the honest candidates. The lio of 952 projects with a budget of Brazilian reais (R$) publication of the CGU auditing results in the municipal- 317 billion. Classified by macro sectors (Social Develop- ities where many cases of corruption have been found re- ment, Economic Infrastructure, Environment, and Infor- duced by 20 percent the chances for mayors to be reelected. mation and Knowledge), it considered the integration of Through the Federal Government Auditing Court activities, liability for results, and allocation of resources by (TCU), the legislative power began measuring the perfor- the private sector. The second innovation was to include in- mance of government policies with a focus on external con- dicators on implementation and outcome, promoting the trol, in addition to the traditional focus on compliance and effective emergence of a results-oriented management cul- the law. In 1999, the TCU began implementing operation- ture and concepts of monitoring and evaluation (M&E). al audits, incorporating case study techniques, surveys, and The third was to assign a manager to the program and in- focus groups to assess the programs’ implementation. The troduce the practice of results-oriented responsibility. With inclusion of performance audits among those carried out that, the MPOG promoted an entrepreneurial culture in by the TCU resulted from the technical assistance provid- the federal government, introducing programs and project ed by the U.K. Department of International Development management tools, and training the public managers in the (DfID) as well as from exchanges of experiences with the use of such tools. The publication of the Methodological U.K. National Audit Office (NAO) and the U.S. Govern- Guide to program indicators stands out as an example of ment Accountability Office (GAO) (Flint 2004). the program’s progress (SPI/MPOG 2010). In the last few years, some states and municipalities The sector-oriented ministries have begun efforts to- have also shown increasing interest in the M&E agenda. ward M&E policies in the fields of education, health, and Among these, the state of Minas Gerais, through its Plan- social development. Created in 2003, the Ministry of So- ning Bureau, State for Results Unite, and the João Pinhei- cial Development and Hunger (MDS) was the first minis- ro Foundation (JPF), has developed and promoted a set of try to assemble a specialized bureau in information man- innovative practices in public management, with a strong agement. The bureau has produced studies, promoted emphasis on the use of indicators and goals to provide fo- training, and implemented systematic methods to monitor cus and direction to public administration (SEPLAG/MG policies on cash transfers, social assistance, and food securi- 2008, 2009). ty (Vaitsman, Rodrigues, and Paes-Sousa 2006). The Infor- Among Brazil’s state institutions, the work developed mation Evaluation and Management Bureau is responsible by the SEADE Foundation,78 is also worth mentioning. for the bulk of studies, baseline research, beneficiary sur- In the last decade, the Foundation has specialized in creat- veys, and impact assessments produced to support the so- ing or improving methodologies and tools for the formu- cial policies (Wenceslau and Hobbs 2008). lation, monitoring, and evaluation of public policies and The federal government’s internal control, in turn, has programs, notably in the social area. The foundation’s activ- undergone an institutional strengthening process, which ities provided inputs to different public management stages, started in 2002 upon the establishment of the Office of by assisting in outlining programs and policies, and facili- the General Comptroller (CGU). The CGU resulted from tating eventual adjustments and path corrections. To sup- the merger of the Federal Internal Control Bureau (SFC) port the agencies that deliver or coordinate the government and the Internal Control Coordination Commission (CC- activities, these activities include assessment studies of spe- CI), with the status of ministry. The CGU’s work has fos- cific public programs; construction of monitoring method- tered public information transparency and quality, focusing ologies, defining indicators, to implement the policies’ or on prevention, inspection, and control of federal resources. public agencies’ monitoring; and, with similar objective, the The CGU’s tools include (1) inspection of municipalities structuring of broad systems on management information. 165 M&E SYSTEMS AND M&E TOOLS Brazilian development banks, notably the Banco do • establish the main incentives to enable the Nordeste do Brasil (BNB),� are also working in the field of ministries to carry out the M&E in their pro- M&E. Among some outstanding initiatives are the Credi- grams; amigo program evaluation, carried out in partnership with • develop M&E-oriented quality standards; and the World Bank, and the assessment of the Northeast Fi- • promote the use and dissemination of the in- nancing Constitutional Fund (FNE). The FNE program formation generated by the M&E.79. includes an assessment system that uses primary and sec- ondary data. It has assessed the FNE-Agroindustrial, the The Brazilian M&E network FNE-Comércio-Serviços e Industrial, the FNE-Rural sub- Brazil’s redemocratization process has increasingly en- programs, and global fund assessments. hanced citizens’ roles, notably, as participants in the pro- The Brazilian private sector has performed an impor- cess of designing, monitoring, evaluating, and control- tant role in strengthening this agenda. Initiatives, like the ling public policy implementation. Participation in the Competitive Brazil Movement (Movimento Brasil Compet- decision-making process and the control currently exer- itivo, MBC), established in November 2001, are mainly cised by citizens regarding the governors’ and the elect- focused on enabling projects aimed at increasing the or- ed representatives’ actions—in forums and councils es- ganizations’ competitiveness and the population’s living tablished specifically to that purpose—have increased the conditions, including the public sector management mod- public policies’ legitimacy. Moreover, the new ways that el. This initiative has always promoted the adoption of re- the private sector participates in the implementation of sults-based management practices and already has served policies, like the public-private partnerships and the pub- many Brazilian ministries, states, and municipalities. lic-private joint ventures, enhance the need for stronger By contracting studies, organizing seminars, and re- M&E systems. questing information, many civil society organizations and As the previous sections suggests, Brazil has broad movements, like the Itaú Social Foundation, Unibanco In- knowledge and practice in M&E of programs and projects. stitute, and the Futura Channel, have worked in the M&E Most of this experience is stored in research and education of programs. centers, or in the consultant’s personal collection. Hence, Finally, as an example of initiatives developed by interna- existing knowledge about programs and policies evaluation tional organizations, the Brazil Evaluation (Brasil Avaliação, is spread out and not systematized. Therefore, access to this BRAVA) program, developed by the World Bank as a techni- knowledge depends on the stakeholders’ research on the cal assistance project, not bound to loan operations, provided topic, and on the dissemination capacity of those produc- the federal government assistance in the development of M&E ing the given knowledge. This situation has given rise to the systems and activities. The BRAVA was launched in 2005 and idea among the experts in this area of promoting an initia- concluded in 2009. The main partner of this initiative was tive that could bring together the institutions that have ac- the Technical Monitoring and Evaluation Chamber (Câmara cumulated experiences in this field and provide forums to Técnica de Monitoramento e Avaliação, CTMA), a committee collaborate and share information. made up of the main federal government bodies interested in The organizations’ networks seek an environment to the matter. The BRAVA’s objectives were as follows: promote debate and strengthen collaboration among sim- ilar interests through the exchange of information. In ad- • Assist the federal government and its ministries dition, they seek to promote congresses, legal support, and in the development of M&E methodologies and development and provision of databases related to the par- policies. ticipating institutions’ interests. The organization of infor- • Disseminate the M&E culture. mation and records, and the socialization of the informa- • Prepare results-oriented management systems tion by its constituent elements can create a system that (GPR) to— enables innovation and interaction with the external envi- • clarify the institutional rules to the GPR coor- ronment. The dissemination of organized and structured dination and implementation; information to the external environment enhances visibil- 166 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K ity while promoting new elements to reconfigure and im- of the existence of a considerable critical mass for this top- prove the current system (Lopes 1996). ic, the lack of forums to exchange and organize experienc- Knowledge management is clearly an important factor es, and the mobilizing effect of the Fourth International for the network to pursue new linkages, expand the possi- Conference of the Regional Network, which took place in bility of working in a dynamic context, and ensure main- 2008 and was hosted in Belo Horizonte. It used the gath- tenance and survival as in the creation of internal control ering in Brazil of different actors in the topic to mobilize a mechanisms capable of promoting better use of existing re- side event during the conference, which led to the founda- sources. tion of the Brazilian chapter of the Latin America and Ca- ribbean Network. The Brazilian M&E network creation process In addition to the JPF, the IDB, and the World Bank, In this section, we explain the building process, objective, the inaugural meeting was attended by various representa- institutional arrangement, technological tool, and results tives of international and national institutions interested in achieved in the first year of the Brazilian Monitoring and the topic. These institutions included the United Nations Evaluation Network. Development Programme (UNDP), SEADE Foundation, the Banco do Nordeste to Brasil, Itaú Social Foundation, Objectives and strategies Unibanco Institute, MDS, TCU, and Ceará State Planning Some of the founding principles of the Brazilian chapter Bureau. of the Latin America and the Caribbean Monitoring and The discussions’ findings assisted in outlining the core Evaluation Network were decentralization, openness, and network objectives, such as: the low cost of failure. In its first year, the Brazilian chap- ter had more than one institutional champion. A number Promote the interchange of experiences and of institutions, including the State Statistics and Econom- knowledge through the organization of an annual ics Foundations, public banks, and international organiza- seminar and the social network platform, and dis- tions supported the initiative. Generally speaking, the rep- seminate M&E-related information through the resentatives of these institutions in the network, in addition publicizing of courses and events, with the mis- to serving as focal points, also represented individuals com- sion of strengthening the culture of monitoring mitted to these issues in their respective institutions. and evaluation in Brazil. A second important principle for the network is its openness. In an effort to maximize the possibilities of co- It targets the public and private actors working in the operation among different stakeholders, the network offers field of M&E of organizations, policies, programs, and membership to individuals or institutions and to other net- projects, including managers, researchers, consultants, pro- works, which translate into the membership’s freedom (ei- fessors, students, academics, as well as in the field of financ- ther personal, institutional, or of other networks) to max- ing, promoting, and assessing organizations. imize the possibilities of cooperation among different stakeholders. Finally, a third and important principle was Institutional arrangement the effort to reduce losses to a minimum if the initiative In June 2009, during the First Network National Seminar failed. The initiative’s existence is justified only if it is use- carried out in Belo Horizonte, a second meeting was held ful and capable of showing its value, which has happened and was attended by representatives of the World Bank; through the growing membership, increased number of vis- JFP; the Sao Paulo state government (SEADE Founda- itors, and growing engagement. tion), Ceará (SEPLAG), Rio Grande do Sul (Planning and The Brazilian Monitoring and Evaluation Network was Management Bureau, Revenue Bureau and Economics and launched as a chapter of the Latin America and the Carib- Statistics Foundation), Bahia (SEPLAG), Espírito Santo; bean Monitoring and Evaluation Network, under the lead- BNB; TCU; UNDP; Ministry of Agrarian Development ership of the JPF, the Inter-American Development Bank (Land Development Secretariat); MDS (Information Eval- (IDB), and the World Bank. Its launch was a recognition uation and Management Bureau); Ministry of Education 167 M&E SYSTEMS AND M&E TOOLS (Continuing Learning, Literacy and Diversity Bureau); Block 2: M&E methodologies and practices Itaú Social Foundation; Ecofuturo Institute and Unibanco This block focuses on (1) evaluation plans, (2) the logical Institute; Brazilian Evaluation Agency; Herkenhoff Prates– framework, (3) selection of outcome indicators and goals of Minas Gerais Technology and Development Institute; Ayr- public management, (4) implementation and process eval- ton Senna Institute; Federal University of Viçosa; Cecaps; uations, (5) impact evaluations, (6) ex anteevaluations, and and A&M Consultancy Services. Some decisions regard- (7) design of a management information system. ing the institutional arrangements were made during the meeting. These arrangements included (1) the creation of Web 2.0 platform a secretariat made up of representatives from the JPF, the Another important tool that contributes to the knowledge World Bank, BNB, SEADE Foundation, and the Brazil- management of this community of practice was the design ian Evaluation Agency; and (2) the definition of the mem- of a collaborative platform using Web 2.0 principles. The ber types, including institutional and individual members, selected platform resulted in the Brazilian M&E Web site as well as other networks. The participants reaffirmed the (http://redebrasileirademea.ning.com) created soon after the need to publish the contents produced by the network, and second meeting in June 2009 (see Figure 33). The main other relevant information on a Web site. functionalities of this platform are as follows: National seminar • Publication of news on a blog One of the network’s initial objectives was to organize an • Creation of discussion forums annual event at which the stakeholders could describe their • Creation of thematic groups projects, exchange experiences, and identify best practices • Publication of an events calendar in the field of M&E of public policies. The network’s first • Creation of individual profiles of the members meeting was organized around different kinds of assess- ment in the public policy cycle: diagnosis for public poli- A crucial aspect in initiatives of this nature is strategic cy design, the logical framework, ex anteevaluation, admin- planning for the launch, dissemination, and communica- istrative records, andex post evaluation. Another principle tion. To be better prepared, one should pay attention to the employed to organize the sections was the emphasis on how following basic aspects: these tools should be implemented and used, rather than on the results from these processes. This organizing princi- • Before launching the Web page, provide relevant ple should be maintained for the network’s second meeting, information to the target population (no one likes with the addition of some topics that have aroused inter- to become a member of a ghost city). est among public managers, like the selection of indicators • Launch the Web site with a small number of func- and goals to design and the management of results agree- tionalities (blog, videos, and events) and add oth- ments. The current structure to organize the network sem- er functionalities as the traffic increases (forums, inars has two blocks: groups). • Take advantage of on-site events to disseminate Block 1: design and institutional strengthening of M&E your network.80 systems for public policies and programs This block is mainly targeted at debating the current per- When it was first launched, the platform was relatively spectives of incorporating M&E practices into the public closed. Few members were invited to administer the page. policies process. These perspectives include (1) the creation All members had to be approved, and the Web site content of legal and managerial mechanisms, (2) the implementa- was available only to the members. The network gradual- tion of institutional units or public organizations devoted ly became flexible and open by removing many of these re- to monitoring and evaluating the policies, and (3) the ap- strictions, which, in practice, proved to be extremely bu- propriation and use of the M&E system’s results to feed- reaucratic. Moreover, an effort was made to promote more back the policies and programs. members to administrator. The network now has eight ad- 168 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K FIGURE 33: Brazilian M&E Web Site Source: http://redebrasileirademea.ning.com/. ministrators. The most recent change was the full and open Network members are mainly professionals from pub- access of the network’s content. Membership is required lic and private institutions that are interested in M&E ac- only to post messages and events, and to participate in the- tivities. Following are some of the main features of the net- matic groups and discussion forums. work’s membership:81 Outcomes • Women account for 54 percent of membership. A strategic aspect to consolidating the networks is to moni- • 14 percent of members are between 17 and 25 years tor results by constructing and updating the members’ pro- of age; 79 percent between 26 and 55 years of age; files, and monitoring and analyzing traffic to the Web page. and 7 percent are older than 56. Members FIGURE 34: As of March 2010, the Brazilian M&E Network has more Growth of Membership in the Brazilian M&E Network than 400 members from 18 states in Brazil and 14 coun- 450 tries. Figure 34 shows the development of the network 400 membership in the last eight months. The discontinuities 350 observed are clearly associated to specific events like (1) the 300 first network meeting in Belo Horizonte (June 2009), (2) 250 the second seminar on M&E in Rio Grande do Sul (No- 200 150 vember 2009), (3) the ANIPS national Meeting (Novem- 100 ber 2009), and (4) the CONSAD 2010 meeting (March 50 2010). The network members who were directly or indi- 0 rectly involved in the event’s organization disseminated 9 9 9 9 09 09 09 0 0 00 00 00 00 01 01 20 20 20 /2 /2 /2 /2 /2 /2 6/ 6/ 6/ 26 26 26 26 26 26 information about the network. The resulting increased /2 /2 /2 6/ 7/ 8/ 9/ 1/ 2/ 10 11 12 membership is remarkable. Source: Author. 169 M&E SYSTEMS AND M&E TOOLS • Our oldest member is 76 years old. retariat to stay informed about the site’s performance and to • 58 percent are civil servants, 18 percent are private improve the initiative’s mobilization activities and planning. employees, 16 percent are consultants, and 8 per- cent are students. User engagement • 39 percent work for state governments; 10 percent Another interesting metric has to do with the level of con- for municipal governments; 14 percent for the fed- tent publication in the platform. Until recently, network eral government; 19 percent for nongovernmental members have posted more than 100 blogs, about 60 events, organizations (NGOs); 10 percent for private insti- and 13 bibliographic references. Moreover, the number of tutions; and the remaining 8 percent for education comments among participants has also increased, suggest- and international institutions. ing deeper engagement of the members. • 48 percent of the members heard about the net- work from a colleague; 25 percent learned about Final remarks and recommendations it in some event where information about the net- The Brazilian Monitoring and Evaluation Network stands work was disseminated. for a partnership of different organizations with different natures. Among some of the success indicators, the capacity Monitoring visitors to the Web page of attracting and retaining members is outstanding. Howev- Another important aspect of the network is monitoring er, ensuring sustainability is one of the network’s main chal- and analyzing its use. Currently, many free tools facilitate lenges. This sustainability will result from the network’s ca- this purpose. These tools allow for the real-time follow-up pacity to make the contents relevant to its members, rotate on the network performance, collecting different use sta- leadership in the secretariat, and attract new facilitators and tistics, including the number of hits, countries and cities, moderators to the network, notably to the thematic groups. and Web site of origin. Moreover, tracking visitors also en- Another important challenge is the geographic distribution ables building dashboards (see Figure 35) that compare the of the members and content presented, making the network network traffic performance over time, identify Web sites space useful to members and institutions in different stag- of the same nature, and automatically send reports to the es of experience. Finally, another concern is to maintain and members. This kind of monitoring allows the network sec- expand the network’s openness and avoid creating a space FIGURE 35: Dashboard of the Brazilian M&E Network Source: Author. 170 S E L E C T P R E S E N TAT I O N S F RO M T H E F O U RT H M E E T I N G O F T H E L AT I N A M E R I C A N A N D C A R I B B E A N M O N I TO R I N G A N D E VA L U AT I O N N E T W O R K captured by a small group of interests. Some of the Web Juliana Wenceslau platform’s important challenges are as following: Graduate student in Public Management, Administration Post-Graduation Program (PPGA), University of Brasilia • The proper use of “virtual spaces” (UnB) • Many members have found it difficult to un- Márcia Paterno Joppert derstand the difference among blogs, forums, General Director, Brazilian Evaluation Agency thematic groups, and fields of comments. Sonia Nahas de Carvalho Thus, they ignore the conceptual differences Technical Advisor to the Deputy Board of Information among these, and post their contents in a dis- Analysis and Dissemination, SEADE Foundation organized way. • Lack of reference links • Another difficulty is the posting of content Session 5 Notes 64 (blogs and comments) with no reference links This presentation is adapted from Bamberger (2009) and on the topic. draws extensively on MacKay (2007). 65 For more information on the Africa Impact Evaluation Initia- • The visual presentation of blogs tive, see the Web site, http://worldbank.org/afr/impact. • Many members have difficulty with the visu- 66 See the Web site, http://www.worldbank.org/ieg/nonie/index.ht- al presentation of their blogs and ignore many ml. editing functions provided by the tool. The 67 See the Web site, http://www.3ieimpact.org. 68 page layout becomes confused.� This paper is based on the author‘s experience as consultant for the Office of Public Policy Evaluation of the National Planning Department, which runs the Colombian evaluation Finally, a network needs (1) a critical mass of individu- and monitoring system SINERGIA. als, (2) a concerted effort to attract and sustain a plurality 69 For more information on the other RDPPS in other parts of of institutions to protect the network from the political cy- the country, see http://www.redprodepaz.org.co and http://www. cle, and (3) a group of champions to facilitate and moder- accionsocial.gov.co/portal/default.aspx. 70 ate the content of the collaborative platform. The champi- For results from the first phase of the evaluation see DNP ons’ duties should include the following: (2009), available at http://www.dnp.gov.co/PortalWeb/Portals/0/ archivos/documentos/DEPP/Evaluacion_Politicas_Publicas/Se- rie_EPP11_Laboratorios_de_paz.pdf. • Attract new members 71 The initial design of the evaluation and the collection of base- • Get to know and analyze the members’ profiles (in- line information were carried out by Unión Temporal Econo- dividuals and institutions) metría-SEI-IFS, under the supervision of the NPD. Baseline • Input relevant contents to the Web site, meeting information was used to produce a descriptive analysis of the the members’ expectations Program beneficiaries and certain preliminary impacts were es- • Promote communication among members timated. The results have been recorded in DNP (2009, ibid.) 72 For an in-depth analysis, see DNP (2009, ibid.) • Prepare regular reports of the activities 73 For an in-depth analysis, see DNP (2009) and Attanasio, • Connect to similar activities Pellerano, and Phillips (2009). 74 The survey of a sample of 2,288 people took place in Septem- Afonso Henriques ber 2009. The response rate was 48.7 percent and covered 24 President, João Pinheiro Foundation countries of the region. 75 Jânia Pinho Argentina, Bahamas, Barbados, Belize, Bolivia, Brazil, Chile, Colombia, Costa Rica, Dominican Republic, Ecuador, El Sal- Manager to the Studies, Surveys and Assessment Environment vador, Guatemala, Guyana, Haiti, Honduras, Jamaica, Mex- of the Banco do Nordeste do Brasil ico, Nicaragua, Panama, Paraguay, Peru, Suriname, Trinidad João Pedro Azevedo and Tobago, and Uruguay. Economist, The World Bank 76 The authors thank the other network members for their com- John L. Newman ments and suggestions and participation in the V Internation- Lead Economist, The World Bank al Conference of the Latin American Monitoring and Evalu- ation Network, notably Eder Campos and Lycia Lima. The 171 M&E SYSTEMS AND M&E TOOLS authors also thank the initiative by Ricardo Santiago and the view the logical framework of the Multiyear Plan programs; support of Ricardo Paes de Barros, Eduardo Rios-Neto, André (3) institutional assessment of the PPA and MDS assessment Urani, Wanda Engles, and Pedro Olinto to the launching and system; (4) participation in the outlining of program assess- development of the first seminar of the Brazilian Monitoring ments (like Bolsa Família, Brasil Alfabetizado, Crediamigo, and Evaluation Network. Quilombolas, and Saúde Indígena);(5) training of the TCU 77 See Web site, http://www.seade.gov.br/. auditors in impact assessment through an IDF grant; and 78 See Web site, http://www.bnb.gov.br/. (6) support to the state of Minas Gerais in the outlining of the 79 In its five years of existence, the BRAVA delivered assistance, first household survey (Pequisa por Amostra de Domicílios, carried out studies, and promoted events with the federal gov- PAD/MG) methodology of impact assessments, and indica- ernment and state and municipal governments. These have tors benchmarking. 80 included (1) workshops on assessment systems from Austra- See the Web site, http://help.ning.com/cgi-bin/ning.cfg/php/en- lia, Canada, and Chile; (2) outlining the methodology to re- duser/workshop.php?p_sid=LDXLDYWj. 81 The Network‘s Web page provides a detailed profile of the net- work members. 172 �References and Additional Bibliography Anastasia, Antonio Augusto J. 2006. “Introduction.” In itor Capacity Building.” Available at http://www.aus- O choque de gestão em Minas Gerais: políticas da gestão aid.gov.au/. pública para o desenvolvimento, Renata Vilhena, Hum- Balridge National Quality Program. 2009. Education Crite- berto Falcão Martins, Caio Marini, and Tadeu Barreto ria for Performance Excellence. Gaithersburg, MD: Na- Guimarães (organizers), 13–17. Belo Horizonte: Edi- tional Institute of Standards and Technology (NIST), tora UFMG. U.S. Department of Commerce. Available at http:// Arnold, Mary. 2006. “Developing Evaluation Capacity in www.baldrige.nist.gov/Education_Criteria.htm. Extension of 4-H Field Faculty.” American Journal of Bamberger, Michael.2009.“Institutionalizing Impact Eval- Evaluation 27 (2): 257–69. uation Within the Framework of a Monitoring and ADB (Asian Development Bank). 2004. “Evaluation of Evaluation System.” Independent Evaluation Group, Capacity Development Assistance to Lao PDR.” Avail- World Bank, Washington, DC. Available at http:// able at http://www.adb.org/. intresources.worldbank.org/INTINFRES/Resourc- ————. 2003. Evaluation of Capacity Development for es/2780429–1173477721799/2009_institutionaliz- Managing Water Utilities.” Available at http://www. ing_ie_framework_me.pdf. adb.org/. ————.2006. “Evaluation Capacity Building.” In Cre- ————. 2002. “Evaluation of Country Assistance Pro- ating and Developing Evaluation Organizations: Lessons gram Evaluation: Mongolia.” Available at http://www. learned from Africa, Americas, Asia, Australasia and Eu- adb.org/. rope, ed. M. Segone. Geneva: United Nations Children ————. 1999. “Evaluation of Technical Assistance for Emergency Fund (UNICEF). Public Expenditure Management to Bhutan, India, Bamberger, M., K. Mackay, and E. Ooi. 2005. “Influential Kiribati and Lao PDR.” Available at http://www.adb. Evaluations: Detailed Case Studies. Operations Evalua- org/. tion Department. Washington, DC: World Bank ————. 1997. “Evaluation of Effectiveness of Training ————. 2004. Influential Evaluations: Evaluations that in Education Projects in Indonesia.” Available at http:// Improved Performance and Impacts of Development Pro- www.adb.org/. grams. Operations Evaluation Department. Washing- Attanasio, O. Pellerano, L. and Phillips. 2009. “Peace and ton, DC: World Bank Goodwill. Using an Experimental Game to Analyse Barkenbus, Jack. 1998.Expertise and the Policy Cycle. Uni- Paz y Desarrollo.” Working Paper W09/20, Institute versity of Tennessee. Available at http://www.gdrc.org/ for Fiscal Studies, London. decision/policy-cycle.pdf. AusAID (Australian Government Overseas Aid Program). BDMG (Banco de Desenvolvimento de Minas Gerais). 2009. “A Staged Approach to Assess, Plan and Mon- 2002. Minas Gerais do Século XXI—transformando o 173 M&E SYSTEMS AND M&E TOOLS poder público: a busca da eficiência. Vol. 9. Belo Hor- Corporación Transparenciapor Colombia. 2009. Índice de izonte: Rona Editora. Transparencia Nacional Resultados 2007 y 2008.Colec- Beck, Ulrich.1998. Risk-fraught Society. Towards a new mo- ciónDocumentos, Observatorio de Integridad No. 10, dernity. Barcelona: Editorial Paidós. Bogotá, July. ————. 2001. “Return to the Theory of the Risk- ————. 2008a. Índice de Transparencia Municipal 2005– fraught Society.” Bulletin of the EGM[[Vol.?]] no. 30 2006.ColecciónDocumentos, Observatorio de Integri- (March). dad No. 9, Bogotá, May. Boyle, Richard, and Lemaire, Donald, eds. 1999. Build- ————. 2008b Índice de Transparencia Departamental ing Effective Evaluation Capacity, Lessons from Practice. 2005–2006. ColecciónDocumentos, Observatorio de New Brunswick, NJ. Transactions Publishers. Integridad No. 8, Bogotá, May. Budget Office (Chile). 2009a. “Public Finance Report.” ————. 2006. Índice de Transparencia Nacional – Enti- Government Draft Budget Law for 2010, 119–61, dades Públicas Nacionales Resultados 2004 y Resultados October. 2005. ColecciónDocumentos, Observatorio de Integr- Budget Office (Chile). 2009b. “Evaluation of Public Sector idad No. 7, Bogotá, December. Financial Management in 2008 and Updated Projec- ————. 2006. Índice de Transparencia Municipal Resul- tions for 2009,” 25–38 and 65–75, June. tados 2004 – Ejerciciopiloto. Colección Documentos, Bureau, Sylvie. 2004. Analyse des besoins de formation. Observatorio de Integridad No. 6, Bogotá, December. Développement International Desjardins, Québec. ————. 2005a. Índice de Transparencia Departamental- Available at http://www.lamicrofinance.org/files/17682_ Resultados 2004–2005.ColecciónDocumentos, Obser- file_Document.pdf. vatorio de Integridad No. 5, Bogotá, December. CIDA (Canadian International Development Agen- ————. 2005b. “Transparency as a Job Category.” In- cy). 2000. “Capacity Development: Why, What and ternal Document. Bogotá. How.” Available at http://www.acdi-cida.gc.ca/home. Cunill Grau, Nuria. 2009. “El Mercado en el Estado.” Revis- Capacity.org. 2005. “Exploring the Soft Side of Capacity De- ta Nueva Sociedad 221 (May–June): 34–53. Available at velopment.” Available at http://www.capacity.org/en. http://www.nuso.org/upload/articulos/3608_1.pdf. ————. 2004a. “Meso-level Capacity Development.” Danida (Danish International Development Assistance). Available at http://www.capacity.org/en. 2005. “A Results-Oriented Approach to Capacity ————. 2004b. “Renewing Approaches to Institutional Change.” Available at http://www.dacp.ait.ac.th/dani- Development.” Available at http://www.capacity.org/en. da.html. ————. 2003. “Capacity Development: The Why’s and ————. 2002. “Capacity Development Evaluation Step How’s.” Available at http://www.capacity.org/en. 1: Contributions to an Analytical Framework.” Avail- ————. 2002. “Capacity for Development: Insights and able at http://www.dacp.ait.ac.th/danida.html. Innovation.” Available at http://www.capacity.org/en. DAC/UNEG (Development Assistance Committee/Unit- ————. 2001. “Tools of the Trade: Capacity Assess- ed Nations Evaluation Group). 2007. “Framework for ment.” Available at http://www.capacity.org/en. Professional Peer Reviews. Joint Task Force on Profes- ————. 2000. “Informing the Capacity Debate: Oper- sional Peer Reviews of Evaluation Functions in Multi- ational Experiences.” Available at http://www.capacity. lateral Organizations.” Available at http://www.uneval. org/en. org/documentdownload?doc_id=103&file_id=210. Chelimsky, Eleanor. 2008. “A Clash of Cultures: Improv- (DfID) U.K. Department for International Development. ing the ‘Fit’ between Evaluative Independence and 2005. “Capacity Development and State Building: Is- the Political Requirements of a Democratic Society.” sues, Evidence and Implications for DfID.” Available American Journal of Evaluation 29 (4): 400–15. at http://www.dfid.gov.uk/. Corporación Chile Transparente. 2006. “About the Span- ECDPM (European Centre for Development Policy Man- ish Concept of Corruption.” Internal Document, agement). 2009. “Capacity Change and Performance: Transparency International, Santiago. Capacity Development: Between Planned Interven- 174 R eferences and A dditional B ibliography tions and Emergent Processes Implications for De- DR, Inter-American Development Bank, Washington, velopment Cooperation.” Policy Management Brief DC. No. 22, ECDPM, Maastricht. Available at http://www. ————. 2009. La sociedad civil y la GpRD, en búsque- ecdpm.org/Web_ECDPM/Web/Content/Download.nsf/ da de una alianza estratégica. Inter-American Develop- 0/5E619EA3431DE022C12575990029E824/$FILE/ ment Bank, Washington, DC. PMB22_e_CDapproaches-capacitystudy.pdf. Guimarães, Tadeu Barreto, and Bernardo Tavares de Al- ————. 2008. “Capacity Change and Performance: meida. 2006. Da estratégia aos resultados concretos: a ex- Insights and implications for Development Cooper- periência de Minas Gerais (2003–2005), n. 12. Cader- ation” Policy Management Brief No. 21, ECDPM, nos BDMG: Belo Horizonte: BDMG. Maastricht.Available at www.ecdpm.org/capacitystudy. Guimarães, Tadeu Barreto, and Eder Sá Alves Campos. ————. 2006a. “Study on Capacity, Change, and Per- 2009. Programa Estado para Resultados: balanço da im- formance,” Final Workshop Report, ECDPM, Maas- plementação de um escritório de apoio à gestão estratégi- tricht. May. Available at http://www.ecdpm.org/Web_ ca e gestão por resultados em Minas Gerais. II CONSAD ECDPM/Web/Content/Download.nsf/0/B03C8B92B Congress, Brasília. AE9E5A1C12571C0003C144E/$FILE/Final%20 Guzmán, Marcela. 2007. “Program Evaluation: Technical Workshop%20Capacity%20Study_May%202006%20 Notes.” Public Management Series, ECLAC, United -%20Report.pdf. Nations, New York. ————. 2006b. “The Concept of Capacity.” Draft Re- ————. 2005. “Systems of Management and Control port, ECDPM, Maastricht. Available at http://www. and Results-based Budgeting, The Chilean Experi- ecdpm.org/Web_ECDPM/Web/Content/Download. ence.” Available at www.dipres.cl/fr_control.html. nsf/0/EF596EFCBB8A417DC1257178004AEB5B/$ ————. 2003. Systemsof Management Control and Re- FILE/Morgan%20-%20Capacity%20-%20What%20 sults-Based Budgeting: The Chilean Experience. Santia- is%20it%2010052006.pdf. go: Ministry of Finance, Government of Chile. � Ferraz, Claudio, and Frederico Finan. 2008. “Exposing IEG (Independent Evaluation Group). 2007. Using Train- Corrupt Politicians: The Effects of Brazil’s Publicly Re- ing to Build Capacity for Development: An Evaluation of leased Audits on Electoral Outcomes.” The Quarter- the World Bank’s Project-based and WBI Training. Wash- ly Journal of Economics (MIT Press) 123 (2): 703–45. ington, DC: World Bank. Flint, M. 2004. Country Study: Brazil 1997–2003. Evalu- ILPES-CEPAL (Instituto Latinoamericano y del Caribe de ation of DFID Country Programmes, London: United Planificación Económica y Social–Comisión Económi- Kingdom Department for International Development. ca de América Latina). 2003. Government Management Garay, Luis Jorge, et al. 2007. «Armed Conflict and Or- by Results. New York City ganized Crime in Corrupt Scenarios. About the Case IDB-CLAD (Inter-American Development Bank–Lat- of Colombia.” Collection of Notebooks on Transpar- in American Center for Development). 2006. Modelo ency No. 13, Corporación Transparencia por Colom- Abierto de Gestión para Resultados en el Sector Público. bia, Bogotá. Inter American Development Bank, Washington, DC. ————. 2002. “On the Role and Nature of Corruption Kaplan, R. 1999. “The Balanced Scorecard for Public Or- in Colombia.” Paper presented at Forum on Fighting ganizations.” Harvard Business Review, Article Reprint Corruption, Governance and Institutional Performance, No. B9911C, Cambridge, MA. March 21, Bogotá. Kusek, J., and R. Rist 2004. Ten Steps to Results-Based García Lopez, Roberto. 2008. La Gestión para Resultados y Monitoring and Evaluation Systems. Washington, DC: el Presupuesto para Resultados en América Latina y el Ca- World Bank. ribe. Paper presented at the Conferencia internacional Larbi, George A. 1999. The New Public Management Ap- de presupuesto por resultados, June 8–9, México proach and Crisis States. Geneva: United Nations Re- ————. 2009. El rol del Congreso en la Gestión para Re- search Institute for Social Development (UNRISD). sultados del Desarrollo. Parlamentarian Forum on Mf- 175 M&E SYSTEMS AND M&E TOOLS Lopes, Sônia Aguiar. 1996. A teia invisível: informação e Políticas da Gestão Pública para o Desenvolvimento, Re- contra-informação nas redes de ONGs e movimentos soci- nata Vilhena, Humberto Falcão Martins, Caio Marini, ais. Unpublished doctoral dissertation. Federal Univer- and Tadeu Barreto Guimarães (organizers), 67–76. Be- sity of Rio de Janeiro, Rio de Janeiro. lo Horizonte: Editora UFMG. Mackay, K. 2007. “How to Build M&E Systems to Sup- Osborne, David, and Ted Gaebler. 1992. Reinventing Gov- port Better Government.” Independent Evaluation ernment. Reading, MA: Addison-Wesley Publishing. Group, World Bank, Washington, DC. Pitman, George K., and Osvaldo Feinstein, and Gregory Marcel, Mario, and Marcela Guzmán. 2008. Hacia un cír- Ingram, eds. 2005. Evaluating Development Effective- culo virtuoso en la reforma presupuestaria en Chile. ness. Washington, DC: World Bank. Presupuesto y Gasto Público 51: 305–23. Available at Porto, Claudio, and José Paulo Silveira. 2006. Implantação http://www.ief.es/default_Ing.htm. e Gestão da Estratégia do Governo, n. 12. Cadernos BD- Marcel, Mario, Marcela Guzmán, and Marcelo Tokman. MG: Belo Horizonte: BDMG. 2002. “Building a Consensus for Fiscal Reform: The Rose-Ackerman, Susan. 2001. Corruption and Govern- Chilean Case.” OECD Journal on Budgeting 2 (3): ments. Causes, Consequences and Reform. Madrid: Siglo 35–55. Veintiuno Editores. Metzenbaum, Shelley H. 2009. “Performance Manage- SEPLAG/MG (Minas Gerais State Planning and Manage- ment Recommendations for the New Administra- ment Secretariat). 2009. Caderno de Indicadores. Pro- tion.” IBM Center for the Business of Government,. grama Estado para Resultados. Belo Horizonte, Minas Minas Gerais Planning and Management Secretariat. 2007. Gerais. Plano Mineiro de Desenvolvimento Integrado 2007– ————. 2008. NOTA TÉCNICA DCMG/SEPLAG 2023: Estratégia de Desenvolviment Estado para Resul- Nº 37 /2008. Belo Horizonte: Minas Geras. tados Belo Horizonte. Available at http://www.almg.gov. Sánchez Albarena, Fernando. 2003. Planificación estratégica br/eventos/pmdi/pmdi.pdf. y gestión pública por objetivos. Santiago: ILPES. ODI (Overseas Development Institute). 2006. “Capacity Segone, Marcos, and Ada Ocampo. 2006. Creating and De- Development for Policy Advocacy: Current Thinking veloping Evaluation Organizations. Lima: DESCO. and Approaches among Agencies Supporting Civil So- Shack Yalta, Nelson. 2003. “Progress in Implementing Per- ciety Organisations.” Available at http://www.odi.org. formance Indicators in Public Entities in Peru.” In- uk/resources/download/136.pdf. Government Management by Results. New York: ILPES- OECD/DAC (Organisation for Economic Co-operation CEPAL. and Development/Development Assistance Commit- Shirky, Clay. 2008. Here Comes Everybody: The Power of tee). 2008. Third High Level Forum on Aid Effective- Organizing without Organizations. New York: Penguin ness. Accra Agenda for Action. Available at http://www. Press. accrahlf.net. SPI/MPOG (Ministry of Planning, Budgeting and Man- ————. 2006. “The Challenge Of Capacity Develop- agement). 2010. Indicadores de programs: Guia Met- ment, Working towards Good Practice.” Available at odológico. Brasilia, MPOG: 128. http://www.oecd.org. Torres Fragoso, Jaime. 2008. “La influencia de las culturas ————. 2002. Glossary of Key Terms in Evaluation and nacionales en el desarrollo de la Nueva Gerencia Públi- Results-Based Management of the Organisation for Eco- ca en América Latina.” In RevistaReforma y Democra- nomic Co-operation and Development. Paris: OECD. cia, no. 42. Caracas: CLAD. ————. 1992. “DAC Principles for Effective Aid.” UNDP (United Nations Development Program). 2007. Reprint available at http://www.oecd.org/datao- “Capacity Development Practice Note.” Available at ecd/31/12/2755284.pdf. http://www.undp.org/cpr/iasc/content/docs/UNDP_Ca- Oliveira, Ana Clara de, Bernardo de Almeida, and Tadeu pacity_Development.pdf. Barreto Guimarães. 2006. A Integração Planejamento- ————. 2007. Capacity Assessment Methodology. User’s Orçamento. In O Choque de Gestão em Minas Gerais: Guide. New York: UNDP. http://europeandcis.undp.org/ 176 R eferences and A dditional B ibliography uploads/public/File/Capacity_Development_Regional_ Villarreal, Julio. 2007. “Case Studies: Colombia and the Training/UNDP_Capacity_Assessment_Users_Guide_ Monitoring and Evaluation System (NMRS).” CLAD, MAY_2007.pdf. World Bank, Washington, DC. UNEG (United Nations Evaluation Group). 2007. “Eth- WBI (World Bank Institute). 2007. “Capacity Develop- ical Guidelines for Evaluation.” Available at: http:// ment in the World Bank Group: A Review of Non- www.uneval.org/search/index.jsp?q=ethical+guidelines. lending Approaches.” Available at http://wbi.world- ————. 2008. “Code of Conduct for Evaluation in the bank.org/wbi/. UN System.” Available at http://www.uneval.org/search/ ————. 2006. “A Revolution in Capacity Develop- index.jsp?q=code+of+conduct. ment? Africans Ask Tough Questions.” Available at ————. 2005a. “Assessment of 20 Capacity Develop- http://wbi.worldbank.org/wbi/. ment Tool.” Available at http://www.uneval.org/search/ Wenceslau, J., and J. Hobbs. 2008. “Institutional Assess- index.jsp?q=Assessment+of+20+Capacity+Development ment of Monitoring and Evaluation Systems (M&E) +Tools. in the Ministry of Social Development and Combat- ————. 2005b. “Measuring Capacities: An Illustrative ing Hunger (MDS).” World Bank, Washington, DC. Catalogue to Benchmarks and Indicators.” Available at World Bank. 2006. “Evaluation Capacity Develop- http://www.uneval.org/search/index.jsp?q=Measuring+C ment.” Available at http://web.worldbank.org/WB- apacities%3A+An+Illustrative+Catalogue+to+Benchma SITE/EXTERNAL/TOPICS/EXTCDRC/0,,menuPK:6 rks+and+Indicators. 4169181~pagePK:64169192~piPK:64169180~theSite UNICEF (United Nations Children Emergency Fund). PK:489952,00.html 2002. “Capacity Building for UNICEF Humanitari- ————. 2005a. Impact Evaluation Study under the Pro- an Response in Review.” Available at http://www.uni- grams Evaluation Plan in Chile. Washington, DC: cef.org/. World Bank. UNIDO (United Nations Industrial Development Or- ————. 2005b. “Institutionalizing M&E systems in ganization). 1990. “Report on the Evaluation of Latin American and Caribbean Countries.” PREM UNDP-Financed and UNIDO-Executed Projects Notes 102. Available at http://www1.worldbank.org/ of Technical Assistance in Metrology.” New York: prem/PREMNotes/premnote102.pdf/. UNIDO. ————. 2005c. “Capacity Building in Africa, an OED United Nations World Food Programme. 2008a. “Evalua- Evaluation of World Bank Support.” Available at http:// tion of WFP’s Capacity Development Policy and Op- web.worldbank.org/WBSITE/EXTERNAL/TOPICS/EX erations.” Available at http://www.wfp.org/. TCDRC/0,,menuPK:64169181~pagePK:64169192~pi ————. 2008b. “Evaluation Policy.” Available at http:// PK:64169180~theSitePK:489952,00.html. www.wfp.org/. ————. 2004. “Evaluation Capacity Development: Vaitsman, Jeni R., Roberto W. S. Rodrigues, and Rômu- OED Self-Evaluation.” Operations Evaluation De- lo Paes-Sousa. 2006. “The System for Evaluating and partment, World Bank, Washington, DC. Monitoring Social Development Programs and Poli- ————. 2002. Institutional Performance and Governance cies: the case of the Ministry of Social Development in Colombia. Bogotá: World Bank. and the Fight against Hunger in Brazil.” Policy Papers World Bank and OECD (Organisation for Economic Co- 17, UNESCO,. operation and Development). 2005. Recently-Identi- Vilhena, Renata. 2006. Rumo à segunda geração do choque fied Good Management Practices for Development Re- de gestão. In: Marini, Caio et al. O choque de gestão sults. Washington, DC: World Bank. em Minas Gerais: políticas da gestão pública para o desenvolvimento, Renata Vilhena, Humberto Fal- Internet Resources cão Martins, Caio Marini, and Tadeu Barreto Gui- Monitoring and Evaluation News. http://www.mande.co.uk. marães (organizers), 351–56.Belo Horizonte: Edito- Mouvement Québécois de la Qualité. 2009. Le Qualimètre. ra UFMG. http://www.qualite.qc.ca/quali/qualimetre3.htm. 177 M&E SYSTEMS AND M&E TOOLS National Institute of Standards and Technology. 2009. Bal- World Bank Independent Evaluation Group. http://www. ridge National Quality Program. http://www.quality. worldbank.org/ieg and http://www.worldbank.org/ieg/ nist.gov/. ecd. OECD/DAC (Organisation for Economic Co-operation World Bank Independent Evaluation Group. “Impact Eval- and Development/Development Assistance Commit- uation.” http://www.worldbank.org/ieg/ie and http:// tee). 2009. MfDR Capacity Scan. http://www.mfdr.org/ www.worldbank.org/impactevaluation. CAP-Scan.html. 178