45270 Quality of Public Expenditure NOTE No. 2 ­ JUNE 2008 The Role of Evaluation in Mexico: Achievements, Challenges, and Opportunities Traditionally, evaluation of public expenditure has dicators for performance evaluation. The PROMAP focused on the size and composition of public expen- discourse introduced the words "accountability" and diture, rather than on its quality. However, evaluation "evaluation" to Mexican public administration. A can enhance the quality of public expenditure by subprogram, Evaluation and Measurement of Pub- identifying approaches that yield best results and can lic Management, sought measurable outcomes for be replicated or scaled up. In the context of perfor- internal evaluation and performance indicators to mance-informed budgeting, planning agencies, line guide the decision-making process, and the Ministry ministries, and groups outside government, such as of Control and Administrative Modernization was the congress and civil society, can use information created and given responsibility for developing per- on program outcomes generated by evaluations. This formance indicators, performance agreements, and note summarizes Mexico's experience in institution- defining budgets in relation to achievement of results. alizing and using evaluation as a tool for learning However, PROMAP achieved little and was not able and accountability and in strengthening the role of to transform the familiar decision-making process evaluation in the framework of performance-informed within ministry offices. budgeting. Box 1. An Influential Evaluation: Oportunidades Background The Oportunidades Human Development Program is a Mexican Government program aimed at improving the education, health, Mexico was a world pioneer in evaluation in the and dietary intake of Mexico's poorest people. In 1997 it started 1970s, using support from the World Bank to focus to operate in rural areas under the name PROGRESA. By 2001 it on evaluating integrated rural development projects.1 had been extended to operate in semi-urban areas, and it reached These first-generation evaluations were antecedents urban areas in 2002. Five million families currently benefit from this of the evaluation of the Progresa/Oportunidades program. The continuation of this program in successive administra- program that began in 1997--one of the most tions was largely due to the results emerging from its evaluation, influential evaluations in the development world2 which was considered credible. (see Box 1). From the outset, an evaluation component was included to quantify PROMAP. In 1996 President Ernesto Zedillo in- the program's impact through rigorous methodologies, using both stituted a Program for Modernization of the Public qualitative and quantitative approaches. The impact evaluation Administration, PROMAP, to promote accountability of Oportunidades was assigned to internationally and nationally and improve the service culture, in part through in- prestigious academic and research institutions. The evaluation of Oportunidades has been a continuous process focused on (a) measur- ing the outcomes and impacts on a short-, medium-, and long-term 1 Some of this work is described in OED, World Bank scale; (b) identifying the outcomes and impacts attributable to the (1983): Mexico ­ INTEGRATED RURAL DEVELOP- program and discerning effects associated with other factors, on MENT PROJECT (PIDER I), Loan 1110-ME (available in individuals, families, and communities; (c) analyzing indirect effects http://www-wds.worldbank.org/external/default/WD- associated with the operation of the program; and (d) identifying ways SContentServer/WDSP/IB/1999/09/10 /000178830_ to improve the program. The work done for this evaluation has yielded 98101901051757/Rendered/INDEX/multi_page.txt an important collection of information--databases, questionnaires, 2See Behrman, Jere R. and Skoufias, Emmanuel (2006): technical and methodological notes, household studies, and review "Mitigating myths about policy effectiveness: Evaluation documents--that is publicly available. Thanks to the systematic of Mexico's antipoverty and human resource investment evaluation of the program, it has become a model for the design of program," in Annals, AAPSS, 606, which includes several other conditional cash transfer programs. useful references to the vast literature generated by this evaluation. Source: SEDESOL, http://evaluacion.oportunidades.gob.mx:8010/en/index.php MEXICO: QualIty Of PublIC EXPEndIturE Transparency and Good Government Agen- evaluation results. The Guidelines, along with the new da. During the presidency of Vicente Fox, the issues performance evaluation system (SED), represent an of administrative reform and responsive government important achievement in promoting a new culture of were placed squarely on the public agenda with results-based management and evaluation. the National Program to Combat Corruption and Promote Transparency and Administrative Develop- In March 2008 for the first time Congress received ment and the Good Government Agenda (ABG). from the Executive 116 external evaluations--Consis- These reforms sought to transform administrative tency and Results Evaluations--, coordinated jointly structures and procedures, the technology used by by CONEVAL, the Ministry of Finance and the Ministry government, and the behavior of public employees, of Public Management. These evaluations analyse to deliver more and better public services at a lower the design, strategic planning results, coverage and cost. Although this reform initiative fell short to fulfill targeting as well as the satisfaction of beneficiaries its promise, it generated important lessons: (a) the of social programs, so general recommendations can need to prioritize and sequence reform activities; be drawn from these results. At the same time, at the and (b) an understanding that the particularities of beginning of 2008, almost 145 federal programs the diverse bodies of government matter; they cannot have elaborated their Logical Frameworks (Matriz be treated as homogenous entities with a "one-size- de Marco Lógico) in order to improve planning and fits-all" approach.3 performance indicators. Evaluations are also being carried out in other Achievements ministries. For instance, there is already evidence of the fruitful use of evaluation and performance information in the case of the Higher Secondary In 2000, to achieve greater transparency and ac- Education Sub-Secretary, which has been conduct- countability and to prevent political manipulation of ing impact evaluations for several basic education public programs, the Mexican Congress passed a law and higher education interventions, designing a new requiring an annual evaluation of all the programs evaluation system and generating better data collec- conducted by the Federal Executive Branch. In 2001 tion on scholarship recipients and survey instruments the Under Secretary for Planning and Social Evalua- for the selection of recipients. The sector, along with tion--SEDESOL--was created, and in 2004 Congress the National Institute for the Evaluation of Education approved a General Social Development Law (LGDS) (INEE)4 has also been active in collaborating with that institutionalized the evaluation process and cre- ated a National Evaluation Council (CONEVAL). In 2006 Congress approved the Ley Federal de Presu- puesto y Responsabilidad Hacendaria (LFPRH), which Box 2. A New Evaluation Institution: The National creates the Performance Evaluation System (Sistema Evaluation Council (CONEVAL) de Evaluación del Desempeño-SED) and details the use of performance indicators. Mistrust has prevailed in several public institutions in Mexico for many years. For this reason CONEVAL was created with a governance These institutional changes promoted evaluation structure that confer to it and to its evaluations a high degree of and performance indicators, but these initiatives and trust. Thus, CONEVAL is led jointly by its Executive Secretary and a arrangements were not integrated into a single sys- panel of six academics who must be members of the National System tem; rather, they were considered as a set of parallel of Researchers and who are elected by the National Commission systems. For this reason in March 2007 the General of Social Development, that has national, state, municipality and Guidelines for the Evaluation of Federal Programs Congress representatives.5 The Council has two important tasks at were published between CONEVAL, the Ministry of hand: the measurement of multidimensional poverty at the national, Finance (Hacienda) and the Ministry of Public Man- state and municipal levels; and second, the evaluation of social agement (Función Pública) with the main objective of policies and programs. aligning the incentives of the previous evaluation and monitoring regulations. The Guidelines describe the basic components for 3 See Mauricio I. Dussauge Laguna, "Paradojas de la the implementation of the monitoring and evaluation reforma administrativa en México." Buen Gobierno, 2 system, including the importance of the linkage of (2007): 28­42. strategic national social policy objectives to program indicators, the types of external evaluations applicable 4 See http://www.inee.edu.mx/ to federal programs (including impact evaluation, 5 See http://www.coneval.gob.mx/ to have access to the design and consistency framework evaluation, recent program evaluations and to poverty figures by process evaluation, etc.) plus the basic instruments State, Municipality and the recent evolution of National for the improvement of program performance using poverty. 2 NOTE No. 2 ­ JUNE 2008 the teachers' union for the dissemination of recent Challenges evaluations and tendencies in the education sector. In this way, evaluations are being developed and used in the context of budget preparation for policy The Mexican evaluation framework presents two development and potentially in the future for budget important gaps. negotiation, as well as for budget execution. Institutionalization Gap. Although Mexico has An important contributor in terms of accountabil- a legal framework for evaluation and CONEVAL is ity is also Congress. Mexico's Supreme Audit Institu- an appropriate institution for the evaluation of social tion (ASF), with its division focused on performance development programs and policies, public sector audits,6 applies the standards of the International evaluation is not fully institutionalized at the central Organization of Supreme Audit Institutions (INTO- and subnational government levels or in the sectors SAI). ASF was created in 2001 and it has been a (except in social development and education). Thus relevant player in promoting a culture of evaluation evaluation at those levels is institutionally homeless--so and results. that economies of scale and of scope in evaluation activities cannot be developed, and the country cannot Strategic Plan. The Strategic Plan 2001­2006 com- benefit from the enhanced accountability and learning mitted the Government to "present a budget reform that would result from broader evaluation. Spain's based on the modernization of the budget process experience of institutionalizing evaluation (see Box 3) to ensure more efficient and transparent public ex- provides an interesting reference model. penditures." It highlighted the important link between management reform and public sector productivity: The Agency's Governing Board, which is chaired "Public sector productivity...requires strengthening the by the Agency's president, includes representatives of flexibility and autonomy needed by public officials for the Ministry for Public Administration, the Ministry of decision-making. This in turn strengthens accountability Finance, the Ministry of the Presidency, the Ministry mechanisms and performance evaluation." The Na- for Foreign Affairs and Cooperation, and the trade tional Development Plan (NDP) for 2007­2012 builds unions, as well as independent professionals of widely on the objectives of the Strategic Plan to focus on acknowledged reputation. Representatives of autono- improving the "management, processes, and results" mous communities (regions) that sign cooperation of the Federal Public Administration. agreements with the Agency may also sit on the Gov- erning Board. In fact, one of the roles of the Agency is Legal Framework. The new legal framework for to promote the development of an evaluation culture evaluation includes the following: at the level of the autonomous communities. · the January 2004 Ley General de Desarrollo Capacity Gap. Although the Government has im- Social (LGDS) which creates CONEVAL proved its program evaluation capacity, most of this expertise is located in CONEVAL/SEDESOL and deals · the March 2006 Ley Federal de Presupuesto y Responsabilidad Hacendaria, which details the re- quirements and use of performance indicators; Box 3. The Spanish Evaluation Agency · the 2007 Fiscal Reform, which details the perfor- The Spanish National Agency for the Evaluation of Public Policies and mance evaluation (SED) framework and requires states and municipalities to adopt results-based Quality of Services was created on January 1, 2007. Its goals are to management; use evaluation to improve the quality of public services, enhance the public accountability of government bodies, and promote a better · the 2007 General Guidelines for the Evaluation use of public funds. Its creation was based on the recommendations of Federal Programs; and of an expert panel of academics, distinguished professionals, and public managers that prepared a detailed analysis of evaluation in · the 2008 Agreement for the implementation of Spain with references to international experiences. the SED, which requires the issuance of an an- nual evaluation program that includes the bud- The Agency is a public-law body with its own legal personality and getary programs to be evaluated and the types property, endowed with management autonomy and flexibility. It of evaluation to be carried out, and establishes is attached to the Ministry for Public Administration. The Agency that the information generated by the SED will evaluates the programs and policies selected each year by the be disclosed to the public. Spanish cabinet, and submits to the Parliament an annual report on central Government agencies' efforts to improve the quality of the services they provide to the public. A management contract governs the Agency's activities and its relations with the Government, which 6 See http://www.asf.gob.mx/ funds those activities. MEXICO: QualIty Of PublIC EXPEndIturE only with the evaluation of social programs and policies, official request for an action plan that addresses which represent only a minority fraction of all public the issues raised by the evaluation. The goal is programs. There is only limited capacity to conduct to link evaluations with decision making and or contract evaluations and monitoring of non-social to strengthen incentives for evaluators to offer programs. Meanwhile, the private market for provid- realistic, meaningful recommendations. During ers of external evaluations remains extremely limited. 2008, CONEVAL, the Ministry of Finance and Thus, though there are some highly competent evalu- the Ministry of Public Management will produce ation experts in CONEVAL and SEDESOL, as well as guidelines for the use of evaluation findings in some universities and research centers, the central and subnational levels of government lack capacity to manage, conduct, and use evaluations. Three symptoms Opportunities of this gap are the proliferation of indicators that are not used, problems in the quality of some evaluations, and the almost total lack of evaluations at the state The Mexican Government's interest in improving the level. A three-pronged approach would help: quality of public expenditure through evaluation is shown by the legal framework that has been developed · The supply side of the evaluations mar- in recent years, and by the commitment of officials in ket should be strengthened to reduce key government institutions to design and implement transaction costs and increase the quali- a performance evaluation system combined with re- ty of evaluations. Evaluations of federal social sults-based budgeting. In the context of performance- programs have been of variable quality and utility. informed budgeting, performance information from High-quality providers of evaluation services are evaluations could be used for stronger program and scarce, and demand, greatly increased by the policy management in line ministries. It could also Congressional mandate, has exceeded supply. be used for more robust negotiation between the It is essential to establish clear quality standards, planning agencies and line ministers during budget and mediation and arbitration mechanisms to preparation. In this process, the use of performance resolve controversies that may arise. information can inform "spending reviews"--a special form of policy evaluation that is used in a number of · Enhancing M&E systems is not just a OECD countries to support the allocation function of supply-side issue or a "technical fix": the the budget. Unlike line ministries' policy evaluations, demand side is important, as well. Even these reviews go beyond effectiveness and efficiency though supply-side issues such as the timeliness under current funding levels and focus on the con- and quality of evaluations are important, M&E sequences alternative funding levels would have for findings must have an important audience if the outputs and outcomes. systems are to be sustainable. M&E information should be used by all stakeholders, including sec- The Mexican Congress is also showing increas- tor ministries and, often, civil society. To increase ing interest in evaluation, confirming another key the impact of evaluations, their key messages role of performance information as an instrument for should be disseminated in user-friendly ways. deepening the debate at all stages of the budget cycle among groups outside of government. International · A mechanism should be established organizations such as the World Bank, the OECD, to ensure that evaluation findings are and the Inter-American Development Bank are provid- used in subsequent decision making or ing valuable support as Mexico develops a suitable action plans. In Mexico evaluators have not evaluation system, taking international best practices engaged often enough in discussions with project into account and adapting them to the Mexican situ- managers (partly because of a misunderstanding ation. There are significant opportunities to enhance about what is required to maintain the indepen- and consolidate the Mexican evaluation system and dence of evaluators), and managers have rarely transform it into a key instrument for accountability, used the evaluation findings to improve program learning, and good governance. performance. Recently, however, CONEVAL and SEDESOL's external evaluations have involved Osvaldo Feinstein, Senior Adviser greater collaboration between program manag- ers and program evaluators to produce objective Spanish Evaluation Agency reports that can be used for decision making. Gonzalo Hernández Licona, Executive Secretary Another approach would be to submit evaluation National Council for the Evaluation of Social findings to program managers, along with an Development Policies (CONEVAL) Mexico: Quality of Public Expenditure is a series of notes produced in collaboration between the Mexican Government's Secretaría de Hacienda y Crédito Público and the Public Sector and Governance unit in the World Bank's Latin American and Caribbean Region.