69638 The World Bank Uruguay: Strengthening Participatory Monitoring and Evaluation of Social Policy Report of Phase I June 22, 2007 Document of the World Bank Currency Unit = Uruguayan Peso (A$) 1 US Dollar = 24.3 uruguayan pesos 1 Peso = US$ 0.04 (As of June, 04, 2007) Abbreviation and Acronyms CPI Corruption Perception Index DINEM Dirección Nacional de Evaluación y Monitoreo IDES Instituto de Investigación y Desarrollo INE Instituto Nacional de Estadísticas MIDES Ministerio de Desarrollo Social M&E Monitoring and Evaluation PANES Plan de Atención Nacional a la Emergencia Social PSM Propensity Score Matching RD Discontinuity Regression Regional Vice President Pamela Cox Country Director Axel van Trotsenburg Sector Director Laura Tuck Sector Manager McDonald P. Benjamin Task Team Leader William Reuben ii Table of Contents Abbreviation and Acronyms ............................................................................................... ii Acknowledgements ............................................................................................................. v Executive Summary ........................................................................................................... vi 1. Origin and Overview of the Technical Assistance Activity ....................................... 1 2. The Country, Institutional and Program Context for the Uruguay Non-Lending Technical Assistance (UY NLTA)......................................................................................... 2 A. Country Context ...................................................................................................... 2 B. Institutional Context................................................................................................ 3 C. Program Context ..................................................................................................... 4 3. Objectives, Expected Outcomes and Activities of the UY NLTA ............................. 7 4. UY NLTA Contributions under Phase I to Strengthen Impact Analysis Capacity in MIDES ................................................................................................................................ 9 A. Data Collection for Impact Evaluation ................................................................... 9 B. Quantitative Impact Analysis ................................................................................ 10 C. Qualitative Impact Analysis .................................................................................. 11 5. UY NLTA Contributions under Phase I to Participatory Monitoring and Evaluation by MIDES ......................................................................................................................... 14 A. Design of software and technological infrastructure for the PM&E system ........ 15 B. Indicators for the Participatory M&E System ...................................................... 16 C. Piloting of Participatory M&E Experiences by the Social Councils .................... 16 D. Sharing of International Best Practices with Participatory M&E ......................... 17 6. Next Steps – Phase II of the UY NLTA ................................................................... 18 A. Technical Note Based on Preliminary Qualitative and Quantitative Impact Analyses ........................................................................................................................ 18 B. Implementation of the Participatory M&E System .............................................. 19 7. Preliminary Lessons Learned and Suggestions for the Implementation of Phase II 21 8. A Road Map for Implementing Phase II of the UY NLTA ...................................... 23 References ......................................................................................................................... 26 References ......................................................................................................................... 26 iii Annexes Annex I Description of Plan de Atención Nacional a la Emergencia Social (PANES) Annex II Events and Bank Missions Annex III Diseño Conceptual de un Sistema de Monitoreo y Evaluación Institucional y Participativa del Bienestar Social del Uruguay – Primer informe de avance. MIDES Annex IV Relatoría de Fase Cero - Plan de Atención Nacional a la Emergencia Social (PANES): ‘Programas Trabajo por Uruguay’ y ‘Rutas de Salida’ – Bases para el diseño de una evaluación cualitativa, MIDES Annex V Pro-Fundación for the Social Studies, Faculty of Social Science of the Universidad de la República (2006) “Proposal: Impact Evaluation of the National Assistance Plan of Social Emergency (PANES). MIDES. Annex VI Concept Note: Uruguay Strengthening Participatory Monitoring and Evaluation of Social Policy. Annex VII Table of indicators: Areas, Sub-areas, and Dimensions for Social Well- Being Monitoring and Evaluation System Annex VIII Propuesta de unificación de bases de datos con variables sociales existentes en las diferentes dependencias del Estado - Fortaleciendo la capacidad del MIDES para el análisis, el monitoreo y la evaluación de la política social del Uruguay Annex IX Outline for Technical Note Annex X Relatoría de los Talleres de Difusión de Resultados Preliminares de la Evaluación de los Componentes del PANES Annex XI Relatoría del Trabajo de Campo de la Evaluación de los Componentes del PANES T Tables Table 1 Programs of the National Assistance Plan of Social Emergency (PANES) 05 Table 2 Applicant and beneficiary household of PANES per geographical region 05 Table 3 Parameters for Evaluation in Qualitative Analysis 13 Table 4 Techniques and Sample for the Qualitative Analysis 13 Table 5 Summary of Preliminary Results of Qualitative Analysis of PANES Program 14 Table 6 Risks Faced in Phase II Implementation and Proposed Measures 23 Table 7 Timetable of implementation for the second phase 24 Box Box 1 Flowchart of Uruguay Non-Lending Technical Assistance (UY NLTA) 08 Box 2 Selected Tools for Participatory Monitoring and Evaluation 18 iv Acknowledgements This report was prepared by task team leader William Reuben (Senior Social Scientist –LCSSO), and team members Maria Beatriz Orlando (Senior Economist – LCSPP), Jose Maria Ghio (Senior Specialist in Civil Society–LCSSO), Rafael Rofman (Lead Social Protection Specialist – LCSHS), Carmen Monico (Consultant–SDV) and Eri Watanabe (Consultant–LCSSO). Blanche Cotlear (Consultant) also contributed to the preparation of this document. Irina Ghobrial provided administrative support. The team would like to thank the Peer Reviewers Dorte Verner (Senior Economist–LCSSO), Benedicte Leroy De La Briere (Social Protection Specialist– LCSHS) and Keith MacKay (Senior Evaluation Officer–IEGKE) for reviewing this report. Many thanks to DFID for its financial support to the technical assistance activity. Special thanks to the Minister of Social Development in Uruguay, Mrs Marina Arismendi for her advice and for providing the Bank team with the opportunity to work with her team on this joint initiative, including Lauro Meléndez (Dir. Evaluación y Monitoreo), Mariella Mazzotti (Dir. Desarrollo Ciudadano), Milton Silveira, Julio Llanes and Marianela Bertoni. Local consultants and consulting firms Ignacio Pardo, Hugo de los Campo, Maria Cecilia Llambe, Nicholas Brunet Adami, Lorena Custodio and Pro- Fundación for the Social Studies, of the Faculty of Social Science of the Universidad de la República also contributed conducting analytical and field work, and preparing reports for the Uruguay Non-Lending Technical Assistance (UY NLTA). The team hopes that the UY NLTA may be helpful in the improvement of social assistance delivery to the poor in Uruguay and to the ongoing debate on the PANES results and the design of Plan de Equidad. v Executive Summary 0.1 The Uruguay Non-lending Technical Assistance (UY NLTA) for Uruguay was undertaken at the request of the Government of Uruguay’s Ministry of Social Development (MIDES). 1 Following a devastating financial and macroeconomic crisis in 2002, MIDES was established with a view to coordinating social programs and providing a safety net program for Uruguayans affected by the crisis: the National Assistance Plan of Social Emergency (PANES), which was established in 2005. 0.2 The new Administration that assumed office sought to improve the targeting and outreach of its safety net program. It identified the relevance to improve the Ministry’s capacity to evaluate the social and poverty impacts of PANES and to establish a participatory monitoring and evaluation system. MIDES invited the Bank to support this initiative via non-lending technical assistance. This document reports on how the Bank has been working with MIDES staff in building the capacity to undertake an impact evaluation of PANES, and designing a participatory monitoring system, which are the key activities of Phase I of the UY NLTA. 0.3 The UY NLTA activities supported by the NLTA have been led by MIDES, with the Bank playing an advisory role. The assistance program’s scope, overall methodology and expected outcomes were defined in a dialogue with the Ministry of Social Development during a Bank mission in May 2006. Subsequent missions and contacts have ensured a strong coordination between MIDES’ National Office for Monitoring and Evaluation (DINEM) and the Bank Team, under the leadership of the Minister of Social Development. 0.4 The objectives of the UY NLTA are to: (i) evaluate the impact of PANES social emergency programs; (ii) inform a dialogue about the future role of social emergency programs and of MIDES as the lead authority responsible for the coordination and oversight of social policy in Uruguay, and (iii) strengthen MIDES capacity to assess, monitor and evaluate social policy in Uruguay, with considerable weight on participatory approaches to monitoring and evaluation. 0.5 During its implementation, the UY NLTA was organized in two phases to respond to the client’s timely and needs. Phase I has accomplished the following: (1) the basic evaluation capacity of MIDES has been built and the participatory monitoring and evaluation (PM&E) system in MIDES was designed; and (2) an intermediate analytical report on two PANES programs has been 1 MIDES was created in March 2005 (Law 17.866) with the main objective of formulating, coordinating, and overseen social development strategies, policies and programs in Uruguay. In order to make available a safety net to households affected by the 2002 crisis, MIDES has focused its efforts since its creation in coordinating with other agencies of the sector the implementation of PANES. vi produced. Phase II of the UY NLTA will generate the following deliverables: (1) a Technical Note that will contribute to inform the policy dialogue on the future of social assistance in Uruguay under the new Plan de Equidad; (2) M&E Reports produced by the system; (3) a Report on Participatory Pilots and articulation of social accountability mechanisms into the overall M&E system, and (4) learning events. 0.6 The UY NLTA has supported MIDES in the design of quantitative methods to assess the social impacts of PANES. The design involves the development of comparative and counterfactual analyses (Regression Discontinuity, Propensity Score Matching, analysis of registry data) as well as complementary statistical approaches using data from a follow up survey of PANES. The UY NLTA supported the design of the survey (identification of indicators, questionnaire and sampling framework). MIDES has run a first panel of the survey that will deliver critical information on the performance and impacts of emergency program. 0.7 The UY NLTA also supported the design and implementation of a qualitative analysis to assess the results PANES exit activity Rutas de Salida and program Trabajo por Uruguay. The analysis was designed in consultation with multiple stakeholders and has produced preliminary results that show important contributions of PANES programs in addressing the social emergency, but also some failures to link training programs to labor-market needs and income generation initiatives. 0.8 To improve MIDES capacity for coordination of social policy, the UY NLTA is supporting the design and implementation of a PM&E system. Phase I helped develop the conceptual framework, the definition of the users and suppliers of the system, and the creation of the technology infrastructure to support the system. A study of international, regional and national systems of socio-economic indicators is underway. Additionally, pilot experiences in Rivera and Tacuarembó are being carried out with the involvement of the Social Councils, which are local civil society mechanisms that will monitor the performance of specific social programs, making use of social accountability mechanisms. The Social Councils are expected to have an active participation in the generation of information to feed the system with the perception of the beneficiaries of the social programs. 0.9 Key lessons can be drawn from the implementation of Phase I of the UY NLTA that will be useful for Phase II, and the design and implementation of similar NLTA programs in other countries: a. It is advisable to ensure early client involvement from the design stage of the NLTA program to promote full ownership. The client was involved in the definition of all of the elements of the UY NLTA, including the adoption of the approaches and methodologies supported by the UY NLTA. This was done through a Bank mission early in the activity, joint planning of approaches and vii activities, DINEM participation in identification of consultant candidates, and the delivery of a training program and adaptable TA that builds institutional capacity for PM&E and impact evaluation. b. To further enhance country ownership, the voice of the protagonists of the development actions and of various government actors will continue to be integrated through the development of participatory approaches. The Team has been working closely with MIDES for the provision of guidance to IDES, who is responsible for the coordination of the pilot program with the Social Councils. c. Technical assistance needs to stay sufficiently flexible to respond to the client needs; this is true, particularly in changing institutional environments. When necessary, it could adopt an approach in stages to better respond to the institutional timetable of the client. The UY NLTA has provided an adequate framework for making adaptations throughout the implementation of phase I activities given the numerous uncertainties about the future of MIDES and the future Plan de Equidad. Similar provisions have been made for phase II. d. Vital to the development of any NLTA operation is that of sharing of international best practices, which must be expanded and include the broad range of stakeholders. Training workshops on international good practice in social accountability and PM&E were conducted by Bank staff with MIDES and IDES staff as to expand their capacity to establish participatory mechanisms that are adequate for Uruguay. e. Inter-governmental coordination is critical for an effective design, monitoring and evaluation of social policy. Pursuing with MIDES a greater involvement on the part of other ministries in the UY NLTA process becomes a priority activity in the early implementation of phase II to address this risk. 0.10 Some of the risks involved in the implementation of the UY NLTA (Phase II) for which measures are needed to be taken include: a. Uncertainty about the future role of MIDES and potential lack of interest and even resistance on the part of other government agencies involved in social policy to develop the PM&E system. The Team will facilitate opportunities for dialogue and coordination to build consensus among the various stakeholders about the main features of the PM&E system, including how to develop a roadmap for the establishment of a unified database supporting the proposed system. b. Resistance on the part of the Social Councils in the adoption of innovative PM&E mechanisms. The Team will coordinate with various ministries and intergovernmental offices involved in social assistance at the local level, and with national and local civil society organizations to expose them and the Social Councils to international experience in monitoring activities. It is important that viii the Bank team present evidence on the effectiveness of social accountability mechanisms to improving performance of service delivery. c. Expansion of PANES coverage may create problems in measuring impact among comparison groups, particularly when members of the comparison groups turn into program beneficiaries. The Team will advise the government to assuage any modifications of the baseline due to changes in the eligibility criteria. d. The commitment shown by the client for phase I may not be the same for phase II. The Team will seek government’s explicit approval of phase II activities and engage multiple stakeholders in ensuring institutional continuity of the PM&E system. 0.11 The Roadmap for Phase II of the UY NLTA includes the following activities: a. Technical Note on the performance and impact of PANES and a discussion on the future of social assistance programs in the framework of the new Plan de Equidad. The Technical Note will use information and analytical results yielded by the First Panel of the Survey and the Qualitative Analysis. It will be presented to the Social Cabinet. b. Implementation of Social Councils’ Pilot Program. c. Development and implementation of social information and M&E systems. d. Analytical reports based on the implementation of second and third survey and qualitative panels assessing the social impact of the emergency programs. e. Knowledge management activities: dissemination of the experience through GDLN activities in coordination with other Bank learning initiatives. f. Production of UY NLTA phase II report and Decision Meeting of phase II. g. Workshop to deliver results to MIDES. h. Seminar in Uruguay of the final report of the UY NLTA. i. UY NLTA closing date: June 2008 ix 1. Origin and Overview of the Technical Assistance Activity 1.1 The Uruguay Non-lending Technical Assistance (UY NLTA) for Uruguay was undertaken at the request of the Government of Uruguay’s Ministry of Social Development ((MIDES). 2 Following a devastating financial and macroeconomic crisis in 2002, MIDES was established with a view to providing a safety net program (PANES) for Uruguayans affected by the crisis. The new Administration that assumed office in March 2005 sought to improve the targeting and outreach of its safety net program through participatory monitoring and evaluation, and invited the Bank to support this initiative via non-lending technical assistance. 1.2 The design of the technical assistance program has been led by MIDES, with the Bank playing an advisory role. For example, the project’s scope, overall methodology and expected outcomes were defined in a dialogue with the Ministry of Social Development during a Bank mission in May 2006, TORs were developed jointly, the selection of consultants or firms was decided upon recommendations of MIDES, and the intermediate products are being commented and revised in conjunction with MIDES staff. Originally, in the concept note, it was stated that the UY NLTA was going to be developed over a period of a year. To make it consistent with the client’s timing and needs, the activity was divided in two phases; each one responding to specific goals and delivering specific products. 1.3 This first phase of the UY NLTA has focused on supporting the Ministry’s capacity to conduct social impact evaluation and on the design and piloting of a participatory monitoring and evaluation system by MIDES. The Bank has been working with the National Office for Monitoring and Evaluation (DINEM) of MIDES staff in the design of a participatory monitoring system and in designing and implementing an impact evaluation of the PANES program. This phase has also supported the design of participatory and monitoring pilots that will involve local civil society groups in the generation of information on user perceptions of the service delivered by social programs.3 Both MIDES and civil society representatives, particularly those involved in the Social Councils, 4 are receiving technical assistance to conduct participatory monitoring at community 2 MIDES was created in March 2005 (Law 17.866) with the main objective of formulating, coordinating, and overseen social development strategies, policies and programs in Uruguay. In order to make available a safety net to households affected by the 2002 crisis, MIDES has focused its efforts since its creation in coordinating with other agencies of the sector the implementation of PANES. 3 Participatory monitoring & evaluation is a process through which stakeholders at different levels engage in monitoring or evaluating a particular activity, program or policy, participate in the process and in the results of the M&E activity, and assume an active role in identifying and making corrective actions. On the part of the government, participatory M&E requires the incorporation of results into a system that generates information used to make decisions and improve the quality of services provision (at the project, program or policy level). 4 The Councils are comprised of beneficiaries of social programs and civil society organizations or neighborhood associations. They are organized and operate at the departmental level. level. In a forthcoming second phase of the UY NLTA, the piloted approaches would be implemented and mainstreamed. 1.4 This report summarizes the outcomes of the first phase of the Non-Lending Technical Assistance. The following sections of this report provide an overview of the context for the UY NLTA, the objectives and phases of the UY NLTA activities, the outputs and preliminary results generated in this first phase and the overall expected outcomes. The report contains a detailed report on the contributions and lessons learned from phase I, as well as a road map for phase II of the UY NLTA, including a risk analysis and a timetable of activities. 2. The Country, Institutional and Program Context for the Uruguay Non- Lending Technical Assistance (UY NLTA) A. Country Context 2.1 Uruguay has benefited from a sound governance structure, a strong civil society, and continued institutional reforms. Uruguay is an upper middle income country with a stable and mature democracy, public institutions of high quality, and a strong system of political representation based on political parties. For instance, in 2006, Uruguay was considered one of the 123 “electoral democracies� in the world and was identified as politically “free� with a score of 28 (out of 100, where higher scores indicate less freedom). 5 The 2006 Corruption Perception Index (CPI) score for Uruguay was 6.4 (out of 10, where higher values indicate less corruption), which is well above the 5.0 threshold that Transparency International defines as the ‘border line’ measure for countries that “do not have a serious corruption problem.�6 Furthermore, the country has a vast and activate network of civil society organizations, with an ample set of demands and activism at the local and national levels. The country has a long history of modernization reforms of the social sector, and development of programs of social assistance dating back to the 1940s. 2.2 The financial crisis of the late 1990s and early 2000s in Uruguay adversely affected the socio-economic conditions of the entire population. The impact was greater on the most vulnerable sections of the society. A recent World Bank 5 The Freedom of the Press scores for A (Laws and regulations that influence media content), (B) Political pressures and controls on media content) and C (Economic influences over media content) were 8, 9, and 11, respectively. The scores used to rate countries are 0 – 30 for those considered ‘free’; 31 – 60 for those ‘partially free’; and 61 – 100 for those ‘not free’. Data was obtained from the Press Historical Data of the Freedom House (http://www.freedomhouse.org/). 6 The CPI is constructed based on a Transparency International survey conducted on an annual basis, which this year was administered to 163 countries. “It is a composite index, drawing on 12 polls and surveys from 9 independent institutions, which gathered the opinions of businesspeople and country analysts.� Scores are measured on a scale of ten (squeaky clean) to zero (highly corrupt). Source: http://www.infoplease.com/ipa/A0781359.html 2 study report on Uruguayan Income Transfers7 found that the inequity and poverty indicators experienced a sustained deterioration in the early part of the crisis and particularly towards the end of it. Citing data from the National Institute of Statistics (INE) for 2006, the same study shows that although the levels of extreme poverty stayed at almost the same levels during the last decade (below 5% of the total population), the Gini coefficient grew from 0.41 in 1993 to over 0.46 in 2004. 2.3 The crisis was followed by a change in Administration, and the new Administration re-launched the country’s reform agenda, including in the area of social assistance. Upon assuming office in March 2005, the Administration of President Tabaré Vazquez committed to sustainable economic growth and of an equitable society the cornerstones of the national agenda. The establishment of a modern, efficient and accountable state that is able to promote economic development became a top priority for the Government, and the Administration initiated or re-oriented several reforms that the crisis had interrupted, including a modernization reform agenda that was re-launched in August 2006. As part of the reform agenda, the Government has been evaluating its emergency social assistance programs, known collectively as PANES (see Annex 1), and designing a new program called Plan de Equidad, to be put in place when the PANES emergency programs expire at the end of 2007. 2.4 If the Plan de Equidad is approved and MIDES mandate and functions are confirmed, MIDES will require substantial institutional strengthening, so as to undertake the reform and assume a coordinating, monitoring and evaluation role within the social assistance system. This strengthening would include the establishment of an impact evaluation capacity and the deployment of new M&E systems with strong participatory elements to strengthen service delivery. Thus the UY NLTA seeks to support MIDES in acquiring PM&E capacity and in establishing close coordination with other ministries, particularly the Ministry of Economy, and other agencies, specially the BPS (Banco de Prevision Social) and the Office of Planning and Budget (OPP). The purpose of this coordination is to build consensus about the PM&E system so that this may be useful in the implementation of the Plan de Equidad. B. Institutional Context 2.5 The Ministry of Social Development was established in March 2005 to promote greater coordination and monitoring of Uruguay’s social policy, and to address the problems of dispersion among the different programs and agencies involved in its implementation. Law No.17,866, which created MIDES, sets out the following key functions for the Ministry: (1) formulate, execute, supervise and evaluate social policies; (2) coordinate the actions, plans 7 Income Transfer Polices in Uruguay: Closing the Gaps to Increase Welfare, whose manager and main author is Rafael Rofman, May 10, 2007. 3 and inter-sectoral programs implemented by the Executive; and (3) design, organize and to operate a social information system with indicators of vulnerable groups of the population, so as to facilitate targeting of social policies and programs. 2.6 The law also authorized MIDES to establish opportunities for coordination and channel the advice of civil society (Art. 9, Law 17,866). This legal provision demonstrates the interest of the Government to promote the inclusion of voice in the assessment of social programs performance. However this provision has not been fully implemented until the present, partly due to the lack of appropriate approaches and tools that allow to implementing an effective PM&E system. 2.7 The multiplicity of institutions that take part in implementing the social policy in Uruguay requires a concerted effort of coordination to mitigate potential tensions, weak communication, or other factors limiting the impact of social programs. The government has been quite efficient in reducing these risks by creating coordination mechanisms and promoting a permanent dialogue between the different agencies. 2.8 To function properly, the formal structure of social policy coordination still needs clear assignment of responsibilities and jurisdictions. In an attempt to create a better coordination mechanism among agencies, the Government created the Social Cabinet (Decree 236/005, July 15, 2005) chaired by the Minister of Social Development and composed of 13 Ministries, plus a representative of the Office of Planning and Budget, and a member of Congress. Its main function is to advise the Executive on social programs and policies. This coordination effort is fundamental. However, dispersion of efforts has prevailed so far in the implementation of social policy, for instance, at the moment, there are three ministries, five governmental agencies, and six independent institutions involved in the design, implementation and evaluation of income transfer program (Ingreso Ciudadano). Some of these institutions focus exclusively on management, others in the macro finances, and others in design and implementation of the different programs. Even though the role of each institution is defined in the legislation, in some cases the political pressures or differences methodological approaches generate conflicting positions with respect to the programs and their implementation (Rofman 2007, Pag. 61). C. Program Context 2.9 PANES was enacted early in 2005 to address the social impact of the macro- financial crisis. Its total budget was US $200 millions. PANES encompasses seven interrelated social programs for a minimum income, workfare, nutritional support, health emergency, assisting the homeless, education for critical areas, 4 and affordable housing (see Table 1 below, and see Annex 1 for details on PANES). Table 1: Programs of the National Assistance Plan of Social Emergency (PANES)8 1. Citizen Income (Ingreso Monetary transfers of 1,360 Uruguayan pesos per Ciudadano) month per household. 2. Public Works Program (Trabajo Temporary work program to retrain in community por Uruguay) work and with compensation. 3. Nutritional Support (Apoyo Provision of basic food staples as well as health and Alimentario) education services. 4. Health Emergency (Emergencia Prevention and basic level of comprehensive health Sanitaria) care, focusing on the at-risk population. 5. Program to Assist the Homeless Assistance to homeless families and individuals (PAST) (Programa de Atención a los (shelter, meals, psychological and health care, and Sin Techo (PAST)) social promotion). 6. Educational Support in Critical Support to schools located in critical zones Zones (Apoyo Educativo en Zonas de (infrastructure, support materials, nutritional Contexto Crítico) coverage; extracurricular activities). 7. Housing Improvement Provision of housing assistance totaling US$1,000 per (Mejoramiento del Hábitat) household, including construction materials. 2.10 According to MIDES data, PANES covers 10 percent of the total Uruguayan population 9 and most beneficiary households reside in the interior, i.e. outside of Montevideo. A preliminary study of the impact of PANES programs found that over 70 percent of PANES beneficiaries live in the interior of Uruguay.(see table 2 below).10 Table 2: Applicant and beneficiary households of PANES per geographical region Group Total – country Montevideo Interior Applicant households 131,354 34,888 96,466 Beneficiary Households 78,701 22,559 56,142 Individual Beneficiaries 352,583 106,141 246,442 Rate of approval (% of 58.6 54.9 60.0 households) Applicant households 100.0 26.6 73.4 Beneficiary Households 100.0 28.6 71.4 Source: PANES indicators, processed by DINEM - MIDES (Pro-Fundación) 2.11 The same MIDES study concluded that 80 percent of PANES beneficiaries are in the first quintile of the population by income (see table 3 below). The data shows the following characteristics for this group: (i) greater percentage of 8 This is a summary from Table 1 of Annex 1. 9 “Socio-Economic Profile of the Population Included in PANES,� a study conducted by Vigorito et al (2007), IECOM, and FCCEE. 10 “Proposal: Impact Evaluation of the National Assistance Plan of Social Emergency (PANES) – Ministry of Social Development (MIDES),� conducted for the World Bank by Pro-Fundacion para Estudios Sociales, Facultad de Estudios Sociales, Universidad de la Republica. 2006 mimeo. 5 women (53.4%), (ii) most of them are young (below 16 years old), (iii) about half of them live in nuclear families, (iv) single parent household are overrepresented in comparison to the Uruguay average, and (v) about two thirds of beneficiaries (73.4%) reside in the interior. 2.12 A World Bank study (Rofman 2007) on Uruguay income transfers mentioned earlier concluded that there are three key mid-term core challenges. These are: (1) the existing gaps and overlaps in program coverage, (2) weak prospects for the fiscal and social sustainability of income transfers, and (3) evident fragmentations in the institutional design of these programs. 2.13 In this context, the Government has identified a need for quantitative and qualitative evaluations of its PANES programs, with support in part from the UY NLTA, as it designs a proposed new social assistance plan, called Plan de Equidad. In order to address the program context effectively, the UY NLTA comprised not only capacity building activities for the design of a participatory monitoring and evaluation system—which may be eventually adopted by all ministries engaged in social policy—but also included analytical elements to inform both the design of the capacity building programs and the Government’s process of transition from PANES to a new social assistance plan. In particular, it included impact evaluations to contribute to an understanding of the outcomes of PANES programs in addressing the post-crisis socio-economic conditions in Uruguay. Thus, the joint Bank-Uruguay government design of the UY NLTA was clearly shaped by key elements of the country, institutional and program contexts, which the operation seeks to address, including the need for:  Ensuring full coverage of the poorest quintile via improvements in program beneficiary identification, registry mechanisms, and monitoring of program implementation.  Systematic evaluation of the impact of PANES, in light of Uruguay’s relatively high investment in social assistance  Strengthening the capacity of MIDES to ensure effective inter-ministerial coordination of social assistance  Developing reliable monitoring and evaluation systems to inform program design and implementation  Incorporating voice mechanisms that could systematically influence policy design and accountability. 2.14 The reform of the social emergency program underway could imply changes that modify MIDES participation in the management and responsibilities of the future social assistance programs. This could constitute both a challenge and an opportunity for this Ministry. Among the challenges is the transition from PANES, which is scheduled to be faced out at the end of 2007, to the new social assistance plan, Plan de Equidad. Since it is still unclear the role of MIDES in the implementation of this new plan, the introduction of a PM&E process in the early stage of its implementation will provide MIDES a unique opportunity and relevant role in inter-ministerial coordination functions. 6 3. Objectives, Expected Outcomes and Activities of the UY NLTA 3.1 The objectives of the UY NLTA are to: (i) evaluate the impact of PANES social emergency programs; (ii) inform a dialogue about the future role of social emergency programs and of MIDES as a lead authority responsible for the coordination and oversight of social policy in Uruguay, and (iii) strengthen MIDES capacity to assess, monitor and evaluate social policy in Uruguay, with considerable weight on participatory approaches to monitoring and evaluation. 3.2 Expected outcomes from the UY NLTA include (a) increased efficiency in the implementation of Uruguay Social Programs by consolidating the existing and disperse databases and strengthening MIDES M&E system; (b) increased transparency and strengthened accountability by piloting a participatory monitoring and evaluation approach in selected departments; (c) stronger engagement of civil society in public sector reform in Uruguay; and (d) informed decisions about the future of social emergency programs, under the Plan de Equidad, and MIDES role, based on the findings of PANES poverty and social impact evaluation. 3.3 To achieve these objectives and reach the expected outcomes, the UY NLTA established specific results for each of the two phases as shown in the flowchart in Box 1. 7 Box 1 Flowchart of the Uruguay Non-Lending Technical Assistance (UY NLTA) RESULTS : 1. Evaluation capacity and design of participatory M&E Phase I system in MIDES built WB-FY07 2. Qualitative assessment of social outcomes of two Current report PANES activities and M&E design reports. Uruguay - Strengthening Participatory Monitoring and Evaluation of Social 1. Technical Note Policy 2. PM&E reports 3. Report on participatory pilots and articulation of PM&E mechanisms into the overall Phase II M&E system WB-FY08 4. PSIA report of PANES 5. Identification of TA needs and an agreed roadmap to Phase II Report establish a unified database. 3.4 Expected results in Phase I and Phase II. The result in phase I was to install MIDES institutional capacity for impact evaluation and PM&E; this was accomplished, as demonstrated in this report. The expected results in Phase II are to deploy these capacities, complete the impact evaluation of PANES, and make the PM&E system operational. A Technical Note will inform the dialogue on the future of social assistance programs, under the Plan de Equidad. The poverty and social impact evaluation of PANES will feed into the implementation strategy of the new social assistance programs. The PM&E system will generate an increased efficiency in the implementation of Uruguay social programs; and greater public transparency and accountability. This report focuses on the implementation of phase I of the UY NLTA, which sets the basis for the development of phase II. The findings and recommendations generated from the analytical work conducted and the design carried out during phase I will become inputs to produce specific deliverables in the phase II. 3.5 The activities undertaken by the Bank team in Phase I of the UY NLTA included three missions for planning and capacity enhancement and a workshop via videoconference. The scope and implementation modality for the UY NLTA were agreed with MIDES in the initial mission in May 2006. In October 2006, pilot Social Councils were selected for participatory M&E and 8 details of the qualitative and quantitative analyses were elaborated with MIDES. Pursuant to the mission, a civil society organization, the Research and Development Institute (IDES) was engaged to develop a training program for the Social Councils on participatory M&E. Preliminary reports on two PANES programs were delivered in December 2006, and in March 2006 a workshop was delivered via videoconference on general concepts of Participatory M&E in Social Policy and Social Audit Mechanisms. A further workshop held in Montevideo in April 2007, focusing on the conceptual part of Monitoring and Evaluation System in Social Policies, at which concrete examples of international best practice on the application of social accountability methodologies and tools in M&E systems for social programs were presented to MIDES and the consultant team. In May 2007, the consultants for the Qualitative analysis delivered their preliminary reports for this phase of the NLTA. A detailed description and timeline of the UY NLTA activities can be found in Annex 2. 3.6 The FY08 deliverables of phase II are (1) a Technical Note produced based on, among other analytical inputs, on preliminary data gathered through the first panel of the household survey and the qualitative study; it will include suggestions on the future of social assistance programs in Uruguay and it will be jointly prepared by Bank teams (LCSSO, LCSHS and LCSPP) with the participation of MIDES’ research team, (2) the production of M&E Reports, (3) the preparation of a Report on Participatory Pilots and articulation of participatory monitoring mechanisms into the overall M&E system, (4) a report analyzing social and poverty impacts of PANES; and (5) an identification of technical assistance needs and an agreed roadmap to establish a unified database. 4. UY NLTA Contributions under Phase I to Strengthen Impact Analysis Capacity in MIDES 4.1 This section describes the contributions the UY NLTA has made to strengthen impact analysis capacity in MIDES. It begins with a brief summary of the assistance rendered during the data collection for impact evaluation, before turning to the contributions on quantitative analysis and qualitative analysis. A. Data Collection for Impact Evaluation 4.2 The UY NLTA contributed to data collection for impact evaluation through the design of a PANES Follow Up Household Survey. In addition to standard socioeconomic status questions, the survey’s questionnaire collects relevant information regarding the use of cash transfers (Ingreso Ciudadano and other government transfers), participation in and effects of other PANES activities (Trabajo por Uruguay and Rutas de Salida). The questionnaire also includes a module for intra-household allocation of resources, gender issues, and the use of 9 time in households, which are all important dimensions to evaluate PANES and derive recommendations for the design of the Plan de Equidad programs. 4.3 A special sampling framework was developed for the PANES follow up household survey because poor households are under-represented in the National Household Survey’s sample. The special sampling framework is based on that of Uruguay’s periodical National Household Survey, but is expected to capture a more representative sample of poor households, female headed households, and Afro-Uruguayans. 4.4 The UY NLTA also supported an analysis of existing Registry data: PANES was implemented in several stages, adding new beneficiaries and graduating beneficiaries periodically. MIDES collected and processed socioeconomic data from all participants both at the point of entry to the program and the point of exit. This feature allows for the use of this data for impact evaluation purposes using different approaches comparing individuals who exit the program with similar individuals entering the program around the same time B. Quantitative Impact Analysis 4.5 The UY NLTA has provided support to MIDES in the design of an impact evaluation using quantitative methods. The evaluation of the impact and results of PANES address the following research areas: (1) direct and indirect impacts of the cash transfer program (Ingreso Ciudadano) on poverty reduction and income distribution among PANES’ beneficiaries relative to a comparison group; (2) the number of beneficiaries who will still need cash transfer and other assistance once PANES is over; (3) differentiated impacts of different types of assistance and interventions (cash, lifelong learning, social inclusion, public works) on poverty reduction; (4) impacts of all PANES interventions on human capital (education, training, and health) and social capital (networks and relationships of trust and mutual information/assistance); (5) impact at the household level on strategies for coping with risk and intra-household allocations of resources; (6) impact on gender roles, women’s economic empowerment, and youth development; and (7) impact on labor markets (labor supply, unemployment, mobility). 4.6 The quantitative impact analysis draws on comparative and counterfactual analyses through various complementary statistical approaches using data from the PANES follow-up survey and other statistical sources. The counterfactual analysis is designed to compare PANES beneficiaries (treatment group) to non-beneficiaries (comparison group) using two techniques: (1) Regression Discontinuity (RD), a parametric method that assumes an underlying regression modeling the relationship between a set of control variables (such as age and education) over the outcome variable of interest (income, for instance), and establishing a kink in the underlying regression due to a shift produced by the intervention that is being investigated. The evaluation can be conducted by 10 observing the differences in outcome between participants and non-participants for individuals (or households) very similar to each other and located close to the shifting point; (2) Propensity Score Matching (PSM), a semi-parametric or non- parametric method, involving the construction of ex-post treatment and comparison groups by “matching� individuals who are statistically identical in a group of characteristics, except for their participation in the program.11 C. Qualitative Impact Analysis 4.7 The design phase of the qualitative evaluation12 was developed from August to October 2006 in consultation with multiple stakeholders. The stakeholders included experts from academic institutions, political parties, officials and technicians from government agencies, territorial delegates from MIDES, and the protagonists of the various programs. The UY NLTA drew on expert panels, stakeholder interviews, in-situ observation of the programs, and analysis of existing databases to inform the design. 4.8 The consultations led to the formulation of several hypotheses that can be contrasted with the preliminary results of the qualitative analysis described below. The consultations gave rise to the following main hypotheses: (a) the participation of different types of civil society organizations (CSOs) generates different program-wise results; (b) the relationship between the protagonists (benefiting from program support) with the community can determine or catalyzing program outcomes, which have the potential to become visible, strong and durable if the right conditions for this relationship are established; (c) the protagonists, in general, carry a stigma that conditions their relationship to the community to which they belong; (d) some forms of capacity building generate greater linkages to the labor market and enjoy higher approval ratings among protagonists than others, i.e. vocational training; (e) the programs lead to a greater value being assigned to work and to a culture of working; (f) the programs contribute to the development of various forms of networks and linkages among protagonists; and finally, (g) a trend of ‘feminization’ (greater number of female protagonists) of PANES programs could be perceived by male protagonists as a threat to their authority. 4.9 A PANES program Trabajo por Uruguay (TPU), and a training activity for PANES beneficiaries Rutas de Salidas were selected as case studies by MIDES. ‘Snapshot’ and ‘tracing phases’ were developed to evaluate each program, using in-depth interviews and focus group approaches. Both activities were considered by MIDES to be the most appropriate ones for demonstrating 11 For more details on the statistical analytical tools, see Annex 1 of the Concept Note. 12 See documents: “Bases para el Diseño de una Evaluación Cualitativa – Relatoría de la Fase 0� by Ignacio Pardo and MIDES. 2006, mimeo. “Plan de Atención Nacional a la Emergencia Social (PANES): Programas ‘Trabajo por Uruguay’ y ‘Rutas de Salida’ – Bases para el diseño de una evaluación cualitativa,� prepared by MIDES. 2006, mimeo. 11 how the target population in extreme poverty can exit the emergency programs and engage in income-generation activities. In each case, the UY NLTA recommended the use of in-depth interviews and focal groups approaches. The in- depth interview methodology is semi-structured, as the gathering of information requires that the format not be too rigid, and in a language that is accessible to the interview and that inspires self-reflection. Focus group approaches enabled the moderators to engage individuals in the construction of a common or collective discourse that involves multiple negotiations among participants.13 The qualitative analysis will be carried out in two phases. The first phase provides a ‘snapshot’ of protagonists, evaluating their satisfaction, results and expectations. Phase two makes ‘tracing evaluations’ recognizing that tracking the trajectory of protagonists into the labor market is essential to measuring the impact of the activities. 14 Tables 3 and 4 illustrate the parameters for evaluation under the ‘snapshot’ and ‘tracer’ phases of the qualitative evaluation, as well as the sample and interview techniques for each phase. Table 3: Parameters for Evaluation in Qualitative Analysis Evaluation Parameters Snapshot - Tracer - Phase I Phase II Labor training and competencies X X Formation of social capital within the household and X X among social networks Formation in citizenship and insertion in a rights culture X X Labor aptitudes X X Communication aptitudes X Math aptitudes X Cognitive aptitudes X Integration in the educational system; X Self-care and access to health services X Equity and intergenerational relations within the X X household equity and intergenerational relations within the household Integration to educational system X Access to health care and oral health care X Summarized from “Plan de Atención Nacional a la Emergencia Social (PANES): Programas ‘Trabajo por Uruguay’ y ‘Rutas de Salida’ – Bases para el diseño de una evaluación cualitativa,� prepared by the World Bank and MIDES. 2006, mimeo. 13 The composition of the focus groups responds to principles such as feasibility, internal homogeneity, and (non-statistical) representativity, and moderators are provided with a minimum guide to facilitate the focus group discussion, which in turn is to focus on one or a very limited number of topics. 14 Part 1 of the evaluation is an exhaustive description of the program components and their relationship to the protagonists, in which strengths, weaknesses and constraints are identified. Part 2 of the evaluation both tracks the protagonist into the labor market and observes possible changes taking place within the household after graduation from TPU and RDS, as well as identifying networks and attitudinal changes generated during the exit period. 12 Table 4: Techniques and Sample for the Qualitative Analysis Design Type of technique, Sample description phase total number of events Snapshot – 16 focal groups based Locations: Montevideo and metropolitan area and Phase I on socio-structural for the interior. Distribution per location: 2 female variables, especially and 2 male groups per age group (18-20, and 30 gender/age distribution plus) by location Tracer – 16 interviews of 4 interviews (2 of women and 2 of men) who have Phase II protagonists conducted evaluated (positive or negative) and are (or not) in at three different search of work [4 possible outcomes] moments (total of 48 interviews) Summarized from “Plan de Atención Nacional a la Emergencia Social (PANES): Programas ‘Trabajo por Uruguay’ y ‘Rutas de Salida’ – Bases para el diseño de una evaluación cualitativa,� prepared by MIDES. 2006, mimeo. 4.10 The preliminary design of the qualitative analysis was elaborated in December 2006, further conceptualized by March 2007. Fieldwork for phase I was carried out in April and a report with preliminary findings was made available in May 2007. The report presents the theoretical basis used in the analysis, as well as the proposed indicators (see ANNEX XI). Table 5 summarizes preliminary results of the qualitative assessment of the Rutas de Salida and Trabajo por Uruguay activities under PANES. Table 5: Summary of Preliminary Results of Qualitative Analysis of PANES Programs Area assessed Rutas de Salida Trabajo por Uruguay Type of training Labor training: Courses were viewed as The workshops covered labor workshops responding the needs of program rights, family violence, sexuality, beneficiaries consulted (computer use, drug addiction, among others. primary education). Beneficiaries felt that in general the Citizenship & rights: They gained trainings were useful but some too better sense of being ‘included’ and theoretical or unproductive. They ‘heard’; learned about obligations and learned better about how to demand how to exercise rights. their rights, and their personal Social/generic competencies: Were development expanded in that they found to be helpful for personal learned personal hygiene and development (self-esteem) and work reproductive methods. ethics (punctuality, presentation) Tasks assigned to Great satisfaction was gained to Beneficiaries that undertook jobs in perform perform tasks that contribute to public institutions (schools, health improve the neighborhood or city, and units) performing construction and to help forge a culture of cleanness and maintenance tasks had a better good maintenance. evaluation than those that performed other public works (landscape, water path clearing). Neither group got uniforms on time, nor they had safe/clean working conditions. 13 Table 5: Summary of Preliminary Results of Qualitative Analysis of PANES Programs Area assessed Rutas de Salida Trabajo por Uruguay Relationships with These generated: mutual support among Some beneficiaries characterized other program program beneficiaries; establishment of relations with NGO technicians as beneficiaries, friendships and comrade linkages. horizontal while others as rigid and NGO workers, There were diverse opinions on unfriendly with too much demand community relationship with NGO workers; on workers. The relationship with members considering them fluid if they existed other program participants was prior to the program. viewed as positive. Relations with the community seemed to have helped them to improve their image. Labor ‘Trajectory’ Most beneficiaries were discouraged Most beneficiaries felt they and Exit strategies because they had not been able to find acquired obsolete techniques not jobs due to lack of training, age or useful in the market place. Those criminal record. Those seeking to not working for public institutions establish small enterprises submitted (see tasks assigned) felt that they proposals but either got a negative did not acquired new skills at all. answer or never heard from authorities. Neither group concluded that the skills learned prepared them enough to find jobs. No project or enterprise among beneficiaries was found to be functioning due to their lack of tools and financing. Complementary These services were considered very [None were described in the social assistance useful, particularly the dental care, preliminary report.] provided which was said to help improve self- esteem and presentation when job searching. Source: Preliminary reports of MIDES qualitative analysis, 2006. 5. UY NLTA Contributions under Phase I to Participatory Monitoring and Evaluation by MIDES 5.1 This section describes the contributions that have been and are being made under the participatory monitoring and evaluation component of Phase I of the UY NLTA. The UY NLTA provided support in the following five key areas, described below: (1) design of software and technological infrastructure for the PM&E system; (2) definition and establishment of an adequate set of indicators for the M&E system; (3) piloting of PM&E experiences led by the Social Councils, and (4) sharing of international best practice in PM&E and social accountability mechanisms. 14 A. Design of software and technological infrastructure for the PM&E system 5.2 The proposed Integrated System of Social Information combines information at the level of beneficiaries, the social programs themselves, and the social expenditures. The definition of these essential elements will contribute to the preparation of a roadmap for the development of a unified database that will enable government agencies to manage information more easily on: family income, access to formal jobs, taxable and non-taxable services, health status, educational performance, citizen participation and rights, and access to public services.15 The system will be created over the medium and long term since it implies construction and articulation of complex initiatives and tools, such as those for planning, monitoring and evaluating social programs. 5.3 The UY NLTA supported the hiring of specialized support to develop the terms of reference for the design of the software for the Information System of the M&E system. The software will be designed to make use of social assistance data and indicators. The unified database will combine data from various sources and become a keystone of the PM&E system. In May 2007, the Information Specialist produced a report proposing TORs that contain the following steps: (i) identification of the physical, technological resources and of communication existing in the current technological platform of information that MIDES uses; (ii) design and installation of the necessary software base for the new system; training plans for user groups designated by MIDES; (iii) preparation of an approach and specifications of the functional and structural tests necessary for making adjustments to the system; (iv) creation of user manuals as well as the technical manuals for implementation, configuration, installation and operation of the system; and (v) installation of the DataWarehouse and creation of a server. 15 See activity report “Fortaleciendo la capacidad del MIDES para el análisis, el monitoreo y la evaluación de la política social del Uruguay. – Propuesta de unificación de bases de datos con variables sociales existentes en las diferentes dependencias del Estados.� prepared by Milton Silveira, Mayo de 2007. 15 B. Indicators for the Participatory M&E System 5.4 To define the PM&E system, MIDES is evaluating the usefulness and appropriateness in PM&E of a set of international, regional and national systems of socio-economic indicators. Among them are those developed by international and regional institutions such as OECD, UNRISD, the United Nations and CEPAL. The proposed country experiences for inquiry are those of Argentina (Statistical System of Social and Demographic Indicators and the Information, Monitoring and Evaluation of Social Programs System), Costa Rica (the System of Indicators on Sustainable Development), Venezuela (the Integrated System of Social Indicators), Ecuador (the Integrated System of Social Indicators), Panama (the Integrated System of Development Indicators), and Chile (the Integrated System of Social Information). In Uruguay, the systems in place proposed for analysis are the National Statistics System and the Statistical Catalogue developed by the National Institute of Statistics, the Observatory of Labor Markets managed by the Ministry of Work, the Observatory of Montevideo on Social Inclusion developed by the Municipality, the Primary Education Monitor established by ANEP, and the Informational System on Infants. The, systems under design or implementation suggested for study are the Statistical Master Plan which is part of the National Strategy for Statistical Development; the Social Protection Survey carried out by BPS; and the Integrated System of Information of Social Areas and the National System of Information on Infants and Adolescents promoted by MIDES. C. Piloting of Participatory M&E Experiences by the Social Councils 5.5 MIDES is committed to strengthening the participation of civil society through social empowerment and considers local Social Councils to be essential for constructing the social agenda at the department level. The Councils are expected to become conduits of citizen ‘voice’ and to fulfill monitoring and evaluation functions at the territorial level to improve the quality of service delivery. This implies a change in the institutional culture of public institutions including information transparency measures and response to recourse mechanisms. The proposal to constitute a system of participatory M&E required the development of a set of mechanisms and procedures for the Social Councils to fulfill their participatory M&E mandate appropriately, as well as training for Social Councils and piloting of these approaches. 5.6 The Social Councils of Rivera and Tacuarembó were selected by MIDES to pilot participatory M&E mechanisms. The Councils are composed of departmental social actors at various organizations levels and from various areas of work, who can propose, contribute and monitor social policies. These Councils, like others in the future, will have a dual function as users and suppliers of information for the participatory M&E system. The two Councils were 16 established in mid-2006 and are now engaged in planning the pilot experiences.16 The Rivera and Tacuarembó Councils have carried out seven working meetings during the past year, with inputs from MIDES and the UY NLTA (specifically, through IDES), to prepare for the launching of the pilots, which will take place under Phase II of the UY NLTA. The training, which will be extended to other social Councils, includes capacity building to ensure close cooperation between local communities and the Councils, and developing a confident public voice vis- à-vis the State. It will also provide support to procedures for accessing public information. The proposed pilot will ensure that the participatory information generated by the trained Social Councils will actually start feeding into the PM&E system. IDES will also work with the Social Councils to select the programs to be monitored, implement the field work, process the data, and prepare a report containing an analysis of results obtained from the pilot experiences. This should enable the lessons learned and approaches to be replicated in other departmental Social Councils. D. Sharing of International Best Practices with Participatory M&E 5.7 At the core of the Non-Lending Technical Assistance for MIDES is the sharing of international knowledge, experience and best practices on participatory M&E. As a new Ministry with a strong commitment to social accountability, MIDES’ staff has rapidly increased its knowledge of theory and practice in participatory M&E through the continued dialogue, training workshops, and the sharing of international experience. The Bank has presented social accountability tools and methods that it has internationally identified and promoted as part of a participatory M&E approach, including for example Participatory Rural Appraisal (PRA), Public Expenditure Tracking Surveys (PETS), Citizen Scorecards and Community Report Cards (see Box 2 below). These have been shared with MIDES during the past year, and MIDES is drawing on several of these elements in the design of its participatory M&E approach. Box 2. Selected Tools for Participatory Monitoring and Evaluation The Public Expenditure Tracking Survey (PETS) is a quantitative survey of the supply side of public services. The unit of observation is typically a service facility and/or local government, i.e. direct providers like schools and clinics. The survey collects information on facility characteristics, financial flows, outputs (services delivered), accountability arrangements, etc Citizen Report Cards are participatory surveys that provide quantitative feedback on user perceptions on the quality, adequacy and efficiency of public services. They go beyond just being a data collection exercise to being an instrument to exact public accountability on the quality of public services through the extensive media coverage and civil society advocacy that accompanies the process 16 Currently there are nine Departmental Social Councils in place, and two departments are in the final phase of establishing their councils. 17 Community Score Cards are tools of qualitative monitoring that are used at the local level for the community to evaluate the functioning of public services, projects and administrative units. The community score card is a hybrid process that involves social audit, participatory community monitoring and citizen report card techniques. Like in citizen report cards, the community score cards are a strong instrument of the users of social programs and the citizens in general to monitor and demand that social services meet certain basic quality standards. They do it by scoring the quality of the services received based on their experiences and perceptions. Community Score Cards are conducted in community assemblies and meetings with the involvement of service providers 6. Next Steps – Phase II of the UY NLTA 6.1 This section lays out the proposed outputs and road map for Phase II of the UY NLTA. It briefly outlines the elements of a technical note to be produced in Phase II drawing on the preliminary results of the quantitative and qualitative impact analyses, and presents the contributions that are expected in relation to the poverty and social impact analysis of PANES and the participatory M&E component, before closing with a road map for Phase II of the UY NLTA. A. Technical Note Based on Preliminary Qualitative and Quantitative Impact Analyses 6.2 Phase II will include the preparation of a technical note, which will contribute analytical elements to the dialogue on the proposed Plan de Equidad. The Technical Note will include: (1) a results assessment of PANES’ Ingreso Ciudadano (income transfer) program on the following dimensions: poverty, employment and labor supply, human capital, social capital, gender roles, and intra-household resource allocation; (2) a qualitative results assessment of the Trabajo por Uruguay and Rutas de Salida activities, including the same dimensions as for the quantitative analysis, as well as an evaluation of the quality of training and other services provided by civil society organizations; (3) conclusions and recommendations for Plan de Equidad, with key messages for policy and program design.. The main inputs for this technical note are the results of the first round of the PANES Follow Up Household Survey and of the comparison group, the results of the qualitative analysis of Rutas de Salida and Trabajo por Uruguay, the Bank study on income transfers, and other studies and analytical work available. 6.3 The main audience of the technical note will be the Inter-Ministerial Social Cabinet, led by MIDES. Preparation of this note has been approved by MIDES Minister and Directors and its objectives have been discussed with the National Office for Monitoring and Evaluation (DINEM) at MIDES. In light of the broad 18 span of the analysis, an inter-disciplinary Bank team with specialists from different World Bank sector staff (LCSSO, LCSHS and LCSPP) will work jointly with MIDES’ core evaluation team to prepare the technical note. During the preparation, inputs from other ministries and government agencies will be sought as well. B. Implementation of the Participatory M&E System 6.4 The participatory M&E system to be implemented is expected to be highly flexible, accessible yet comprehensive. The system is intended to be simple enough to serve as an input to the work of the Social Councils in terms of their participatory M&E activities. However it should also be sufficiently reliable and complete to guide the design of new social policies or reformulate existing ones. It is expected to allow the consultation and preparation of reports for different users including technicians, political decision makers, social organizations, and citizens in general. The system will be capable of generating on-line consultations by remote users, perform data security and privacy control, create a users network that receives thematic updates of data, issue specific reports according to the selected thematic modules or indicators, record visitors based on a pre-established protocol, and generate reminders on the updating of data available to administrators. Users would include the Social Councils, the Social Cabinet, the National Council of Coordination of Public Policy, the Inter-institutional Roundtables (see below; most of these coordination bodies are also providers of data for the system). The system will ensure the possibility of incorporating changes and expansions to the system in each of the abovementioned aspects, via a modular or expandable structure. 6.5 Integrating Participatory M&E feedback information into the eventual development of a unified database is a key objective of MIDES’ National Office for Monitoring and Evaluation (DINEM). 17 Through the UY NLTA, DINEM will promote an inter-institutional dialogue that will increase the results orientation of social policy and facilitate more rapid adjustments in social policy in response to conditions among the protagonists. Specifically, the PM&E system under construction will indicate if a program or policy being implemented is in accordance to its plan; identify problems, irregularities and difficulties, as well as opportunities in the implementation of social policies; audit and control the execution of the social policy budget, and continuously evaluate public policy actions that are relevant to the community. More specifically, the activity will increase capacity related to the selection, combination, and recommendation of technical instruments for the system’s information development (characteristics of the solution, requirements, functionalities, application models, etc.), as well as the most suitable institutional tools for the system’s management (Management 17 “Consideraciones respecto del proceso de construcción de un sistema de Evaluación y Monitoreo Participativo. Ministerio de Desarrollo Social - Uruguay� - discussion paper on the construction of the PME system prepared by Milton Silveira, Mayo de 2007. 19 Committee, etc.). This capacity building will be deepened and expanded to other government agencies in Phase II of the NLTA. 6.6 Uruguay produces large volumes of data on its social programs but there is no unified database for PANES and other social assistance programs. In particular, the State does not have adequate information systems to update, integrate, disaggregate, regularly generate and disseminate comprehensive reports on the results of the programs. In fact, various government agencies have registers with critical information about PANES beneficiaries and applicants but there is no data coordinated for running the PANES.18 The creation of a unified database will be an important step towards addressing problems of duplication in datasets, as well as the existence of similar (but different), non-harmonized data. The Team will engage key stakeholders like BPS, OPP, INE, and MIDES in a consensus building process to agree upon a roadmap for the development of this unified database. 6.7 The UY NLTA will support the development of data audits in order to ensure quality data. The system already considers including “auditing tracks� that will enable system managers to track down who and when information was entered into the system. The feasibility of creating an independent mechanism for data quality responsible for conducting external data audit will be assessed in phase II. The UY NLTA team will promote an assessment of the installed capacity within civil society and among universities and other think tanks to perform this function. This will also include an assessment of the methodological approaches more appropriate for PM&E. 6.8 The new system will seek to combine the M&E conducted by public offices with the M&E carried out by social groups and neighborhood associations. MIDES will make the effort to articulate the production of statistical data based on social indicators, and the feedback of community groups. The keystone of this approach will be the capacity building and piloting of participatory M&E mechanisms by the Social Councils. A preliminary decision to pilot the Community Score Card methodology has been adopted by MIDES for the pilot experiences. Furthermore, MIDES plans to link this effort to existing inter- institutional coordination bodies such as the Inter-institutional Roundtables (Mesas Interinstitucionales) comprising representatives at the departmental level from public institutions engaged in social policy implementation. They seek to promote horizontal and transversal coordination of social policies at the territorial level. Because MIDES convenes these instances of local coordination, it is positioned to promote linkages with the Social Councils during the piloting phase. 18 Those in charge of the individual program databases met in a workshop held on June 14, 2006 to map the existing databases to which the PANES system can access, i.e. from of the National Institute of Statistics (INE), Banco de Prevision Social (BPS), Ministry of Public Health (MSP), Administration of Public Education ANEP, enterprises of public services (UTE, OSE and ANTEL), the electoral court, the Ministry of Housing, and other programs within MIDES (Infamilia). 20 6.9 The UY NLTA will continue to promote international best practice in the design of PM&E. The Team will support MIDES to facilitate the creation of a space for inter-institutional and territorial policy dialogue and review good practices, such as those of the Scandinavian countries. These practices will enable MIDES to implement new approaches that will make possible the sustainability, legitimacy, informed citizen participation, and transparency that the social public policy requires in Uruguay. 6.10Análisis de Impacto Social y Sobre la Pobreza de PANES. The counterfactual analysis will compare the beneficiaries of PANES (treatment group) with the no beneficiaries (comparison group) using parametric and nonparametric methods to consider the impacts on PANES households that the plan has reached. 19 More specifically, it will analyze the direct and indirect impacts of the program of income transfers (Ingreso Ciudadano) on poverty reduction and income distribution. It will contribute to estimate the number of beneficiaries who will still need cash transfers and other assistance once PANES concludes. It will analyze the differential impact from different interventions (monetary, permanent qualification, work in exchange for cash) on poverty reduction. It will look into the impacts on human and social capital; the intra-home allocations of beneficiaries of PANES resources, and other gender and ethnic dimensions. It will also look into the impacts on the job markets (employment, unemployment and mobility). This evaluation will offer important elements for the implementation of the Plan de Equidad as of 2008. 7. Preliminary Lessons Learned and Suggestions for the Implementation of Phase II 7.1 From May 2006, when the NLTA started its implementation, to-date, the key lessons learned could be summarized into six broad areas. These lessons will inform the strategy for the implementation of Phase II:  It is fundamental to undertake an empowerment approach in the delivery of technical assistance that has a strong capacity building focus. It is essential to facilitate a process of appropriation by the client of the Bank contributions, whereas this is provision of technical assistance, transfer of international experiences, or financing of activities. Taking this approach in the UY NLTA has been critical for MIDES to assume ownership of the design and implementation of both the impact evaluation and the PM&E system. The permanent and open dialogue that has been established between MIDES management and teams and the Bank’s multidisciplinary team has been a critical element so that MIDES assumes a leadership role in the UY NLTA, having the Bank play a support role, responding to the Ministry’s capacity building needs and demands, and involving MIDES management in leading the implementation of the technical assistance activities funded by the UY NLTA. 19 For more details about the quantative analytical tools, see Annex 1 of the Concept Note. 21  To broaden country’s ownership of the social policy, it is imperative to recognize the need to integrate the voice of the protagonists of the development actions, specially the people living in extreme poverty who benefit from social programs. It is necessary to develop technical assistance operations that provide simultaneous assistance to governments and to civil society organizations or beneficiary groups involved in the development of social policy. The UY NLTA has been focused not only in strengthening institutional capacity within MIDES but also on building capacity at the local level. As part of these efforts, it is most relevant the support provided to the Social Councils and the articulation that MIDES is pursuing between these Councils and the Inter- institutional Roundtables at the regional level. During phase I of the UY NLTA implementation, MIDES strengthened its participatory monitoring and evaluation mechanisms so that users’ voice may improve the focus and quality of social programs that MIDES manages or coordinates. MIDES has understood this and is working with IDES to build greater capacity among the Social Councils.  It is important to deliver technical assistance operations that are sufficiently flexible to respond to the client needs, particularly in changing or uncertain environments. Technical assistance operations delivered with a rigid approach are less likely to succeed in making programs adaptations necessary when the country, institutional or program context is modified or uncertain. Uruguay is going through substantial institutional changes, particularly in the area of social policy. These conditions are characterized by uncertainty about the future of MIDES role and of social assistance programs. The flexible nature of the UY NLTA has enabled the Bank’s multidisciplinary team to be more responsive to the specific needs of MIDES during this transition period with the purpose of serving the target population.  Vital to the development of any technical assistance operation is that of sharing of international best practices. Examples from other countries inform and emulate the action of client countries in the path of innovation and excellence. However, a delicate balance has to be achieved between educating clients about good practices and promoting their ownership. The sharing of best practices with MIDES has started only recently In this process, it is important that MIDES receives detailed and updated information on the international best practices so that it may internally consider and debate which of them are adequate for adaptation to the context. This can contribute to illustrate the way in which other countries have managed to overcome resistance to participatory processes, often entrenched in vertical institutional cultures preferring "top-down" approaches. In Phase II of UY NLTA, a multidisciplinary team of the Bank could support the deepening of the application of international good practices adopted in this context, giving special emphasis to the peer-to-peer exchange of experiences with the institutions and processes of other countries. Also, the World Bank team will promote a frank dialogue with MIDES management to contribute to identify and 22 to overcome any possible perception that the Bank may be promoting intervention models that do not correspond with the reality of the country.  Inter-governmental coordination is critical for an effective design, monitoring and evaluation of social policy. Because there are many actors involved in social policy in Uruguay, a concerted effort of coordination is required. The PM&E system that the UY NLTA supports has help MIDES gain increased capacity for the evaluation of the social policy of the country. In the second phase of the UY NLTA could be opportune to extend the access of PM&E methodology to other ministries, given their role in the design and the joint execution of the country’s social programs. Some ways to do this are: (a) to expand and update the stakeholder analysis to include the other ministries involved in social policy; and (b) to support MIDES in the socialization of the preliminary results of the qualitative analysis in the Social Cabinet, as well as the design of the quantitative analysis and the plans for the establishment of the PM&E system (including the experiences of the PM&P pilots that will be executed with the Social Councils). 8. A Road Map for Implementing Phase II of the UY NLTA 8.1 An important working relationship has been developed with MIDES in Phase I of the UY NLTA. MIDES is highly committed to incorporating social accountability in the design of a new social policy for Uruguay with improved inter-ministerial coordination, enhanced information systems and policies grounded both on citizen voice and on careful qualitative and quantitative impact analyses. It will be important to consolidate this work via phase II by maintaining a fruitful dialogue with MIDES and the Social Cabinet. 8.2 Special attention will be paid to the risks associated to the implementation of phase II of the UY NLTA. Table 6 below depicts those risks and the way in which they will be addressed. Table 6 - Risks Faced in Phase II Implementation and Proposed Measures Risks Measures Potential lack of interest on the To develop a broader base of support within the UY part of other ministries and public government, the Team will promote a greater dialogue institutions to engage in the between MIDES and BPS (Banco de Prevision Social), proposed activities. the Ministry of Economy and other key actors in order to build consensus on the objective and reach agreement about the UY NLTA process. The development of a road map The Team will coordinate with various ministries and for the establishment of the intergovernmental offices involved in social assistance, unified database may encounter a the OPP (Office of Planning and Budget) and other strong institutional resistance on public offices engaged in monitoring activities. It will the part of ministries and other suggest the government to establish clear incentives so public institutions that control that the information is released completely and in a social assistance data. This is timely manner. Relevant international experience should 23 Table 6 - Risks Faced in Phase II Implementation and Proposed Measures likely to result in continued be presented to the UY government so that it may institutional overlaps in M&E generate a dialogue among the main stakeholders functions associated with weak (MIDES, BPS, National Institute of Statistics and inter-institutional coordination in others). the implementation and use of the M&E system. Resistance on the part of MIDES The team will collaborate with MIDES and IDES in the to engage the Social Councils in development of training activities for the Social Council the development of innovative that include international best practices shared with PM&E mechanisms. them during phase I. Peer-to-peer learning with other government institutions in other Latin American countries will be promoted as well. Expansion of PANES coverage To advice DINEM to coordinate closely with MIDES may create problems in management to assuage any modifications of the measuring impact among baseline due to changes in the eligibility criteria. comparison groups, particularly when members of the comparison groups become program beneficiaries. Commitment on the part of client The Team will seek the necessary client’s explicit may not be same as in phase I. support to carry out the proposed activities for phase II contained in this report 8.3 The key steps and activities of phase II are presented in the road map below (table 7), which closes out this report on the NLTA for participatory M&E of Social Policy in Uruguay. Table 7 - Timetable of implementation for the second phase Date Type of Event Activities July - August Linkages with Joint mission of Rafael Rofman and William Reuben 2007 other Bank to further link the UY NLTA to the Income Transfer operations Policies ESW. William will further coordination with Enrique Fanta to link the NLTA with Institutions Building TAL. August 2007 Technical Note Discussion with Social Cabinet based on First Panel draft of Survey and Qualitative Analysis results. September 2007 Revised version of Technical Note August to Pilot Program Implementation of Social Councils’ Pilot Program December 2007 July to Social Information Development and implementation of social December 2007 System information and M&E systems February 2008 Internal Knowledge management activities: translation of the Dissemination document prepared by local consultants into English March 2008 Impact Evaluation Impact Evaluation Report April 2008 Report UY NLTA Phase II Draft Report May 2008 Decision Meeting Decision Meeting UY NLTA Phase II May 2008 First Delivery to Workshop to deliver results to MIDES the Client 24 Table 7 - Timetable of implementation for the second phase Date Type of Event Activities May 2008 Closing Date May 30, 2008 is TF closing date June 2008 Final Delivery to Seminar in Uruguay of the final report to UY Client Authorities and GDLN events on experiences in Participatory Monitoring 25 References Campos, H., et al. (2006) Diseño Conceptual de un Sistema de Monitoreo y Evaluación Institucional y Participativa del Bienestar Social del Uruguay – Primer informe de avance. MIDES Pardo, I. (2006) Presentación Preliminar de los Resultados del Campo de la Evaluación Cualitativa. Componente: Trabajo por Uruguay (Mirada y voz de las y los protagonistas). MIDES. Pardo, I., et al. (2006) Plan de Atención Nacional a la Emergencia Social (PANES): ‘Programas Trabajo por Uruguay’ y ‘Rutas de Salida’ – Bases para el diseño de una evaluación cualitativa – Relatoría Fase Cero, MIDES. Pardo, I., (2006) Plan de Atención Nacional a la Emergencia Social (PANES): ‘Programas Trabajo por Uruguay’ y ‘Rutas de Salida’ – Bases para el diseño de una evaluación cualitativa, MIDES. Pro-Fundación para Estudios Sociales, Facultad de Ciencias Sociales de la Universidad de la República (2006) “Proposal: Impact Evaluation of the National Assistance Plan of Social Emergency (PANES). MIDES. Rofman, Rafael. (2007) Income Transfer Polices in Uruguay: Closing the Gaps to Increase Welfare. World Bank. Silveira, Milton (2007) Fortaleciendo la capacidad del MIDES para el análisis, el monitoreo y la evaluación de la política social del Uruguay. – Propuesta de unificación de bases de datos con variables sociales existentes en las diferentes dependencias del Estados. MIDES. Silveira, Milton (2007) Consideraciones respecto del proceso de construcción de un sistema de Evaluación y Monitoreo Participativo. Ministerio de Desarrollo Social - Uruguay. MIDES ED4Admin C:\Documents and Settings\wb188597\My Documents\Operations\Uruguay\UY NLTA report - phase I - Final Draft.doc 22/06/2007 0:34:00 26