Document of The World Bank Report No: ICR00002865 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-75470 IBRD-78710) ON A LOAN IN THE AMOUNT OF US$1,435.56 MILLION TO THE STATE OF MINAS GERAIS, BRAZIL FOR THE SECOND MINAS GERAIS DEVELOPMENT PARTNERSHIP PROJECT April 29, 2015 Governance Global Practice Brazil Country Management Unit Latin America and the Caribbean Region CURRENCY EQUIVALENTS (Exchange Rate Effective March 13, 2015) Currency Unit = Real (R$) 1.00 = US$ 0.353 US$ 1.00 = 3.22 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS AF Additional Financing CPS Country Partnership Strategy DER Roads Department of the State of Minas Gerais (Departamento de Estradas de Rodagem de Minas Gerais) DLI Disbursement-Linked Indicator DPL Development Policy Loan EEP Eligible Expenditure Program FHEMIG Hospital Foundation of Minas Gerais (Fundação Hospitalar do Estado de Minas Gerais) FRL Fiscal Responsibility Law (Lei de Responsabilidade Fiscal) GoMG Government of Minas Gerais ICR Implementation Completion Report IDEB Index of Basic Education (Índice de Desenvolvimento da Educação Básica) M&E Monitoring and Evaluation MG Minas Gerais ONA National Accreditation Organization (Organização Nacional de Acreditação) PAD Project Appraisal Document PAF Fiscal Adjustment Program (Programa de Ajuste Fiscal) PDO Program Development Objective PEP Vocational Education Program (Programa de Educação Profissional) PIP Program for Educational Intervention (Programa de Intervenção Pedagógica) PPAG Government Pluri-Annual Plan (Plano Plurianual de Ação Governamental) PPP Public-Private Partnerships PROALFA Literacy Cycle Evaluation Program (Programa de Avaliação de Ciclo de Alfabetização) PROEB Evaluation Program of Public Basic Education Network (Programa de Avaliação da Rede Pública de Educação Básica) PROHOSP Hospital Strengthening and Quality Improvement Control Project (Programa de Hospitais) ii SEE Secretariat of Education (Secretaria de Estado de Educação de Minas Gerais) SEPLAG Secretariat of Planning and Management (Secretaria de Estado de Planejamento e Gestão) SES State Secretariat of Health (Secretaria de Estado de Saúde) SETOP State Secretariat of Transport and Public (Secretaria de Transportes e Obras Públicas) SICA Student Informational System SWAp Sector-Wide Approach Vice President: Jorge Familiar Country Director: Deborah L. Wetzel Senior Practice Director: Mario Marcel Practice Manager: Arturo Herrera Gutierrez Project Team Leader: Roland N. Clarke ICR Team Leader: Fanny Weiner iii BRAZIL SECOND MINAS GERAIS DEVELOPMENT PARTNERSHIP PROJECT CONTENTS Document of The World Bank ............................................................................................ i Data Sheet .................................................................................................................. vi A. Basic Information.................................................................................................. vi B. Key Dates .............................................................................................................. vi C. Ratings Summary .................................................................................................. vi D. Sector and Theme Codes...................................................................................... vii E. Bank Staff ............................................................................................................. vii F. Results Framework Analysis ............................................................................... viii H. Restructuring ....................................................................................................... xvi I. Disbursement Graph ........................................................................................... xvii 1. Project Context, Development Objectives and Design ................................................... 1 1.1 Context at Appraisal ................................................................................................. 1 1.2 Original Project Development Objectives (PDO) and Key Indicators ..................... 2 1.3 Revised PDO and Key Indicators, and reasons/justification .................................... 3 1.4 Main Beneficiaries .................................................................................................... 3 1.5 Original Components ................................................................................................ 3 1.6 Revised Components ................................................................................................ 5 1.7 Other significant changes.......................................................................................... 5 2. Key Factors Affecting Implementation and Outcomes .................................................. 5 2.1 Project Preparation, Design and Quality at Entry ..................................................... 5 2.2 Implementation ......................................................................................................... 6 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization .......... 7 2.4 Safeguard and Fiduciary Compliance ....................................................................... 8 2.5 Post-completion Operation/Next Phase .................................................................... 9 3. Assessment of Outcomes .............................................................................................. 10 3.1 Relevance of Objectives, Design and Implementation ........................................... 10 3.2 Achievement of Project Development Objectives .................................................. 11 3.3 Efficiency ................................................................................................................ 15 3.4 Justification of Overall Outcome Rating ................................................................ 16 3.5 Overarching Themes, Other Outcomes and Impacts .............................................. 17 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops ....... 19 4. Assessment of Risk to Development Outcome ............................................................. 19 5. Assessment of Bank and Borrower Performance ......................................................... 19 5.1 Bank Performance................................................................................................... 19 5.2 Borrower Performance ............................................................................................ 21 iv 6. Lessons Learned............................................................................................................ 21 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners............... 24 Annex I. Project Costs and Financing............................................................................... 25 Annex II. Outputs by Component ..................................................................................... 26 Annex III. Economic and Financial Analysis ................................................................... 41 Annex IV. Bank Lending and Implementation Support/Supervision Processes .............. 44 Annex V. Beneficiary Survey Results .............................................................................. 46 Annex VI. Stakeholder Workshop Report and Results .................................................... 46 Annex VII. Summary of Borrower's ICR and/or Comments on Draft ICR ..................... 47 Annex VIII. Comments of Cofinanciers and Other Partners/Stakeholders ...................... 50 Annex IX. List of Supporting Documents ........................................................................ 51 Annex X: An Overview of Minas Gerais’ Management Reforms ................................... 52 v Data Sheet A. Basic Information Minas Gerais Country: Brazil Project Name: Partnership II SWAP IBRD-75470,IBRD- Project ID: P101324 L/C/TF Number(s): 78710 ICR Date: 01/26/2015 ICR Type: Core ICR THE STATE OF Lending Instrument: SIL Borrower: MINAS GERAIS, BRAZIL Original Total USD 976.00M Disbursed Amount: USD 1,434.88M Commitment: Revised Amount: USD 1,435.56M Environmental Category: B Implementing Agencies: State Secretariat of Planning and Management (Secretaria de Estado de Planejamento e Gestão – SEPLAG) Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 10/16/2007 Effectiveness: 09/23/2008 09/23/2008 Appraisal: 02/19/2008 Restructuring(s): 04/06/2010 Approval: 05/01/2008 Mid-term Review: Closing: 06/30/2011 10/31/2014 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Satisfactory Risk to Development Outcome: Moderate Bank Performance: Satisfactory Borrower Performance: Highly Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Satisfactory Government: Highly Satisfactory Implementing Quality of Supervision: Satisfactory Satisfactory Agency/Agencies: Overall Bank Overall Borrower Satisfactory Highly Satisfactory Performance: Performance: vi C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments (if Indicators Rating Performance any) Potential Problem Quality at Entry Project at any time No None (QEA): (Yes/No): Problem Project at any Quality of No None time (Yes/No): Supervision (QSA): DO rating before Highly Satisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) General education sector 15 15 General industry and trade sector 10 10 Health 15 15 Rural and Inter-Urban Roads and Highways 25 25 Sub-national government administration 35 35 Theme Code (as % of total Bank financing) Administrative and civil service reform 25 25 Managing for development results 24 24 Other economic management 13 13 Public expenditure, financial management and 25 25 procurement Regulation and competition policy 13 13 E. Bank Staff Positions At ICR At Approval Vice President: Jorge Familiar Pamela Cox Country Director: Deborah L. Wetzel John Briscoe Practice Arturo Herrera Gutierrez Lily L. Chu Manager/Manager: Deborah L. Wetzel Project Team Leader: Roland N. Clarke José Guilherme Reis ICR Team Leader: Fanny Weiner ICR Primary Author: Fanny Weiner Angela Nieves Marques Porto Till Johannes Hartmann Laura Zoratto and William Dillinger (Annex X on Results- based management reforms) vii F. Results Framework Analysis Project Development Objectives The original Project Development Objective was to (i) help the Borrower improve the efficiency and effectiveness of public resource use and allocation for economic and social development; (ii) support the adoption of innovations in public management by the Borrower; and (iii) support the Borrower in strengthening its results-based management system of monitoring and evaluation of results. Revised Project Development Objectives (as approved by original approving authority) N/A (a) PDO Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Implementation of 1st stage Results Agreements (number of Secretariats with Indicator 1 : verified results-agreements performance above 70%) Value quantitative or 4 7 8 Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2010 Comments (incl. % Achieved. achievement) Implementation of 2nd stage Results Agreements (number of agreements Indicator 2 : evaluated in due time) Value quantitative or 10 11 11 Qualitative) Date achieved 12/31/2009 12/31/2011 12/31/2010 Comments (incl. % Achieved. achievement) Indicator of MG State Restructuring and Fiscal Adjustment (PAF) agreed with Indicator 3 : the STN (Ratio of personnel expenditures with respect to net current revenue mentioned in the current PAF) Value quantitative or 59.43% 60% 63.1% Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2011 Comments (incl. % Not achieved. achievement) Indicator 4 : Participation of Strategic Projects in Budget Expenditures Value 6.95% 9.50% 9.85% viii quantitative or Qualitative) Date achieved 12/31/2012 12/31/2010 12/31/2009 Comments (incl. % Achieved. achievement) Indicator 5 : Number of PPPs contracted Value quantitative or 1 3 4 Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2010 Comments (incl. % Achieved. achievement) Percentage of hospitals of PROHOSP program formally assessed for Indicator 6 : accreditation by an institution certified by the National Accreditation Organization (ONA) Value quantitative or 0% 75% 72.66% Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2011 Comments (incl. % Partially achieved. achievement) Indicator 7 : Implementation of a standardized costing system in FHEMIG hospitals Cost system Cost system Value module developed module developed quantitative or - and piloted in 2 and piloted in 22 Qualitative) hospitals hospitals Date achieved 12/31/2007 12/31/2011 12/31/2011 Comments (incl. % Achieved. achievement) Indicator 8 : Cost-Student Informational System (SICA) Regular reports Twelve reports Value and analysis about the monthly System with information quantitative or produced on evolution of the updated once a year Qualitative) evolution of costs cost per student per student produced Date achieved 12/31/2007 12/31/2011 12/31/2011 Comments (incl. % Achieved. achievement) Percentage of the state highway grid paved with contracts of maintenance and Indicator 9 : renovation for results Value quantitative or 11% 42% 36.7% Qualitative) ix Date achieved 12/31/2007 12/31/2011 12/31/2011 Comments (incl. % Partially achieved. achievement) (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Indicator 1 : Certification of Public Sector Positions (Number of certified job categories) Value (quantitative 2 6 or Qualitative) Date achieved 12/31/2007 12/31/2009 Comments (incl. % This indicator was altered at the time of the Additional Financing. achievement) Indicator 2 : Certification of Public Sector Positions (Compliance with selected actions) recertification of recertification of civil servants for civil servants for Value specific positions; specific positions; (quantitative - ii) training for ii) training for or Qualitative) specific certified specific certified civil servants civil servants Date achieved 12/31/2009 12/31/2010 12/31/2010 Comments (incl. % Achieved. achievement) Indicators of MG State Restructuring and Fiscal Adjustment (PAF) agreed with Indicator 3 : the STN (primary surplus) Value (quantitative R$ 1,704 million R$ 1,406 million R$ 2,005 million or Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2011 Comments (incl. % Achieved. achievement) Indicator 4 : Gross Operating balance Value (quantitative 11% 14% 14.95% or Qualitative) Date achieved 12/31/2009 12/31/2011 12/31/2010 Comments (incl. % Achieved. achievement) Indicator 5 : Management Systems for Procurement x System of Electronic management of invoicing system Value No system of electronic civil construction developed and (quantitative invoicing, management of contracts, implemented, or Qualitative) civil contracts fragmented integrated with integrated with SIAD and SIAFI, SIAD and SIAFI in use. Date achieved 12/31/2007 12/31/2011 12/31/2010 Comments (incl. % Achieved. achievement) Indicator 6 : Improvement of results management Value Final reports of 3 Final reports of 3 (quantitative - impact evaluations impact evaluations or Qualitative) concluded concluded Date achieved 12/31/2007 12/31/2010 12/31/2010 Comments (incl. % Achieved. achievement) Indicator 7 : Strengthening of Evidence-based policy-making Executive Executive evaluation of evaluation of indicators is done; indicators is done; a Strategic projects under a pilot of pilot of executive Value implementation and executive evaluation is (quantitative ongoing monitoring of evaluation is completed and a or Qualitative) output and outcome completed and a document for indicators. document for transition about transition about M&E agenda is M&E agenda is prepared. prepared. Date achieved 12/31/2009 12/31/2010 12/31/2010 Comments (incl. % Achieved. achievement) Indicator 8 : Time to start a business at Minas Fácil in Belo Horizonte Value (quantitative 26 days 10 days 9 days or Qualitative) Date achieved 12/31/2007 12/31/2010 12/31/2009 Comments (incl. % Achieved. achievement) Implementation of Minas Fácil in the entire State (number of Minas Fácil Indicator 9 : physical units) Value (quantitative 21 29 29 or Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2009 xi Comments (incl. % Achieved. achievement) Indicator 10 : Technology innovation centers Value (quantitative 4 15 15 or Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2011 Comments (incl. % Achieved. achievement) Indicator 11 : Impact Evaluation of Viva Vida Centers Methodology for using DATASUS panel data with Value monitoring (quantitative - indicators for or Qualitative) continued impact evaluation process development tested Date achieved 12/31/2007 12/31/2011 Comments (incl. % This indicator was excluded from the project in 2011. achievement) Percentage of 8 year old students registered in state schools with expected Indicator 12 : reading level in the 3rd year of elementary school Value (quantitative 65% 78-80% 87.10% or Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2010 Comments (incl. % Achieved. achievement) Indicator 13 : System of evaluation and results feedback PROALFA and PROEB conducted at classes of 3rd, Value 5th, 9th grade of (quantitative - elementary school or Qualitative) and 3rd grade of high school. Plan of action in 70% of the schools. Date achieved 12/31/2007 12/31/2009 Comments (incl. % This indicator was substituted at the time of the Additional Financing. achievement) xii (i) Number of New Test Items; (ii) Participation rate (Percentage of schools that Indicator 14 : apply the test for the 9th grade classrooms in five subjects - mathematics, Portuguese, physics, chemistry and biology) Value i) 35,000; ii) 1,000 i) 24,633; ii) 500 State (quantitative State secondary i) 36,036; ii) 2049 secondary schools or Qualitative) schools Date achieved 12/31/2008 12/31/2011 12/31/2010 Comments (incl. % Achieved. achievement) Indicator 15 : Students registered at courses in the secondary vocational schools Value (quantitative 0 25,000 or Qualitative) Date achieved 12/31/2007 12/31/2009 Comments (incl. % This indicator was substituted at the time of the Additional Financing. achievement) Indicator 16 : Enrollment in PEP courses (secondary vocational school courses) Value (quantitative - 30,000 42,626 or Qualitative) Date achieved 12/31/2009 12/31/2011 12/31/2009 Comments (incl. % Achieved. achievement) Indicator 17 : Percentage of municipalities in the State with paved access Value (quantitative 83% 96% 96.8% or Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2011 Comments (incl. % Achieved. achievement) Indicator 18 : Development of new highway PPPs Value PMIs for 3 (quantitative - highway lots or Qualitative) published Date achieved 12/31/2007 12/31/2009 Comments (incl. % This indicator was substituted at the time of Additional Financing. achievement) Indicator 19 : Number of electronic speed control equipment in operation Value (quantitative 16 80 125 or Qualitative) Date achieved 12/31/2009 12/31/2011 12/31/2010 Comments Achieved. xiii (incl. % achievement) Average percentage delay of Pro-Acesso paving works concluded in the current Indicator 20 : year Value (quantitative 37.9% 25% 23.6% or Qualitative) Date achieved 12/31/2007 12/31/2010 12/31/2010 Comments (incl. % Achieved. achievement) Average time for the concession of environmental licenses for activities with Indicator 21 : environmental impact in the state, for classes 3 and 4 Value (quantitative 180 days 90 days 86 days or Qualitative) Date achieved 12/31/2006 12/31/2011 12/31/2009 Comments (incl. % Achieved. achievement) Indicator 22 : Number of inter-disciplinary teams of indigenous health funded by SES Value (quantitative 12 16 16 or Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2008 Comments (incl. % Achieved. achievement) Indicator 23 : Number of certified teachers in the Curso de Magistério Indígena Value (quantitative 137 212 213 or Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2008 Comments (incl. % Achieved. achievement) Number of families benefited by the Rural Poverty Reduction project starting in Indicator 24 : 2010 (per year) Value (quantitative 0 16,800 21,900 or Qualitative) Date achieved 12/31/2009 12/31/2011 12/31/2010 Comments (incl. % Achieved. achievement) Intensive monitoring of Rural Poverty Reduction Project (number of monitoring Indicator 25 : meetings) Value 0 6 6 xiv (quantitative or Qualitative) Date achieved 12/31/2009 12/31/2011 12/31/2010 Comments (incl. % Achieved. achievement) Indicator 26 : Improvement of data on poverty in Minas Gerais Analytical report of Value Field work of PAD-MG poverty data (quantitative 2009 completed on the "Great or Qualitative) North" completed Date achieved 12/31/2009 12/31/2011 Comments This indicator was discarded by agreement between the World Bank and the (incl. % Government of Minas Gerais. achievement) Number of secondary schools in the Great North that offer initial work Indicator 27 : preparation courses for the assembling and maintenance of computers Value (quantitative 59 90 90 or Qualitative) Date achieved 12/31/2009 12/31/2010 12/31/2010 Comments (incl. % Achieved. achievement) Proportion of live births from mothers with 7 or more pre-natal consultations in Indicator 28 : the Great North of Minas Gerais Value (quantitative 46.1% 48% 57.8% or Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2010 Comments (incl. % Achieved. achievement) G. Ratings of Project Performance in ISRs Date ISR Actual Disbursements No. DO IP Archived (USD millions) 1 06/06/2008 Satisfactory Satisfactory 0.00 2 12/15/2008 Satisfactory Satisfactory 197.00 3 05/14/2009 Satisfactory Satisfactory 347.50 4 08/26/2009 Satisfactory Satisfactory 503.50 5 05/05/2010 Satisfactory Satisfactory 739.14 6 01/26/2011 Satisfactory Satisfactory 1329.71 7 08/13/2011 Satisfactory Satisfactory 1330.15 xv 8 03/28/2012 Satisfactory Satisfactory 1422.48 9 11/14/2012 Satisfactory Satisfactory 1426.43 10 04/08/2013 Satisfactory Satisfactory 1427.15 11 11/02/2013 Satisfactory Satisfactory 1430.15 12 12/15/2013 Highly Satisfactory Highly Satisfactory 1430.28 13 06/27/2014 Highly Satisfactory Satisfactory 1430.28 H. Restructuring ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Approved Reason for Restructuring & Key Restructuring Date(s) PDO Changes Made DO IP in USD Change millions To address a financing gap arising from 04/06/2010 N S S 738.16 unanticipated circumstances, and to scale up project activities Some delays in procurement processes required an extension of the closing date 03/28/2012 S S 1422.48 to allow for the full disbursement of the technical assistance component. This extension was approved to allow for the implementation of two consultancies (a feasibility study on a 03/25/2013 S S 1427.15 train network and an Economic Master Plan for the Belo Horizonte metropolitan area) in the technical assistance component. The project closing date was extended until July 31, 2014 to be able to use remaining funds under the TA 01/06/2014 HS HS 1430.28 component to extend the scope of the Economic Master Plan consultancy by including 14 additional municipalities in the study. The project's closing date was extended due to delays in approving the 07/24/2014 HS S 1430.60 procurement of the last contract (Economic Master Plan consultancy). xvi I. Disbursement Graph xvii Introduction Bank engagement with Minas Gerais The Bank has supported the reform program of the State Government of Minas Gerais (GoMG) to introduce and mainstream result-based management systems across its most important public management functions, since 2006 through the “Minas Gerais Partnership Series (MGPS)” of operations. The MGPS was composed of three successive Bank operations, which sought to help the GoMG expand and deepen its public sector management reform program. Minas Gerais Partnership for Minas Gerais Partnership II Third Minas Gerais Development DPL (2006) SWAp (2008) + Additional Development Partnership DPL Financing (2010) (2012) • Choque de Gestão - • Estado para Resultados - State • Estado em Rede - Management Shock (2003- for Results (2007-2010) - Management for 2006) - stressing fiscal stressing efficiency and results Citizenship (2011-2014) - balance stressing citizen participation The GoMG’s reform program started in 2003 with what is known as Choque de Gestão – Management Shock. The Choque de Gestão was based on an initial fiscal adjustment, and a series of management and human resource reforms to allow the implementation of a series of Government priority programs (e.g. in the health, transport and education sectors) and the close monitoring of their related performance indicators. The targets were negotiated with sector secretariats that became accountable for delivering program results. The Bank’s support started in 2003 through technical assistance and was operationalized in 2006 through the Minas Gerais Partnership Development DPL (P088543) of US$170 million. That engagement was followed by the Minas Gerais Partnership II SWAp – object of this ICR – to support the reform program which had entered into a second phase - the Estado para Resultados (State for Results). In this phase, indicators were negotiated at the departmental level, with monetary rewards linked to the achievement of goals. Subsequently the Government strategy evolved to a focus on citizen participation, called Estado em Rede (Management for Citizenship) with the aim of consolidating the previous phases, increasing efficiency and building a permanent constituency through citizen participation while ensuring the model’s sustainability, through devolution of performance management to individual secretariats. This third phase of the State’s reform program was supported by the Third Minas Gerais Partnership for Development DPL in 2012 (P121590) of US$450 million. The evolving series of reforms undertaken by the State grew to impact every facet of its economic and administrative management. The reforms undertaken since 2006 supported the State’s economic recovery as well as improving service delivery. In effect, it transformed Minas from a bankrupt and poorly managed under-performer to a fiscally responsible state, which set up and fine-tuned a results-based management system of monitoring and evaluating results. This improvement has led to an increase in efficiency and effectiveness of public resource use in public institutions. This comprehensive management reform was one of the first of its kind in Brazil and is regularly mentioned as a benchmark for other (state) governments in Brazil and elsewhere. xviii Preparation and implementation of the SWAp The Government asked the Bank to go beyond its traditional investment lending instruments in providing support for the reforms. Although Minas was in a position to borrow from other sources, it chose the Bank as its preferred partner due to its capacity to focus on cross-cutting issues in an integrated multi-sectorial program, for its knowledge and for its technical skills. The support requested involved not only investments, but also needed to link disbursements to the achievements of indicators, specialized technical assistance and day-to-day advice on a variety of issues. The operation included two components covering Public Sector, Private Sector Development, Health, Education, and Transport. The first component (SWAp), of US$1,413 million (including an additional financing) was disbursed to the State Treasury against expenditures incurred under Eligible Expenditure Programs (EEPs) –specific Government priority programs included in the Government Pluri-Annual Plan (PPAG).1 Disbursement, which occurred at three to six month intervals, was also dependent on the achievement of Disbursement-Linked Indicators (DLIs) measuring the progress of the EEPs. A Technical Assistance (TA) Component in the amount of US$18.5 million supported the implementation of EEPs and the achievement of the DLIs through consultancies, and disbursed as a traditional investment lending operation. Results and achievements This was a highly successful project whose overall impact and success cannot be captured solely by assessing the achievement of the PDO indicators. The project paved the path for significant reforms and a culture change in the public administration of Minas Gerais. As one of the first successful models of result-based management in Brazil, it had spill-over effects to other states and countries, which adapted the system. Certain sector policies, such as in education, influenced national policy-making taking Minas as a successful example. The operation also served as incubator for innovative operational approaches. The SWAp modality as a lending instrument heavily influenced the discussion for the need to adjust Bank instruments to the needs of clients, especially in middle-income countries, which eventually led to the introduction of the PforR instrument. Another element to be taken into consideration when assessing the overall outcome of the project is its innovative nature at the time of preparation, as well as its size, both in multi-sector coverage and lending amount. The project served as a pioneer in many of those aspects and valuable lessons were learned from its success, but also from its shortcomings. Those lessons served to further develop the SWAp instrument and have been integrated in similar operations elsewhere. 1 Examples of EEPs include Government programs for “Professionalization of Public Managers”, “Improvement of Quality and Efficiency of Secondary Education”, “Regionalization of Health Care”, and “Pro-Acesso Municipality Road Surfacing Program”. The complete list of EEPs can be found in Annex II. xix 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. Minas Gerais can be considered a microcosm of Brazil. Its income level is close to Brazil’s national average (ranked 10th out of 27 States in 2012), and the State has deep regional disparities mirroring those at the national level. For instance, the State’s northern region has conditions of poverty, subsistence agriculture and semi-arid climate similar to the Brazilian Northeast, the poorest region in the country. Minas Gerais also has Brazil’s second largest population (19.9 million, representing 10.2 percent of Brazil’s total) and the economy is the third largest after São Paulo and Rio de Janeiro, and about the size of Peru. 2. Back in 2002, Minas Gerais’ fiscal situation was one of the most difficult in Brazil, and it was one of seven States (out of 27) that had been unable to meet the Fiscal Responsibility Law (FRL) targets. The incoming administration in 2003 thus inherited a bankrupt state with large levels of debt, poor management and low investment. Consequently, the Government launched a results-based management reform program, “Choque de Gestão” (Management Shock), which was based on programs that were of priority to the Government (e.g. in the health, transport and education sectors) and performance indicators to manage them. As the system evolved these targets were negotiated with sector secretariats that became accountable for delivering program results 2. 3. At the time of appraisal of the SWAp in early 2008, the administration had been overwhelmingly re-elected to a second term, ensuring continuity of its reform process. From 2004 to 2008 Minas Gerais had consolidated the fiscal adjustment efforts initiated in 2003, significantly improving revenue and operating balances, increasing investments and reducing debt. By 2006 the State had improved its fiscal situation sufficiently that the Federal Government allowed it to borrow again. 4. The reforms had also led to improvements in poverty and social indicators. The poverty headcount ratio in Minas had fallen from 25 percent of the population in 2002 to 15.4 percent in 2006, implying a poverty headcount of about 3.9 million, representing the third lowest headcount in Brazil and a relatively high per capita family income, although issues of inequality remained. 5. By 2008, reforms building on the “Choque de Gestão” entered into a second phase - the “Estado para Resultados” (State for Results). Implementation of this subsequent phase was based on the signing of Results Agreements between the Governor and the Secretariats associated with the identified “results areas”. Each Secretary committed outcome, output, process and input indicators, and in return the Governor committed to giving the Secretariats sufficient resources to implement the strategic projects and 2 This comprehensive management reform was one of the first of its kind in Brazil and is regularly mentioned as a benchmark for other State governments. Because of its relevance, this ICR contains a separate analysis of these reforms, summarized in Annex X. This annex also presents the results of a survey with approximately 2,000 government employees regarding their perception of the management reforms of this period. 1 management autonomy over how to achieve results. See Annex X for more details on the reform process in Minas Gerais, from 2003 until 2014. Rationale for Bank Involvement 6. The Bank’s engagement with the State of Minas Gerais had begun in the early 2000s through analytical and knowledge activities. This continuous dialogue prompted the State to partner with the Bank for the implementation of its reform agenda. The first loan under the Minas Gerais Development Partnership Series was a two-tranche DPL of US$170 million (P088543), in 2006, supporting the “Choque de Gestão”. As Minas Gerais moved into the second phase of its reform process, it asked for additional Bank support for the implementation of the State for Results Program. Although Minas was in a position to borrow from other sources, it chose the Bank as its preferred partner due to its capacity to focus on cross-cutting issues in an integrated multi- sectorial program, for its knowledge and for its technical skills. In its request for the new operation, GoMG asked the Bank to go beyond its traditional investment instruments, because it needed a simultaneous combination of investments, results-based disbursements and management support, specialized technical assistance, and day-to-day advice. This operation presented an opportunity for the Bank to increase its role at the state level in Brazil and address complex development challenges in an integrated and multi-sector fashion, in line with the Brazil Country Partnership Strategy (CPS) for 2008-2011 3. 1.2 Original Project Development Objectives (PDO) and Key Indicators 7. The original Project Development Objective was to (i) help the Borrower improve the efficiency and effectiveness of public resource use and allocation for economic and social development; (ii) support the adoption of innovations in public management by the Borrower; and (iii) support the Borrower in strengthening its results-based management system of monitoring and evaluation of results. Table 1: PDO Indicators PDO Key indicators Help the Borrower • Ratio of personnel expenditures with respect to net current revenue improve the efficiency and mentioned in the current PAF. effectiveness of public • Existence of cost data in hospitals with time series that indicate resource use and allocation trends and costs at organizational level (hospital level) and at the for economic and social departmental level (e.g., surgery, maternity, etc.) development • Cost per Student Informational System (SICA) implemented and cost data posted on the web • Average value of cost addenda in contracts for road work as percentage of initial value of works (for contracts included in a year) to decrease • Investment expenditure as a share of total expenditure • Expenditure of strategic projects as a share of total expenditure Support the adoption of • Successful implementation of results agreements innovations in public • Number of PPPs contracted by the state management • Percentage of hospitals of PROHOSP program assessed by the ONA 3 World Bank (2008): Country Partnership Strategy Brazil 2008-2011. Report N. 42677-BR. 2 • Percentage of state-pave road network under results-based maintenance and rehabilitation contracts Support the Borrower in • Satisfaction of vice-governor’s office and secretariats with available strengthening its results- M&E information based management system • Strengthening of results/causal chains underlying the government’s of monitoring and program evaluation of results 1.3 Revised PDO and Key Indicators, and reasons/justification 8. An Additional Financing 4 (AF) of US$461 million was requested by the GoMG in 2010 in order to withstand the unanticipated deep global and domestic recession and fiscal challenges resulting from significant drops in state and federal revenues. The original project was restructured to include support to the Government’s rural poverty reduction program, trigger additional safeguard policies, change the financing percentage, and introduce new and modify existing intermediate indicators. The Project Development Objective and its indicators remained the same, although some targets were revised. 1.4 Main Beneficiaries 9. The gains from improved public sector efficiency were shared by the majority of the population and particularly the poor population. More specifically, the transport component improved the connectivity of mostly poor municipalities by providing them with paved access to the city. The education component focused on public schools, which generally have a higher proportion of students from poor backgrounds. Similar to the public education sector, through self-selection, the public health system and centers treat those who cannot afford more expensive private care. 1.5 Original Components 10. This Sector-Wide Approach (SWAp) operation included two components covering Public Sector, Private Sector Development, Health, Education, and Transport. The first component (SWAp), of US$957.5 million, disbursed to the State Treasury against expenditures incurred under Eligible Expenditure Programs (EEPs). EEPs are Government programs included as specific lines in the Government Pluri-Annual Plan (PPAG) and in the annual budget law. 5 Disbursement, which occurred at three to six month intervals, was also dependent on the achievement of 24 Disbursement-Linked Indicators (DLIs) measuring the progress of the EEPs. A Technical Assistance component in the amount of US$18.5 million supported the implementation of EEPs and the achievement of the DLIs. The TA component disbursed as a traditional investment lending operation against statements of expenditures (SOEs). 11. The project provided a package of financial, policy, and technical support in the priority sectors as well as support on key cross-cutting and multi-sectorial issues. EEPs 4 Additional Financing for the Second Minas Gerais Partnership Development SWAp, P119215, Loan #78710, approved by the Board of Directors on April 6, 2010. 5 Examples of EEPs include Government programs for “Professionalization of Public Managers”, “Improvement of Quality and Efficiency of Secondary Education”, “Regionalization of Health Care”, and “Pro-Acesso Municipality Road Surfacing Program”. The complete list of EEPs can be found in Annex II. 3 funded the sector objectives and institutional reforms. Some, such as the public sector program and the private sector program, did not involve large expenditures but had important implications for the other programs in terms of cross-cutting institutional and policy reforms and contributed to the PDO’s cross-cutting focus on public sector efficiency, innovation and implementation of the results-based management model. The education and health programs were relatively large, but expenditure focused more on recurrent expenditures and needs. Finally, the transport program was among the largest and funded considerable public works, but also promoted institutional reforms. These sector programs were linked to the PDO as they contributed to insert the goals of efficiency and effectiveness of public resources, innovative approaches and finally, the implementation of results-based management on a sectorial level. 12. Public Sector Management (PSM). As reflected in the PDO, an overriding goal of the Government was to extend the culture of results management across all line secretariats, thus PSM was ranked among the most important results areas, addressing four cross-cutting challenges: (i) implementation of the results agreements; (ii) professionalization of public managers; (iii) continued improvement in budget and financial management; and (iv) continued strengthening of the procurement system. 13. Private Sector Development. Contributing to the PDO by supporting innovation, this area focused on the five main objectives of the Minas Gerais Integrated Development Plan 2007-2023, which were to increase dynamism and innovativeness to promote economic growth and sustainable social transformation. An improved investment climate was seen as the best way to establish the conditions for long-term growth and continuous job creation. Activities focused on fostering innovation and adopting technology, facilitating the creation of new firms, and strengthening the State’s leading role in implementing Public Private Partnerships (PPPs). 14. Health Sector. The Minas Gerais health sector faced challenges such as (i) unequal access to basic and specialized care; (ii) low quality and insufficient coverage; and (iii) little coordination among health care providers among different levels of care with implications for resource allocation and management, quality of spending, and ultimately health outcomes. Those challenges were addressed by improving the quality of hospitals and family health care, and strengthening the efficiency of hospital management through, inter alia, hospital accreditation and the implementation of standardized cost systems. The contribution to the PDO was a more efficient use of resources in the health sector and the implementation of innovative approaches, such as the PROHOSP (Hospital Strengthening and Quality Improvement Control Project), a State program which uses targets and performance indicators to evaluate hospitals. 15. Education Sector. The two most pressing challenges in the education sector were insufficient information on student learning achievement and low early primary school literacy levels which highlighted the need to improve the quality of education in the State. This was addressed by strengthening the management of elementary education services, improving secondary vocational education programs, and supporting personnel expenditures of the State Secretariat of Education, while contributing to the PDO through more efficient use of public resources. 4 16. Transport Sector. The State’s long-term development plan emphasized reducing transport costs and expanding access to markets through a safe road network while overcoming transport bottlenecks to better integrate the competitive economy and help develop underperforming regions. Increased efficiency and improved quality of spending was to be achieved through a results-based approach to management in the transport sector, and private sector participation in the provision of transport infrastructure through PPPs and other mechanisms. Furthermore, State programs for highway maintenance and rehabilitation, and road surfacing were supported to ensure paved access to every municipality. 17. In addition to the five priority sectors, the operation included indicators for the monitoring of social and environmental aspects. While not linked to a specific sector EEP, two DLIs monitored the number of indigenous health teams and certified indigenous teachers. These fit into both the education and health areas and also addressed social safeguards and inclusion concerns. Another DLI linked PSM efficiency and environmental management as it tracked the average time to obtain certain environmental licenses. 18. The Technical Assistance Component supported activities in each of the five priority sectors. The activities under this component were carefully and specifically selected with the objective of supporting and complementing the fulfillment of DLIs in all areas; some TA activities explicitly supported the achievement of DLIs, while others were for more general institutional strengthening, including safeguards. 1.6 Revised Components 19. The original components were maintained when the Minas SWAp was scaled- up through the Additional Financing. Rural Poverty was added as a new sector to provide additional direct support to the Government’s poverty reduction strategy by means of supporting the application of the Government’s results-based management approach to programs aimed at poverty reduction. The Rural Poverty Reduction Program was included as an EEP and linked to five new DLIs to measure rural poverty reduction. 1.7 Other significant changes 20. During its lifetime the project was extended a total of five times. The first extension occurred with the AF to allow for disbursement and TA implementation of the scaled-up project. Although funds under the SWAp component were fully disbursed by 2011, four subsequent extensions were approved to allow for the use of remaining TA funds. 6 This enabled the Government to commission a series of consultancies and studies that informed further development of the Government agenda. One of the most important is the Economic Master Plan for the Metropolitan Region of Belo Horizonte that lays out an integrated vision on the economic and social development of the state capital. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 6 1st extension to 3/2012 (9 months); 2nd extension to 3/2013 (12 months); 3rd extension to 12/2013 (9 months); 4th extension to 7/2014 (7 months); 5th extension to 10/2014 (3 months). 5 21. Project preparation was fully in line with the Country Partnership Strategy 2008 - 2011, which highlighted the Bank’s increasing role at the State level in Brazil. The operation was the second sub-national SWAp in Brazil, preceded only by the Ceará Multi-Sector Social Inclusion Project (P082142). 22. The GoMG asked the Bank not only to help with specific aspects of sector policy, but also to support the implementation and strengthening of its ability to handle cross- cutting issues. The selection of the SWAp modality as opposed to a DPL allowed the Bank to respond to this request in a unique manner, through support for their priority expenditure programs, technical assistance, and on-going Bank support and guidance. The multi-sector design of a SWAp facilitated the collaboration between secretariats during the reform process. Furthermore, the SWAp modality was particularly suited to this phase of the State’s reform agenda given the mutual focus on multi-sector performance. 23. The preparation of the operation was complex due to the number of sectors and agencies involved, the size of the loan, and the fact that the SWAp modality was still a novelty. During Project preparation Bank specialists were present in Minas Gerais almost on a weekly basis to work closely with the counterparts. The importance of this close engagement has since been stressed by both the Bank and the Minas Gerais teams. 24. However, the design of the project had some shortcomings. While the multi- sector nature of the project made it challenging to establish and measure an overarching outcome, the PDO intended to overcome this difficulty by comprising three sub-PDOs, which were somewhat intermediate objectives. Some of the PDO level indicators measured outputs rather than outcomes and the number of indicators measuring sector achievements were not reflecting the relative share of the sectorial EEPs, leading to a slight disconnect between the scope of project activities and PDO. 25. During preparation a review of national procurement legislation was carried out. One of the challenges was ensuring compatibility of the Bank procurement requirements with State and Federal legislation. This included finding an acceptable manner to embed antifraud and corruption clauses in the procurement documents, and to prepare specific reports on eligible expenditures under the SWAp component. As one of the first SWAp operations in Brazil, the review helped to define certain standards for subsequent operations. 26. Although the specific risks of the project were seen to be low, the overall risk at the time was rated ‘moderate’ due to the size and visibility of the operation. Most risks were correctly identified and adequate mitigation measures put in place. 2.2 Implementation 27. Overall implementation was ‘satisfactory’ throughout project cycle. The loan was signed on August 8, 2008 and became effective on September 23 of the same year. ISR ratings for overall implementation progress of the project remained as ‘satisfactory’ throughout the operation. Some delays were encountered in the initial phases of implementation, due mainly to the lack of familiarity of both the Bank and the Borrower with each other’s procedures and mechanisms related to procurement and financial management, especially in the context of a SWAp. 6 28. Project restructuring at mid-term. No mid-term review (MTR) was carried out as the restructuring (accompanying the AF) took place at around the time when a mid-term review would have been undertaken, and entailed a far more in depth review of the project than an MTR. In addition, at the time MTRs were not mandatory. The AF modified some of the indicators to delete or adjust those that had already been achieved, or to add new indicators to better reflect progress towards the PDO. 29. Strong Borrower performance coping with - staff turnover and coordination challenges. The continuous and strong commitment of the GoMG throughout the project was of great importance to the successful implementation. While there was staff turnover due to rotations and a change of Governor, the high quality of the technical staff involved in the project on the Borrower’s side was instrumental and ensured follow-through. The number of sectors and agencies involved in the operation was initially seen as a potential risk but was quickly overcome and led to better coordination among Government entities. That trend has continued and can be considered as a direct result of the project. 30. Technical Assistance was critical. While particular difficulties regarding Bank procurement procedures arose early on in the implementation process, it is important to note the impact of the Technical Assistance component of the loan and the fundamental role that it played in generating innovation within the Government. Most of the TAs were highly successful, because the government team had precise expectations from the consultants and saw an opportunity to undertake activities that the Government normally would not have been able to set resources aside for. The successful TAs were in general those with a clear link to the operation and close oversight by the implementing agency. 31. Additional Financing triggered by global financial crisis. Although identified in the original PAD as moderate, one of the potential risks of the operation, “a major significant world-wide shock”, did materialize in 2009. Following the global financial crisis the state suffered from plummeting commodity prices and less demand for their agricultural products and the GDP of Minas contracted more than the national GDP. The AF supported the original project’s activities which might have otherwise been postponed and possibly not fully implemented. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 32. The Project’s M&E framework is considered ‘satisfactory’. Strengthening the system of monitoring and evaluation of results in Minas Gerais was part of the PDO, therefore integral not only to the operation itself but also to the Government program. 33. Working together with Bank specialists, the GoMG developed an extensive framework to monitor the implementation and outcomes of its programs at many levels. The existence of this framework ensured the continuous and thorough monitoring of indicators and helped Government agencies create a new organizational culture focused on achieving results. Some indicators were only measured until 2011, when the SWAp component was concluded. However, most of the indicators continued to be monitored by the State, given their importance to the Government program. 34. However, the Bank’s M&E system showed some inconsistencies. For instance, the Project Appraisal Document (PAD) classifies some indicators as both PDO and Intermediate, causing discrepancies in the ISRs, where they are associated with only one indicator category. 7 Further complicating the monitoring and evaluation process, a number of indicators appear in the ISRs under different names as those in the original project documents, and some were included in the ISRs only in later years. 35. Changes in the Bank’s operational procedures. Since project approval in 2008, the Bank’s operational procedures underwent a number of changes, and the design and use of indicators became more sophisticated and integral to reporting on development effectiveness. The ISR format underwent two significant sets of changes, which affected the ability of the Bank team to consistently and coherently monitor the indicators. 2.4 Safeguard and Fiduciary Compliance Safeguard Compliance 36. The operation was a Category “B” with six safeguard policies triggered. While the original project triggered four safeguard policies, the project’s Safeguard Frameworks (i.e. Environmental Management Framework, Resettlement Policy Framework and Indigenous Peoples Planning Framework) were revised at the time of the Additional Financing, which triggered two additional safeguards. Safeguard Policies Triggered by the Project Yes No Environmental Assessment (OP/BP 4.01) [X] [] Natural Habitats (OP/BP 4.04)* [X] [] Pest Management (OP/BP 4.09)* [X] [] Physical Cultural Resources (OP/BP 4.11) [X] [] Involuntary Resettlement (OP/BP 4.12) [X] [] Indigenous Peoples (OP/BP 4.10) [X] [] Forests (OP/BP 4.36) [] [X] Safety of Dams (OP/BP 4.37) [] [X] Projects in Disputed Areas (OP/BP 7.60) [] [X] Projects on International Waterways (OP/BP 7.50) [] [X] * Safeguard policies triggered with the Additional Financing 37. Overall, and as expected, the project did not have negative social impacts. The frameworks containing the set of policy principles to address the two social Operational Policies, Indigenous Peoples (OP/BP 4.10) and Involuntary Resettlement (OP/BP 4.12) requirements that were triggered for both phases of the project were properly assessed, and overall well prepared. These frameworks were also based on the Brazilian legislation. Monitoring reports have confirmed the general appropriateness of this strategy and ISRs throughout project life consistently rated safeguard compliance as ‘satisfactory’. 38. However, implementation of the Social Safeguards had some shortcomings, although the design of the instruments was in general adequate. Guidance provided by the Resettlement Policy Framework (RPF) in general ensured compliance of the policy until the stage of compensations to affected people. However, at the time of the preparation of the ICR, about 18 percent (or 500 of 2,776 processes) of the compensation processes were not yet concluded and were in an administrative completion stage. The Government has elaborated an action plan to finalize these processes. The Bank will continue to closely monitor the implementation of that action plan until the pending compensation processes are fully and satisfactorily completed. 8 39. The Environmental Management Framework ensured compliance with environmental safeguards. The project did not have any environmental safeguard compliance issues and implementation was consistent as Category “B” as the impact was limited in scope, localized, temporary and reversible. Having followed the environmental safeguards agreed with the World Bank, the activities supported by the project are not expected to cause any significant negative externalities. Fiduciary Compliance 40. Financial Management arrangements were considered ‘Moderately Satisfactory’ throughout project implementation, and required close support by the Bank team. Because the use of country systems was prioritized, the adaption of the existing systems to specific Bank requirements was somewhat challenging. Initial weaknesses in the reporting system, the design of IFRs, and the use of spreadsheets to prepare financial statements led to data inconsistences, which were resolved by close Bank support during implementation. 41. However, the State was diligent in proposing alternatives and implementing action plans to mitigate the identified issues. The lack of familiarity with the Bank’s auditing arrangements during project preparation was overcome during implementation. The Tribunal de Contas do Estado de Minas Gerais (TCE-MG) was an important partner in strengthening executing agencies’ capacity and producing high quality auditing reports. 42. In fact, the State’s financial management capacity was significantly improved during the project. While the fiduciary overall capacity at the beginning was considered rather weak, as a result of respective mitigation measures in the project’s fiduciary arrangements, the counterpart improved monitoring, communication among state sectors and TCE, and increased implementation awareness in sectors. The concept of a designated account was adopted by the government for other operations and the adoption of International Public Sector Accounting Standards (IPSAS) had a positive impact on the project. 43. Procurement arrangements were considered ‘Satisfactory’ throughout implementation. Despite initial challenges in applying the Bank’s procurement policy, procurement throughout the project had no major issues and the Bank’s procurement rules were complied with. The application of the Bank’s guidelines in the selection of consultants had a positive impact on the quality of the service providers contracted. 44. Complex disbursement rules. The complex reporting rules on the compliance with the eligible expenditure programs also posed challenges during implementation and sometimes caused delays in disbursements. However, as this was one of the first SWAp projects, valuable lessons learned derived from this “learning-by-doing” experience for later projects in other States. 2.5 Post-completion Operation/Next Phase 9 45. The Third Minas Gerais Development Partnership DPL (P121590) 7 complemented the SWAp operation in the Partnership Series and supported the GoMG in the next phase of its reform process. By 2012, the Government’s reform strategy had evolved from the Estado para Resultados (State for Results), supported under the SWAp operation, to a focus on citizen participation, named Estado em Rede (Management for Citizenship). The aim of this operation was to maintain the support for the reforms being carried out by the GoMG, consolidate previous phases, increasing efficiency and building a permanent constituency through citizen participation while ensuring the model’s sustainability, through devolution of performance management to individual secretariats. 46. This longstanding relationship between the State of Minas Gerais and the Bank is likely to continue in the future with the new administration. Following elections in October 2014, a new administration took office in January 2015. A first step will be the dissemination of the findings of this ICR with the incoming authorities to explore potential areas of future collaboration. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 47. The project and its objectives remain highly relevant. A significant indication that the operation was not only relevant but that its results are sustainable is the fact that the project’s priority areas continued to receive attention in the Government Action Plan (PPAG) for 2012-2015. At the time of ICR preparation, the new State Government that took office in January 2015 had not yet presented its policy program indicating its priority areas. However, a proposal 8 for the new Government agenda indicates that the improvement of public services especially in the health and education sectors will continue to be priorities. 48. The choice of design proved to be an important factor for successful implementation, and continues to be relevant. The operation supported the consolidation and improvement of an innovative results-based public sector management model in Minas which has since been emulated by other states and municipalities, as well as replicated outside of Brazil. Similarly, the innovative design using a multi-sector approach and disbursement-linked indicators gave further incentives to implement reforms across different sectors and over an extended period of time. The instrument has since been replicated in several projects, and indeed was the model that led to the creation of the Program for Results (PforR) instrument in the Bank. 49. Implementation arrangements and the availability of funds for technical assistance played a crucial role in the achievements of the development outcomes. The involvement of all relevant sector secretariats facilitated coordination of implementation and further led to improved cooperation among different State agencies. This coordination 7 The Third Minas Gerais Development Partnership DPL (P121590) in the amount of US$450 million was approved by the Board of Executive Directors on July 26, 2012. For further results of the operation see the Implementation Completion Report, Report# ICR-2866. 8 See paragraphs 44 and 45 of Proposals and Guidelines for the Government Program of Pimentel 2015-2018 (Propostas e Diretrizes para o Programa de Governo Pimentel 2015-2018). Available on the internet under: http://divulgacand2014.tse.jus.br/divulga-cand-2014/menu/2014 (Accessed 2/13/2015). 10 was and remains embedded in the day-to-day operations of the Government, and presents an important achievement in the institutional change. The rather long implementation time gave room to accompany reform efforts, supporting maturity through embedment into the government’s culture and processes. The availability of funds for technical assistance under Component 2 allowed for a certain flexibility beyond the Government budget to obtain not only support for the implementation of government programs, but also for additional in- depth and innovative activities, which otherwise might not have taken place, such as studies on sectorial, economic and regional development and integration, which currently form an active part in informing and influencing policy-making of the GoMG. 3.2 Achievement of Project Development Objectives 50. The overall efficacy of the project is rated “high”. The disbursement of funds under the SWAp component (US$1,413 million) was completed in 2011, which marked a de facto completion point for the project, as (i) almost 99 percent of the funds were disbursed, and (ii) all (except one) DLIs had achieved their targets established for the period of 2008-2011. However, as the technical assistance under Component 2 continued focusing on a small number of consulting services, almost all indicators continued to be monitored for the purpose of project reporting. This allowed for a snapshot of the project at the time of completion of the SWAp Component, and for assessing the sustainability and further progress until the final closing date of the project. Annex II describes in further detail the achievement of the indicators at those different times in the project. Achievements by PDO sub-parts Part 1: To improve the efficiency and effectiveness of public resource use and allocation for economic and social development. 51. The achievement of the first part of the PDO was substantial. The existence, measurement and tracking of cost information in the various sectors was one of the indicators measuring the efficiency and effectiveness of public resources. All 21 of the health facilities under FHEMIG administration have implemented a standardized costing system to collect and monitor clinical and non-clinical costs, and the education management information system (Sistema Mineiro de Administração Escolar – SIMADE) was successfully developed and implemented. Nonetheless, the composite indicator measuring achievements (changes in primary surplus and personnel expenditures) of the Fiscal Adjustment Program (Programa de Ajuste Fiscal - PAF) was not fully achieved. This indicator, however, was outside the direct influence of the project and was affected by unexpected factors such as a teacher’s strike in 2011. In fact, one lesson learned from this operation is to avoid indicators that are outside the control of the project. 52. On the allocation side, efficiency and effectiveness were measured through investment expenditure as a share of total expenditure and expenditure on the strategic projects as a share of total expenditure. This was highly achieved as the percentage of strategic programs in total expenditures, ensuring that resources are allocated in accordance with State priorities, was more than doubled from 2007 to 2013. Efficiency of public resource use and the effectiveness of public resource allocation were also enhanced by supporting the PROMG program which aimed at rehabilitating and maintaining the state paved road network through results-based contracts, and by assisting the State Roads 11 Department (DER) and the State Secretariat of Transport and Public Works (SETOP) in institutional strengthening activities. Success in increasing efficiency and effectiveness was also measured by the average percentage delay of paving works concluded under the Pro-Acesso road program which decreased from close to 40 percent in 2007 to around 10 percent within five years. Part 2: Support the adoption of innovation in public management of the State. 53. This part of the PDO was highly achieved. As originally envisaged, the success of this section of the PDO was to be measured by the extent of implementation of Results Agreements in key sectors, the number of PPPs contracted by the State, the percentage of hospitals under the hospitals program (PROHOSP) assessed for accreditation by the accreditation entity, and the percentage of state paved roads network under results-based maintenance and rehabilitation contracts. 54. The targets involving the successful implementation of results agreements were surpassed, and by 2013 a total of 28 secretariats had achieved a performance of 70 percent or more. Annex X provides more details about the results agreements focusing on the education sector. By 2013 the PROHOSP Program supported 140 hospitals in the State, 78 percent (109 hospitals) of which had been assessed for accreditation by an institution certified by the National Accreditation Organization (Organização Nacional de Acreditação – ONA). The number of PPPs contracted increased from one PPP in 2007 to four in 2010 and six in 2013, surpassing the targets established at the outset, and making Minas Gerais one of the most advanced Brazilian States in terms of institutional setup of PPPs. 55. Although the final target for the indicator that measured the percentage of state paved roads network under results-based maintenance and rehabilitation contracts was narrowly missed at the time of the SWAp disbursement, it was later achieved and surpassed. The proportion of the State road network in good condition increased from 43 percent in 2007 to 72 percent in 2011, substantially contributing to the reduction of transport costs for road users. Part 3: Support the State in strengthening its results-based management system of monitoring and evaluation of results. 56. Although the original results framework 9 did not specify indicators for the measurement, the relevance and efficiency of this part of the PDO are rated high. The strengthening of Minas Gerais’ results-based management system was at the core of the project and a cross-cutting theme and monitoring of results was essential for the entire results-based management system established by the State. The project was instrumental in the achievement of this part of the objective, primarily through technical assistance activities and the engagement of Bank specialists with the Government team in the creation and subsequent fine-tuning of the M&E system. Specifically, the successful introduction 9 The table on results indicators in the original PAD (Table 1 in Section 1.2 of this ICR), identifies “Satisfaction of vice-governor’s office and secretariats with available M&E information” and “Strengthening of results/causal chains underlying the government’s program” as the two indicators to measure this PDO part. As they were not reflected in the results framework, they were not monitored during the project. 12 of the first and second generation of results agreements (see Annex X) provides strong evidence for the satisfactory achievement of the project objective. Achievements by sector themes 57. Public Sector Management. Rating: Substantial. Bearing in mind the GoMG’s overriding goal to extend the culture of results management across all line secretariats, the operation included a large number of public sector management indicators that encompassed implementation of Results Agreements, strengthening M&E, and improving information sources. Those were directly linked to the achievement of the PDO in the areas of adopting innovations in public management and strengthening the State’s results-based management system of monitoring and evaluation of results. The targets involving the successful implementation of results agreements were surpassed. The percentage of strategic programs in total expenditures, ensuring that resources are allocated in accordance with State priorities, was more than doubled from 2007 to 2013. A very significant degree of commitment on the part of the State ensured that the procurement reform that was carried out as part of the “Fiscal Management Modernization” EEP greatly exceeded expectations, and good practices have now become ingrained in the management culture of the State. The composite indicator targeting two PAF targets, primary surplus and personnel expenditures, was not fully achieved, which resulted in a US$1.44 million cancellation, but otherwise the State met all targets in this sector. Given the relevance of the PSM sector in this project, results achieved in this area are further analyzed in Annex X of this ICR. The annex also presents the results of a survey with 2000 employees of the Minas Government regarding their perception of these reforms. 58. Private Sector Development. Rating: High. The GoMG believed that improving the investment climate in the State was the best way to establish the conditions for long- term growth and continuous job creation. The Minas Fácil units and Technology Innovation Centers (hosted by Science and Technology Institutions) spread throughout the State were deemed important for disseminating knowledge and supporting innovations and entrepreneurial activities. Streamlining of time to start a business (consultations, taxes, licensing, payments, and registrations) was an important efficiency measure to facilitate such activity. The reduction of days necessary to start a business from 26 in 2007 to 6 in 2013 is a notable feat, and was possible due to the high degree of ownership and commitment of the implementing agency. Furthermore, it is indicative of the project’s innovative character that the number of PPPs contracted increased from one PPP in 2007 to four in 2010 and six in 2013, surpassing the targets established at the outset. Minas Gerais is now one of the most advanced Brazilian States in terms of institutional setup of PPPs. The maturing use of PPPs as a tool in public management directly contributed to the achievement of the PDO, which called for the adoption of innovations in public management. 59. Health Sector. Rating: Substantial. The interventions supported by the project to improve the State health systems included the introduction and consolidation of management mechanisms and tools. These contributed to the achievement of the PDO by improving the efficiency and effectiveness of public resource use. Quality assessments and the implementation of a standardized costing system in a number of public hospitals have improved efficiency in allocating public resources to effectively better public health 13 outcomes. By 2013 the PROHOSP Program supported 140 hospitals in the State, 109 (78 percent) of which have been assessed for accreditation by an institution certified by the National Accreditation Organization (Organização Nacional de Acreditação – ONA). All 21 of the health facilities under FHEMIG administration have implemented a standardized costing system to collect and monitor clinical and non-clinical costs. Other State programs like “Health at Home” and “Viva Vida” have been successful at the municipal level, with 841 municipalities benefited by the “Health at Home” program and 59 Viva Vida Centers implemented. In addition, data on health outcomes shows that municipalities made great progress in reducing infant deaths. The State Infant Mortality Rate decreased from 16.9 to 12.4 deaths per 100,000 live births in the last ten years, and Primary Health Care coverage reached 79.5 percent in 2014. 60. Education Sector. Rating: High. The Education sector effectively fostered monitoring and evaluation, efficiency, and innovation. Concrete actions that contributed toward the achievement of the PDO include the rising of 3rd grade literacy, the utilization of the secondary school classroom-level testing program (PAAE), the development of the SIMADE management information system, and the increase in vocational training program enrollments (PEP). SIMADE is a system that incorporates student, teacher, school and regional district information on results and human resources into a single integrated system that allows calculation of unit costs by school, district, level of education, and program. Furthermore, the operation helped the GoMG to scale-up its most promising and innovative education programs. Those programs were not only scaled-up with success under the program, but also influenced national education policy. 61. Transport Sector. Rating: Substantial. The operation’s transport sector component has particularly contributed towards achieving the PDO. Efficiency of public resource use and the effectiveness of public resource allocation were enhanced by supporting the PROMG program which aimed at rehabilitating and maintaining the state paved road network through results-based contracts, and by assisting the State Roads Department (DER) and the State Secretariat of Transport and Public Works (SETOP) in institutional strengthening activities. Success in increasing efficiency and effectiveness was measured by the average percentage delay of paving works concluded under the road program Pro- Acesso which decreased from close to 40 percent in 2007 to around 10 percent within five years. By increasing the number of highway PPPs and the percentage of the paved highway grid under results-based maintenance and rehabilitation contracts, the transport program also helped to support the adoption of innovations in public management of the State. 62. Environment and Social Quality. Rating: High. This area included an efficiency DLI relating to both the private sector and environmental efficiency, as well as two social inclusion DLIs benefitting indigenous communities. All three indicators were achieved before the target date of 2011. The environmental DLI called for the average time for concession of environmental licenses for two classes of processes to be halved to 90 days primarily through the use of IT technology. This had been achieved by 2009, and was maintained through 2011. The number of certified teachers in indigenous areas was increased from 137 to 213 during project life, leading to greater empowerment of indigenous communities and contributed to a rise in the level of indigenous education. On indigenous health, increasing the number of inter-disciplinary indigenous health teams from 12 to 16 in 14 five years has contributed to the increase in quality of the service of Primary Indigenous Health and Care in the State. 63. Poverty Reduction. Rating: High. The AF provided additional direct support to the Government’s poverty reduction strategy by supporting the application of the results-based management approach to programs aimed at poverty reduction, and including the Rural Poverty Reduction Project (RPRP) as a new EEP. Two DLIs were related to the EEP: the number of families to be benefitted by the program, which was not only achieved before the end target date, but also surpassed; and the intensive monitoring of the RPRP by SEPLAG and the Instituto de Desenvolvimento do Norte e Nordeste de Minas (IDENE), which was achieved by holding a total of 6 monitoring meetings a year in 2010 and 2011. The two additional poverty DLIs were related to the health and education sectors. One called for the State Secretariat of Education to develop 13 courses for initial job training in the area of information technology. The target for this indicator (90 schools in the Great North Region offering initial work preparation courses for the assembling and maintenance of computers) was achieved in 2010. The final DLI measured the proportion of live births from mothers with 7 or more pre-natal consultations in the Great North. The target for this indicator was not only achieved, but the proportion of live births has increased from 46.1 percent in 2007 to 66.6 percent in 2013. As a comparison, the proportion of live births in the State of Minas Gerais was 61.4 percent in 2007 and 71.5 percent in 2013. 64. In addition, Technical Assistance (Component 2) activities in all of the sectors contributed to the achievement of the Objectives. Those activities included strengthening procurement and improving the knowledge base on which sector strategies, budgets and targets were based. Some of the activities carried out were studies including a household survey, impact evaluations in education, health and transport, a quality assurance survey, and a study of output indicators and links to outcomes. 65. The TA activities were directly linked to support the achievements of DLIs. For instance, a study measuring the quality of service provision to businesses can be directly linked to achievement of the DLI that measured the time necessary to open a business in the State capital. Similarly, an assessment of the state capacity to handle road safety and the identification of a set of actions to improve capacity informed the efforts of the GoMG toward the achievement of DLIs in the transport sector. 3.3 Efficiency 66. The overall efficiency of the operation is rated as substantial, as efficiency gains are likely to have occurred. Given the multi-sectorial nature of the project, a broad economic assessment was unfeasible and therefore only partially undertaken during project preparation. A more detailed ex-ante economic analysis was conducted in the areas of education and transport focusing primarily on economic returns to education and economic rates of return based on calculations comparing traffic volume and cost. 67. An analysis comparing current with pre-intervention outcomes in education, health, and the security sector in Minas and other Brazilian States that implemented management reforms was carried out. This 2013 World Bank analysis examined changes in relation with regional trends and results achieved by other States to determine 15 if the improvements depart in meaningful ways from the overall trend. 10 For the education sector, the study finds positive results going beyond the average of the Southeastern region. In the health sector, however, the study finds no direct impact of the reforms since trends in health outcomes remained within the regional average. 68. A simplified ex-post economic analysis was carried out for the transport sector, which confirmed that the programs supported under the SWAp operation remain economically viable. At project appraisal, a cost-benefit analysis was carried out to evaluate the transport sector’s two principal programs, namely PROMG-Pleno and Pro- Acesso. The NPV of investments under the PROMG-Pleno program was calculated as R$2.245 at a 12 percent discount rate over an evaluation period of 20 years yielding an IRR of 35.60 percent. The Pro-Acesso program’s NPV was estimated to be R$998 million at a 12 percent discount rate with an IRR of 26.42 percent. However, a simplified ex-post evaluation based on available traffic volume and cost data was carried out and came to the conclusion that IRR and NPV have improved compared to the ex-ante evaluation in both programs (see Annex III for details). 69. To capture civil servants’ perception of the management reforms, a survey was conducted of Minas’ Government employees. The survey results (see Annex X) indicate that the majority of respondents welcomed the introduction of results agreements and have a clear understanding of their role and objectives. 3.4 Justification of Overall Outcome Rating Rating: Satisfactory 70. Measured by the ICR guidelines, the overall outcome rating is considered ‘satisfactory’ as the project has largely achieved its development objectives. Despite some design shortcomings, it has effectively contributed to the improvement and expansion of the Borrower’s results-based management system of monitoring and evaluating results. This improvement has led to an increase in efficiency and effectiveness of public resource use in public institutions in Minas Gerais. Furthermore, the rating is based on the operation’s positive efficiency ratings in the sectors it included and prevailing relevance, particularly in the development areas tackled and the importance of results-based management, not only in the State of Minas Gerais, but throughout Brazil. While most impact and significant results were achieved by 2011, when the SWAp component was disbursing, those achievements proved to be sustainable as most of the indicators showed sustained or increased achievement through the subsequent years in which only TA activities were under implementation. 71. However, it should be pointed out that this was a highly successful project whose overall impact and success cannot be fully captured by only assessing the achievement of the PDO indicators. The project paved the path for significant reforms and a culture change in the public administration of Minas Gerais. As one of the first successful models of result-based management, it had spill-over effects to other states and countries, which replicated the system with adaptations. Certain sector policies, such as in 10 Viñuela and Zoratto (2013): Performance Gains in Results-Based Management in Brazilian States, Report No. 82592-BR. See also Annex X for more details. 16 education, influenced national policy-making taking Minas as a successful example. Another element to be taken into consideration when assessing the overall outcome of the project is its innovative nature at the time of preparation, as well as its size, both in multi- sector coverage and lending amount. The project served as a pioneer in many of those aspects and valuable lessons were learned from its success, but also from its shortcomings. Those lessons served to further develop the SWAp instrument and have been integrated in similar operations elsewhere. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 72. The poverty rate in Minas Gerais has been drastically reduced since 2008. The poverty rate decreased from 13.3 percent in 2006 to 5.4 percent in 2013. This implies that over the same period the poverty headcount was reduced by 1.5 million from 2.5 million in 2006 to 1 million in 2013. 11 Although the extent to which the project has contributed towards the reduction of overall poverty is difficult to determine due to the many factors influencing poverty rates, several DLIs related to poverty across various sectors, such as education and health, showed considerable improvements. Also, State’s Rural Poverty Reduction Project (RPRP) was supported by the project with the AF. 73. The transport sector contributed to reduce poverty and inequalities. An impact evaluation study 12 conducted by the client demonstrated that the Pro-Acesso program in the transport sector contributed to alleviate poverty and to reduce inequality in rural areas of the State through fostering economic growth. The economic development index 13 of the 225 municipalities under the Pro-Acesso increased by 51 percent between 2000 and 2010, which is higher than the State average of 35 percent. While the index of the Pro-Acesso municipalities was 24 percent lower than the State average in 2000, the difference had been reduced to 17 percent by 2010. 74. Components added with AF improved indigenous health outcomes. Targets set to increase the numbers of certified teachers in indigenous areas and inter-disciplinary health indigenous health teams were achieved and surpassed, leading to an improvement in the quality of primary indigenous health care in Minas Gerais. (b) Institutional Change/Strengthening 75. Institutional strengthening was at the very core of this project. The implementation of a results-based management system, the introduction of Results Agreements and the procurement system reform significantly improved institutional functioning. The Results Agreements, with their link to monetary bonuses, supported the 11 There have been many changes in methodology to compute poverty rates since when the PAD was written. The numbers cited are based on national income definitions (using PNAD) and refer to the part of the population that lives on less than R$140 per month. 12 Fundação Joao Pinheiro (2013): Avaliação do Impacto Socioeconômico do Programa de Pavimentação de Ligações e Acessos Rodoviários aos Municípios (PROACESSO). 13 The impact evaluation study calculated an economic development index by combining 20 indicators in the areas of productivity, municipal finance, and employment and income. 17 change towards a results-oriented culture in the administration, while the procurement reform introduced an e-invoicing system which it is being used by both the State administration and its suppliers The results of the survey with Minas’ civil servants (see Section 3.6 and Annex X for more details) indicate that the majority of respondents believe that introduction of results agreements helped them better understand their role and purpose inside the organization. (c) Other Unintended Outcomes and Impacts 76. The operation leveraged the State’s successful results-based management model as an example for other States. The Minas Gerais Partnership Series helped to develop solutions to enhance public sector performance and service delivery issues which were used by other States and countries. The success of the public sector reforms in Minas Gerais proved to be an important model with continuing and useful demonstration effects for Brazilian States and municipalities such as Pernambuco, Rio de Janeiro, Manaus, and other countries facing similar development problems. 77. Incubator for innovative operational approaches. This operation was one of the pioneers of the SWAp modality for sub-national projects in Brazil. The approach then spread out to projects in other Brazilian States and countries (i.e. Argentina), where the design of performance-linked disbursement projects matured through further refinement and integration of lessons learned from the implementation of the first operations. The SWAp modality as a lending instrument heavily influenced the discussion for the need to adjust Bank instruments to the needs of clients, especially in middle-income countries, which eventually led to the creation introduction of the PforR instrument. 78. Adjustment of procurement arrangements for SWAp operations. Significant preparation work was done on the procurement arrangements of the operation, particularly through an in-depth review of national procurement laws, which led to adjustments of the procurement arrangements, including the acceptance of all competitive procurement procedures under the national procurement laws and increasing procurement method thresholds. The effort helped to set an example and define certain standards which were then used for subsequent SWAp operations. 79. Expansion of the Government agenda towards integrated regional development. The preparation of the Economic Master Plan for the Metropolitan Region of Belo Horizonte set a cornerstone for increasing the Government’s focus on strategic economic development of the metropolitan area. The Master Plan has provided a long-term vision of development needs in terms of social, economic, physical, and environmental planning and infrastructure accompanying this development. Its implementation is now a priority of the Government and intended to be replicated for other urban areas to foster municipal development. 80. Transition to the next reform phase. The activities and support for maturing the reforms from the Choque de Gestão under the operation underpinned the State’s desire to continuously expand and deepen reforms and led to the definition and transition into the next reform phase. This phase, the Estado em Rede, was supported by a DPL operation and focused on citizen engagement in priority setting of policies. (See also Section 2.5) 18 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops 81. A beneficiary survey was not conducted. However, the team conducted a survey among public servants in Minas Gerais directly affected by the implementation of results agreements. The survey was carried out in mid-December 2014 among public servants to capture their perceptions about the introduction and expansion of the results-based public management system since 2003. 82. The majority of respondents agreed or strongly agreed with the assertion that the introduction of results agreements had improved the functioning of the public service in Minas Gerais. Likewise, the majority of respondents believe that the results agreements established are relevant to the overall mission of the respective institutions. 92.7 percent of the respondents that work for the Minas Government for 5 years or more indicate that they have a clear understanding of their role and goals at work. Two thirds of those respondents also assert that they understand their role and goals better than five years ago. 83. Limitations but overall robust findings. The survey was conducted ex-post and the lack of a baseline and thus reliance on recall questions limit the validity of the findings. In general, however, the survey is indicative of positive perceptions regarding the results- based management reforms (see Annex X for survey results). 4. Assessment of Risk to Development Outcome Rating: Moderate 84. Given the nature of the reforms supported by the operation and the ensuing results- based management culture that is now embedded in the system, it is not expected that the Development Outcome will be reversed or significantly changed in the foreseeable future. However, as a new Government took office in January 2015, and the degree of future Bank involvement in Minas Gerais is yet be determined, the assessment of the risk to Development Outcome is considered ‘moderate’. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Satisfactory 85. Special attention was given to the preparation of the project. Given the importance in the Brazil portfolio, the size of the project both in terms of sectors involved and lending amount, and the complexity of the SWAp modality, the Bank deployed a large team of sector specialists with the required knowledge and who were dedicated, in cooperation with the counterpart, for the preparation of the project. As emphasized by current and previous civil servants, the Bank’s technical support was extremely helpful in defining the indicators during the preparation of the SWAp. The close preparation with the counterpart did not only allow for designing a project that addressed the necessities of the Borrower, but also laid the ground for a close relationship, which facilitated implementation. 19 86. At the time of preparation it was expected that changes to the original design would be needed during implementation. As the operation was one of the first SWAp operations in Brazil and the instrument was still in a stage of fine-tuning, changes to adjust the design were anticipated during preparation. As part of the Additional Financing, the majority of the original results framework was changed as the indicators and targets had either already been achieved or were no longer relevant. Similarly, the achievement of some DLIs was beyond the control of the implementing agencies. (b) Quality of Supervision Rating: Satisfactory 87. Supervision carried out by Bank staff occurred on a regular basis and, during most of the project. In the first years of implementation, a close supervision by the Bank team took place, including the preparation of the AF, support for the complex disbursement arrangements and for the large procurement packages. Furthermore, Bank staff also supported the preparation of TORs and gave guidance on the selection of consultants and firms, as per Bank guidelines, which seemed to have an impact on the quality of the service providers. 88. Of critical importance for the supervision was the high level of participation of staff from the Brazil Country Office. Most of the time, at least one TTL was based in the Brasilia office, as were the fiduciary staff and several sector specialists. This facilitated the communication and ensured a responsive and continuous dialogue. Furthermore, the fact that one of the TTLs had previous experience within the Government of Minas Gerais and with the implementation of the Choque de Gestão proved to be a valuable asset. 89. However, over the time of implementation, Bank supervision decreased. Once the SWAp component had been fully disbursed in 2011 and only TA activities remained to be carried out, supervision slightly decreased. Also, the quality of the ISRs varied over the time of the project and sometimes lacked information; however, the format of the ISR changed twice during the implementation period. During the extension periods of the project, implementation of safeguards was not followed as closely, with the result that the delays on completion of the compensation processes was not flagged as early as it could have been. (c) Justification of Rating for Overall Bank Performance Rating: Satisfactory 90. The overall rating is satisfactory, as the Bank team was able to respond to and successfully address the Government’s development issues through a highly complex operation. Even though, the design at entry and the supervision during the extension periods show some shortcomings, in the overall scheme, the Bank team prepared and implemented a complex project, which at the time of preparation was a new, highly innovative instrument with few predecessors. Therefore, adjustments to the project design were expected (and desired) as they offered the opportunity to fine-tune the design based on lessons learned during implementation. 20 5.2 Borrower Performance (a) Government Performance Rating: Highly Satisfactory 91. The Government played an exemplary role during the preparation and implementation of the project. The continuous commitment and high level of ownership of the reforms was essential to the success of the operation. Prior to project implementation, the Government had created an institutional framework that facilitated the implementation of the SWAp. Arising implementation issues were addressed in a timely manner, and the Bank’s large and sometimes intrusive supervision efforts were well supported. (b) Implementing Agency or Agencies Performance Rating: Satisfactory 92. Despite the challenges associated with the implementation of a project with such a large scope, the implementing agency’s performance was satisfactory, which to a great extent can be attributed to the high technical capacity of the Minas Gerais public servants, which itself was a reflection of the success of the reform process initiated by the State in 2003. Throughout the life of the operation PDO indicators, Implementation Progress, FM, Project Management, Procurement, M&E were rated at least ‘satisfactory’. The counterpart team facilitated Bank supervision and played a crucial role in refining and consolidating the Choque de Gestão, and in achieving the project objectives. However, the internal control arrangements were at times considered weak due to high staff turnover, and the implementation of the social safeguards lacked some attention. (c) Justification of Rating for Overall Borrower Performance Rating: Highly Satisfactory 93. In aggregate, the overall borrower performance is rated highly satisfactory. The rating is based on the strong commitment shown by the Government and the exemplary responsiveness and performance of the implementing agency throughout the successful preparation and implementation of this challenging and innovative multi-sectorial operation. 6. Lessons Learned 94. The completion of this operation represents an important opportunity to draw lessons learned. The importance and success of the project, both in terms of its multi- sectorial, performance based-disbursement design and the public sector reforms, namely in results-based management implemented by the State since 2003, have drawn attention to the project during the years of implementation and have been adapted in other Bank operations. Annex X presents additional lessons learned from the introduction of results- based management in Minas. Technical Assistance 21 95. The importance of a technical assistance component to foster innovation. While the financial support for government programs was crucial for the financing of those, the relatively small (US$18.5 million out of US$1,437 million) component reserved for technical assistance activities proved to have a significant impact. Besides supporting the implementation of government programs under the SWAp component, it allowed the State to undertake projects which otherwise would not have been affordable; and innovative and therefore somehow risky activities, which might not have been approved in the State’s budget process. Those activities financed under the technical assistance also supported and contributed input for Minas Gerais’ continuous fine-tuning of reforms as well as the process towards next reform steps (such as integrated regional development and citizen participation in priority setting of public policies). 96. The value-added through technical assistance provided by the Bank. As emphasized by current and previous government staff, the Bank’s technical support was seen as one of the main value-added. Technical discussions during preparation and implementation with sector specialist were received as a very valuable mechanism to brainstorm ideas and to receive feedback on the government programs. Similarly, the assistance from Bank staff in the definition of the indicators was seen as helpful. Furthermore, the continuous assistance by fiduciary staff led to strengthening the State’s capacity and quality of financial management and public procurement. 97. The importance of technical assistance as a vehicle for continued dialogue. As the activities under the technical assistance component required continuous Bank engagement and was implemented at a slower pace, the dialogue between the Government and the Bank continued beyond the end of the disbursement of the SWAp component. This has led to a close and constant engagement with the State of Minas Gerais, including the implementation of a Development Policy Loan in 2012 and strengthening the role of the Bank as an important development partner for the State. Performance-linked disbursement and multi-sectorial design 98. Disbursement-linked indicators as a performance push. Tying disbursements to targets seemed to have prompted progress that would not otherwise have occurred. As the achievement of the targets was directly linked to the frequent disbursement of funds, anecdotal evidence indicates that recalcitrant secretariats were incentivized to meet the targets as they did not want to hinder disbursements. 14 Sectors such as health and education have earmarked revenues linked to them and as such are prioritized. The link between the DLIs and the Results Agreements probably contributed to the success of State organizations in meeting their targets. A similar effect was observed in the budget execution of the EEPs, where some previously underperforming programs significantly increased their execution during the operation. 99. Attention needs to be given to the design of the DLIs. The careful and appropriate design of the DLIs is crucial to serve as a suitable incentive scheme. First, the achievement of targets should be in the scope of control of the respective entity in charge. Otherwise, targets might be perceived as unfair. Similarly, DLIs need to be properly defined so they 14 For the this reason, the SWAp was considered more effective than a DPL (at least a one-tranche DPL) as the Bank leverage lasted longer. 22 can be easily measured and tracked over a prolonged period of time, without giving margin to alternative interpretations. 100. The financing of eligible expenditure programs (EEPs) can serve to protect certain categories of spending in certain sectors. As the loan disbursed against eligible expenditures, budget allocations could not be cut in those categories or sectors without jeopardizing disbursements of the loan. While this had the advantage that sectors such as transport benefitted by receiving higher budget allocations than previous to the operation, and partially offset the traditional bias towards cutting investment in a crisis, it did have the negative effect that the State could not re-assign funds to emerging priorities, as for example, during the fiscal setback of 2009, when the State wanted to allocate more funds to programs that would increase citizen participation. 101. The importance of aligning EEPs with the indicators. The relative representation of EEPs should be aligned with the number of DLIs measuring achievements in those EEPs to avoid that some sectors with less EEPS are expected to contribute to a large amount of DLIs, and vice versa. 102. Trade-offs between a multi-sectorial approach and efficient implementation. While the inclusion of multiple sectors provided a platform to approach cross-cutting issues and ensured involvement and commitment across the Government, they can bear certain difficulties for the implementation. For example, the involvement of multiple entities with varying implementation capacity, staff turnover, and additional layers of reviews and decision-making can slow down implementation. Fiduciary Management 103. It is crucial that capacity building and familiarization with the Bank’s fiduciary requirements is initiated prior to project effectiveness. Reporting can be improved by synchronizing deadlines, for example, setting the same deadlines for financial and procurement audit report, and for IFRs and disbursement requests. Particular care must be taken when two different fiduciary arrangements (SWAp component and TA component) are included in the same operation. Primary among the lessons learned from the project is the need for a continuous and careful supervision effort on the part of the Bank. Role of the Government and the Bank 104. One of the main success factors undoubtedly was the high Government ownership and commitment. Starting with the Minas Gerais Partnership in 2006, commitment and ownership to the reform process and the supporting Bank operations was present on the highest political and technical levels in the State. Strong political leadership from the Governor and Vice-Governor was perhaps the single most important factor in the success of both the reforms and the Bank operation. Some high-level staff, such as the Planning Secretary, remained throughout the life of the project. This ensured continuity as well as a good working relationship with the Bank, both of which were of great importance to the success of the operation. At the same time, the Results Agreements served as an instrument to trickle down the implementation commitment to the technical secretariat- level. 105. High capacity and fostering innovation. The overall high capacity of the counterpart, including many motived and innovative young staff members facilitated 23 preparation and implementation of this complex operation. Furthermore, the administration, in part through this operation, provided a breeding ground for innovative approaches and encouraged staff to engage in them. This environment contributed to transforming Minas into a ‘good practice’ experience for results-based management and other public administration reforms. 106. Bank involvement and support gave credibility to the process. Anecdotal evidence suggests that the Bank financing, and more importantly the role of the Bank as a partner and its technical expertise, gave legitimacy and credibility to the reform process and the definition and monitoring of the results agreements. Similarly, the external recognition provided by the Bank reportedly elevated staff’s morale and was fundamental to help the teams achieve results. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies 107. In the Borrower’s Completion Report, which was received by the Bank in October 2014, and although was very complementary overall. However some detailed problems were raised. Most of those issues were overcome and successfully addressed during project implementation. With regard to the TA, the report regrets that, occasionally, it took the Bank an unnecessary long time to send letters of no objection thus delaying the hiring of personnel. The Bank recognizes the difficulties raised by the Borrower and can report that lately there has been progress made to shorten approval processes and increase responsiveness to client’s needs. 108. The comments provided by the Borrower on the Bank’s ICR were primarily editorial and have been included in this version of the ICR. (b) Cofinanciers N/A (c) Other partners and stakeholders N/A 24 Annex I. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Appraisal Additional Revised Costs Actual/Latest Estimate Financing Components (USD Estimate (USD (USD millions) (USD millions) millions) millions) Component 1 (SWAp Component) 955.06 459.85 1,414.91 1,413.48 Component 2 (TA Component) 18.5 0.00 18.5 17.81 Total Baseline Cost 973.56 459.85 1,433.41 1,431.29 Physical Contingencies 0.00 0.00 0.00 0.00 Total Project Costs 973.56 459.85 1,433.41 1,431.29 Front-end fee IBRD 2.44 1.15 3.59 3.59 Total Financing 976.00 461.00 1,437.00 1,434.88 (b) Financing Appraisal Additional Actual/Latest Estimate Financing Estimate Percentage of Source of Funds (USD (USD (USD Appraisal millions) millions) millions) Borrower 0.00 0.00 0.00 International Bank for Reconstruction 1,437.00 461.00 1,434.88 99.85 and Development 25 Annex II. Outputs by Component 1. The original project contained 24 indicators, all of which were Disbursement- Linked Indicators (DLIs); with the Additional Financing, the number of DLIs was increased to 32. DLIs were linked to Eligible Expenditure Programs (EEPs) which are presented in the tables below in the beginning of each sector section, jointly with a summary of the achievement of the indicators. 2. The disbursement of the funds under the SWAp component (US$1,413 million) was completed in 2011, which marked a de facto completion point for the project, as (i) almost 99 percent of the funds were disbursed, and (ii) all (except one) DLIs had achieved their targets established for the period of 2008-2011. However, as the technical assistance under Component 2 with a small number of consulting services, almost all indicators continued to be monitored for the purpose of project reporting. Therefore, the tables below by sector show (i) the originally established target date, (ii) the year the final target was actually achieved, and (iii) the latest available status. This allows to create a snapshot of the project at the time of completion of the SWAp Component, and allows for showing the sustainability and further progress until the final closing date of the project. 1. SWAp Component Public Sector Management Eligible Expenditure Programs Tax Efficiency and Simplification Modernization of Fiscal Management Professionalization of Public Managers Sectoral Public Reform Agenda Indicator Baseline End Target Value (year Status (latest (PDO indicators in bold) (year) (year) achieved) year available) Implementation of Results 4 (2007) 5 (2009) Substituted at the time of Agreements (number of AF with indicator below. Secretariats with verified results-agreements performance above 70%) Implementation of 1st stage 7 (2009) 7 (2011) 8 (2010) 28 (2013) Results Agreements (number of Secretariats with verified results-agreements performance above 70%) Note: Substitutes previous indicator. Implementation of 2nd stage 10 (2009) 11 (2011) 11 (2010) 61 (2013) Results Agreements (number 26 of agreements evaluated in due time) Certification Number of 2 (2007) 6 (2009) 6 (2009) - of Public certified job Sector categories Positions Compliance - i) i) Second round with selected recertification recertification of actions; of civil of civil recertification Note: AF servants for servants for held and 70 modified the specific specific managers unit of positions; ii) positions; ii) trained measurement. training for training for (2012) specific specific certified civil certified civil servants servants (2010) (2010) Indicators of MG State i) R$ 1,704 i) R$ 1,406 i) R$ 2,005 i) R$683 Restructuring and Fiscal million; ii) million; ii) million; ii) million; ii) Adjustment (PAF) agreed with 59.43% (2007) 60% (2011) 63.01% 65.14% the STN (i. primary surplus; ii. (2011) Not (2013) Ratio of personnel achieved expenditures with respect to net current revenue mentioned in the current PAF) Gross Operating balance 11% (2009) 14% (2011) 14.95% 7.91% (2013) (2010) Participation of Strategic 6.95% (2007) 9.50% 9.85% 16.62% Projects in Budget (2010) (2009) (2013) Expenditures Management Systems for No system of System of Electronic - Procurement electronic management invoicing invoicing, of civil system management of construction developed civil contracts contracts, and fragmented integrated implemented, (2007) with SIAD integrated and SIAFI, in with SIAD use (2011) and SIAFI (2010) Improvement of results - Final reports Final reports - management of 3 impact of 3 impact evaluations evaluations concluded concluded (2010) (2010) Strengthening of Evidence- Strategic Executive Executive - based policy-making projects under evaluation of evaluation of implementation indicators is indicators is and ongoing done; a pilot done; a pilot monitoring of of executive of executive 27 output and evaluation is evaluation is outcome completed completed indicators and a and a (2009) document for document for transition transition about M&E about M&E agenda is agenda is prepared prepared (2010) (2010) 1. An overriding goal of the Government was to extend the culture of results management across all line secretariats. This was addressed in the operation by including indicators that encompassed the successful implementation of results agreements, strengthening M&E, and improving information sources. More concretely, it included increasing the percentage of structural (priority investment) programs in total expenditures to ensure resources were allocated in accordance with state priorities. 2. Strengthening of the Choque de Gestão was done primarily by requiring seven main secretariats to meet a 70 percent performance requirement. This indicator was quickly met and surpassed, and by 2013 a total of 28 secretariats had achieved a performance of 70 percent or more. The operation also supported expanding the results- based management framework to secretariats, requiring that second stage results agreements in eleven main agencies be evaluated by the end of the first quarter of the following year. Annex X of this ICR provides a closer look into the results-based management model pioneered by Minas in Brazil. The indicator “strengthening of Evidence-based policy-making” focused on strategic projects making use of indicators, again with the intention to strengthen an important facet of RBM framework monitoring and evaluation. 3. Procurement reforms under the project were highly successful and contributed to the use of country systems. The procurement reform that was part of the “Fiscal Management Modernization” eligible expenditure program had the following objectives: (i) to rationalize purchases and contracting procedures for at least two service families and (ii) design and implement standard pre-purchasing procedures in at least one secretariat. These objectives were not only fulfilled, but exceeded their original scope. The rationalization of purchase and contracting procedures included three groups of services and three of goods, and the pre-purchasing procedures were piloted in six secretariats in 2010 and rolled out to a total of 22 in 2011, and to all purchasing units of the State Government by 2014. The online publication of a unified procurement calendar for the State has also been a result of this reform. 4. One of the most remarkable aspects of procurement reform in Minas Gerais is how seriously it was taken. To ensure different secretariats adhered to the reforms, so that they were and remain sustainable, SEPLAG included both activities as indicators in the results agreements that it signed with the secretariats. 5. The ratio of personnel expenditures with respect to net current revenue increased from 59.43 percent in 2007 to 63 percent in 2011 and further to 65.7 percent in 2012. With this result, the Minas Government surpassed the PAF limit and did not 28 achieve the DLI, which led to a reduced disbursement and a partial cancelation of loan funds. The main reason for the increase in expenditures with personnel was the raise of the wage floor (piso salarial) for teachers, doctors and police, in compliance with the federal law. This raise reflected demands from teachers following the 2011 strike – when public schools closed for several months - and a general pressure for improvements in the citizen security sector. Minas Gerais had one of the lowest piso salarial for teachers in Brazil, before the abovementioned increase. In retrospect, and after discussions with officials from Minas, the Bank team believes that this type of indicator can be avoided in the future, as it was already part of a commitment with the Federal Government (through the PAF) and as such, from the point of view of the Borrower, it was a redundant indicator. The Bank team which prepared the operation, however, mentioned that this indicator was positively seen during Bank internal reviews and therefore it was kept as a DLI. 6. The indicator regarding the participation of strategic projects in budget expenditures was met. However, it is worth mentioning that both the Bank and Minas’ teams recognized that similar indicators should be avoided in the future, for a couple of reasons: i) with high turnover of staff in Minas, the memory of calculus of this indicator was lost a few times and different people calculated it in different ways, with different results; ii) because “strategic projects” was not precisely defined, the indicator can easily be manipulated to meet its target. Private Sector Development Eligible Expenditure Programs Simplification of Doing Business PPPs Network for Innovation Technology Certification Systems Indicator Baseline End Value Status (last (PDO indicators in bold) (year) Target (year year available) (year) achieved) Time to start a business at Minas Fácil in 26 (2007) 10 (2010) 9 (2009) 6 (2013) Belo Horizonte (days) Implementation of Minas Fácil in the 21 (2007) 29 (2011) 29 (2009) 111 (2013) entire State (number of Minas Fácil physical units) Number of PPPs contracted 1 (2007) 3 (2011) 4 (2010) 6 (2013) Technology innovation centers 4 (2007) 15 (2011) 15 (2011) 18 (2013) 7. The Government believed that improving the investment climate in the State was the best way to establish the conditions for long-term growth and continuous job creation. The Minas Gerais Integrated Development Plan 2007-2023 had as one of its main objectives the need to increase dynamism and innovativeness to promote economic growth and sustainable social transformation, making Minas Gerais “the best place to do business in Brazil”. The Minas Fácil units and Technology Innovation Centers (hosted by Science and Technology Institutions) spread throughout the State were deemed important for disseminating knowledge and supporting innovations and entrepreneurial activities. 29 Streamlining of time to start a business – consultations, taxes, licensing, payments, and registrations – was an important efficiency measure to facilitate economic activity in Belo Horizonte. The reduction of days necessary to start a business from 26 in 2007 to 6 in 2013 is a notable feat, and was possible due to the degree of ownership and commitment presented by the implementing agency. Although not included as a DLI, the Government also succeeded in reducing the numbers of days to close a business (from over 100 to 13 days). The reduction of the numbers of days to start and close a business was achieved through a successful technical assistance contract. The Borrower considers this technical assistance to have been highly successful due to its close definition and monitoring by SEPLAG, and by the fact that the (local) consulting company knew the Minas context and institutions well. 8. Minas has the largest numbers of PPP contracts in Brazil. With funds from the TA component, the State created a PPP Unit in the Secretariat of Economic Development (Secretaria de Desenvolvimento Econômico – SEDE) and provided training to its staff members. Some of the PPP contracts in Minas Gerais included a prison, an Integrated Services Unit (Unidade de Atendimento Integrado – UAI), the “Mineirão” soccer stadium and a highway (MG-050). Health Eligible Expenditure Programs Regionalization of Healthcare Home Healthcare Live Life Indicator Baseline End Value Status (last (PDO indicators in bold) (year) Target (year year (year) achieved) available) Percentage of hospitals of PROHOSP 0% 75% 72.66% 78% (2013) program formally assessed for (2007) (2011) (2011) Not accreditation by an institution certified achieved. by the National Accreditation Organization (ONA) Implementation of a standardized - Cost Cost - costing system in FHEMIG hospitals system system module module developed developed and and piloted in piloted in 2 2 hospitals hospitals (2011) (2011) Impact Evaluation of Viva Vida Centers Excluded from the project in 2011 9. The project supported activities focused on improving the State’s healthcare systems through the introduction and consolidation of management mechanisms and tools at all three healthcare levels (primary, secondary and tertiary). Progress was 30 observed in reducing inequalities in health services and increasing access to the Unified Health System (Sistema Único de Saúde - SUS) network, especially in primary care. As health was a Government priority, a significant and increasing amount of resources had been allocated to the SUS, and health expenditures represented an important amount in the municipal budgets. As a result, State programs such as “Health at Home” (Saúde em Casa) benefited about 841 municipalities and under the “Viva Vida” Program with 59 Viva Vida Centers inaugurated. However, those program required co-financing from the municipalities, meaning more a political induction than a sustainable financial support. 10. Municipal healthcare systems were improved by developing instruments that supported an integrated coordination among different levels and strategies of the healthcare system, as for example the Family Health Strategy with other levels in the healthcare system, such as primary healthcare. This was implemented by setting up and strengthening regulatory structures involving the Municipal Health Secretariats and decentralizing responsibilities of the Family Health Units to the local level. Improvements also included the re-organization of processes and protocols, introduction of electronic patient charts and the expansion of the provision of specialized services at the municipal level. While Minas has some mechanisms for integration among different levels of care, especially referral and counter-referral systems, there is still a lack of integration among different healthcare providers, insufficient policies for care of medium complexity cases and further need for process and protocol enhancement. 11. Data on health outcomes shows that municipalities made great progress in reducing infant mortality, as the State Infant Mortality Rate decreased from 16,9 to 12,4 deaths per 100,000 live births in the last ten years, and Primary Health Care coverage reached 79,5 percent in 2014. Nevertheless, these positive results cannot only be attributed to specific programs or incentives, as the social and epidemiological dynamics of infant mortality in the municipalities within the state are much more complex. 12. The operation also supported the improvement of hospital infrastructure. The PROHOSP (Hospital Strengthening and Quality Improvement Control Project), is part of the State’s regionalization process, which was initiated in 2003 and is an umbrella program that seeks to rationalize the set-up of the health care network and improve the quality of care by reducing the amount of small, inefficient hospitals, and creating regional networks with strong linkages among medical care organizations, developing information systems to support the networks, and establishing incentives and accountability schemes to improve hospital performance. 13. PROHOSP uses targets and performance indicators to evaluate hospitals, encouraged improvements in both organizational and physical infrastructure. In return, participating and complying hospitals received funds transferred by the Government of Minas Gerais. In 2013, the Program supported 140 hospitals in the State, 109 (78 percent) of which have been assessed for accreditation by an institution certified by the National Accreditation Organization (Organização Nacional de Acreditação – ONA) 15. Distinct patterns of influence in the municipalities were observed, with 15 Accreditation is a formal process by which a body accredited by ONA assesses that a health care organization meets applicable, predetermined and published quality-of-care standards. A hospital is 31 regionalization being marked by important gains in institutionality and governance. Nevertheless, inherent difficulties of context undermined greater advances. There is a pressing need to broaden the territorial focus in government planning and to integrate sectorial policies for medium and long-term regional development in order to empower regionalization and to overcome obstacles to access to health care services in the State. 14. The Hospital Foundation of Minas Gerais (FHEMIG) also played an important role in this process, by using information derived from the use of hospital cost management mechanisms as a management tool for strategic decision-making hospital organizations. The hospital cost management provided information that was more specific and accurate than the previous system of costing per absorption, and it allowed to identify shortcomings and bottlenecks along the entire chain of hospital internal processes. The 21 health facilities under FHEMIG administration now have implemented a standardized costing system to collect and monitor clinical and non-clinical costs. Education Eligible Expenditure Programs Management of Elementary Education Secondary Vocational Education Improvement in Quality and Efficiency of Secondary Education Administrative Support to Elementary Education Administrative Support to Secondary Education Development of Elementary Education Indicator Baseline End Value Status (last (PDO indicators in bold) (year) Target (year year (year) achieved) available) Percentage of 8 year old students 65% (2007) 78-80% 87.10% 92.3% registered in state schools with (2011) (2010) (2013) expected reading level in the 3rd year of elementary school System of evaluation and results - PROALFA Substituted at the time of feedback and AF with following PROEB indicator. conducted at classes of 3rd, 5th, 9th grade of elementary school and 3rd grade of high considered assessed when an ONA-certified institution completes a formal quality assessment and issues a report on the results. 32 school. Plan of action in 70% of the schools (2009) (i) Number of New Test Items; (ii) i) 24,633; ii) i) 35,000; i) 36,036; i) 97,911; ii) Participation rate (Percentage of 500 State ii) 1,000 ii) 2049 7924 (2013) schools that apply the test for the 9th secondary State (2010) grade classrooms in five subjects - schools secondary mathematics, Portuguese, physics, (2008) schools chemistry and biology) Note: (2011) Substitutes previous indicator. Cost-Student Informational System System Regular Twelve - (SICA) with reports reports information and about the updated analysis monthly once a year produced evolution (2007) on of the evolution cost per of costs student per produced student (2011) (2011) Students registered at courses in the 0 (2007) 25,000 Substituted at the time of secondary vocational schools (2009) AF with following indicator. Enrollment in PEP courses (secondary - 30,000 42,626 171,625 vocational school courses) Note: (2011) (2009) (2013) Substitutes previous indicator. 15. The EEPs included elementary, secondary vocational, and personnel expenditures, and constituted the main part of all expenditures under the SWAp. The DLIs tracked the education secretariat’s implementation of its progress in raising 3rd grade literacy (Proalfa scores), utilization of the secondary school classroom-level testing program (Programa de Avaliação da Aprendizagem Escolar - PAAE), development of the SIMADE management information system, and vocational training program enrollments (PEP). 16. The objectives of the education component were to support expansion of Minas Gerais’ innovations in student learning measurement and integrated management information, as well as a new voucher program to offer state secondary students a range of vocational training options through partnerships with private (enterprises), quasi-public (“Sistema S”) and other public (municipal and federal) providers of skills training. Minas has a tradition of education innovation and the goal of the operation was to help the State expand some of its most promising programs. It is impressive that the programs targeted for support under the SWAp were not only scaled- up with success under the program, but also influenced national education policy. Minas’ early grade literacy program (Alfabetização na Idade Certa) and 3rd grade reading test were 33 the inspiration for the Ministry of Education’s 2012 Programa Nacional de Alfabetização na Idade Certa, PNAIC, and the national 3rd grade literacy test, ANA (Avaliação Nacional de Alfabetização) beginning implementation in 2014. 17. A second testing innovation supported under the program was the PAAE, also a first in Brazil. The PAAE is an online item bank that the Secretariat, working with secondary school teachers, has developed for all major secondary school subjects. The PAAE program allows classroom teachers in 9th grade (the first year of secondary school) to generate their own tests and apply these at the classroom level at the beginning of the school year and again at the end of the school year. It generates confidential, classroom level results for each student that helps teachers tailor their support to different students. Comparing end of year results with students’ scores at the beginning of the year also allows school directors and regional supervisors to compare the performance of different teachers in generating learning improvements (value-added learning gains) for their students at the classroom level. This is a state of the art educational assessment and has become an important formative input for school directors and school system managers. There are no stakes for individual teachers associated with these results, which would have destroyed confidence in, and use of, the system. Instead, school directors use the results for informal feedback to individual teachers and the Secretariat uses the results for training design. 18. Minas’ vocational training voucher program, PEP (Programa do Ensino Profissionalizante) has also influenced national education policy. In 2013, the Rousseff administration launched PRONATEC, which, just like PEP, channels resources to different training providers on a per capita basis, to fund a range of different short-term technical and vocational training course options. Students take this training either concurrently with their secondary school studies, or directly after they complete secondary school. Part of Minas’ innovation was to establish a sound accreditation program to assure the quality of the wide variety of course options and training providers permitted under the program. As of 2014, PEP has effectively been absorbed into PRONATEC, after scaling up rapidly from zero to over 170,000 (cumulative) students between 2007 and 2013. 19. Finally, Minas’ education management information system, SIMADE, is now the leading example in Brazil. SIMADE is a system that incorporates student, teacher, school and regional district information on results (test scores, promotion/repetition) and human resources (classroom and school assignments, promotions, salaries) into a single integrated system that allows calculation of unit costs by school, district, level of education, and program. Transport Eligible Expenditure Programs ProMG-Pleno – Highway Maintenance and Recovery Pro-Acesso Indicator Baseline End Value Status (last (PDO indicators in bold) (year) Target (year year (year) achieved) available) 34 Percentage of the state highway grid 11% 42% 36.7% 32.88% paved with contracts of maintenance (2007) (2011) (2011) Not (2013) Note: and renovation for results achieved 16 target . achieved in 2012 with 44.7% Percentage of municipalities in the State 83% 96% 96.8% 98.9% (2013) with paved access (2007) (2011) (2011) Development of new highway PPPs - PMIs for Substituted at the time of 3 AF with following highway indicator. lots published (2009) Number of electronic speed control 16 (2009) 80 (2011) 125 (2010) 240 (2013) equipments in operation. Note: Substitutes previous indicator. Average percentage delay of Pro-Acesso 37.9% 25% 23.6% 11.77% paving works concluded in the current (2007) (2010) (2010) (2012) year 20. The project’s transport sector supported two large and important road programs in the State: Pro-MG and Pro-Acesso. The Pro-MG rehabilitated and maintained the state paved road network through results-based contracts, and the Pro- Acesso provided surfaced road access to poor, rural municipalities in order to improve their integration into the State and regional economy. In addition, the Project assisted in institutional strengthening activities for the State Roads Department (DER) and the State Secretariat of Transport and Public Works (SETOP) mainly in the areas of traffic safety and transport and logistics planning/management. 21. Starting in 2003, the Pro-Acesso program has provided paved road access in 214 municipalities with a population of 1.3 million people. As a result, out of 853 municipalities in the State, 842 municipalities (98.7 percent) had paved road access to the trunk State highway network by 2014. Out of 11 municipalities that had not yet been concluded, six were delayed mainly due to natural incidents such as large-scale erosions, and five were implemented by the Federal Government (DNIT). The Pro-Acesso demonstrated very fast progress between 2008 and 2010 and most of the works were concluded by 2011. The contracts in 61 municipalities with the value of R$1,183 million were eligible for project financing, which accounts for 30% of the total investment of R$3,910 million. 22. Pro-Acesso contributed not only to reduce transport/logistics costs, but also to improve the living standard of rural poor populations. The impact evaluation 16 5,756km or 36.7 percent of the paved state highway network were under the result-based contracts in 2011. While the value increased to 44.7 percent in 2012, it dropped down to below 40 percent after 2013 because some contracts had not been renewed after the initial 4-year contract period mainly due to the sudden elimination of the part of the gasoline tax dedicated to road infrastructure in 2011. 35 conducted by the client 17 indicated that the municipalities receiving Pro-Acesso works had more rapid growth in economic indicators than ones that already had paved accesses before the Pro-Acesso, resulting in a reduction of economic inequality among municipalities in the State. Furthermore, according to a traffic count survey on several sample roads, traffic volumes doubled after the completion of the works, which indicates that the program stimulated local economic and social activities in rural communities. 23. In total, 7,600km of the State paved road network in 14 (out of 40) sub-regions were rehabilitated and maintained through the 4-year results-based contracts (CREMA) under the Pro-MG program in the last 8 years. The project contributed to the expansion of the Pro-MG between 2007 and 2012. Out of 22 contracts 18, three contracts with the total value of R$375 million were eligible to the Project’s financing. 24. The Pro-MG program enhanced sustainability of the State’s road asset: the proportion of the State road network in good conditions increased from 43 percent in 2007 to 72 percent in 2011, which substantially contributed to reduce transport costs of road users with both freight and passenger vehicles. Furthermore, the Pro-MG also ensured sustainability of the Pro-Acesso program by including newly surfaced roads of Pro-Acesso. However, the progress of the Pro-MG was slow in contrast to the Pro-Acesso, which had strong political support from the state government. 25. The project provided technical assistances to the DER and the SETOP in strengthening institutional capacities mainly on road safety and logistics/transport management. One of the main achievements was the creation of a logistics planning and management unit under the SETOP, which the project assisted through the hiring of consultants. The objective of the unit is to evaluate strategies on the State transport network in consideration of territorial integration and supply chain improvement. The unit has capacity of technical analysis on transport planning and has conducted economic feasibility studies on the concession/PPP roads (MG-050 and ring road), micro traffic simulation, demand analysis of agricultural products, and freight Origin-Destination (OD) survey in the metropolitan area. Furthermore, the unit plans to revise the State highway concession program and update the mid-term road investment plan. The project also financed the feasibility study on the Northern Minas railway to improve mineral transport. These analyses provide a solid grounding for decision-making in transport/logistic policies in the State. 26. The project also supported initiatives on road safety though the Bank did not finance the activities. By 2014, DER has installed 240 speed control units as well as 18 intelligent radars which can automatically read plate numbers, resulting in reduction in traffic accidents by 35 percent on road sections with speed control units. Furthermore, the Bank team assisted the SETOP in conducting the road safety capacity review study funded by the Global Road Safety Facility and in organizing a workshop to disseminate the results. Also the team helped organize a study tour in Spain to learn best practices in this area. 17 Avaliação do Impacto Socioeconômico do Programa de Pavimentação de Ligações e Acessos Rodoviários aos Municípios (PROACESSO), Fundação Joao Pinheiro, September 2013. 18 One contract covers all paved state road network in a CRG. Out of 14, 8 CRGs renewed the first contracts and are implementing the second contracts. 36 Environmental and Social Quality Indicator Baseline End Value Status (last (PDO indicators in bold) (year) Target (year year (year) achieved) available) Average time for the concession of 180 90 days 86 days 93 days environmental licenses for activities with days (2011) (2009) (2013) environmental impact in the state, for (2006) classes 3 and 4 Number of inter-disciplinary teams of 12 16 16 (2008) 19 (2013) indigenous health funded by SES (2007) (2011) Number of certified teachers in the Curso 137 212 213 (2008) 110 (2013) de Magistério Indígena (2007) (2011) 27. As in the other sectors, activities under the SWAp component in the environmental and social quality sector were concluded in 2011, with all three indicators achieved. The environmental DLI called for concession of environmental licenses for two classes of processes to be halved to 90 days primarily through the use of IT technology. This had been achieved by 2009, and was maintained through 2011. However, by 2013, the duration had increased to 93 days, slightly over the initially established target. 28. The number of certified teachers in indigenous areas was increased from 137 to 213 during project life. This led to greater empowerment of indigenous communities and contributed to a rise in the level of indigenous education through the creation of the Intercultural Training of Indigenous Teachers at the Federal University of Minas Gerais. On indigenous health, increasing the number of inter-disciplinary indigenous health teams from 12 to 19 within the project’s duration and surpassing the final target. This contributed to increase the quality of the service of Primary Indigenous Health and Care in the State. Poverty Reduction Eligible Expenditure Program Rural Poverty Reduction Program Indicator Baseline End Value Status (last (PDO indicators in bold) (year) Target (year year (year) achieved) available) Number of families benefited by the 0 (2009) 16,800 21,900 2,400 (2013) [Rural Poverty Reduction] project starting (2011) (2010) in 2010 (per year) Intensive monitoring of Rural Poverty 0 (2009) 6 (2011) 6 (2010) 3 (2013) Reduction Project (number of monitoring meetings) 37 Improvement of data on poverty in Minas Due to difficulties in measuring improvements, this Gerais indictor was discarded by agreement between the World Bank and the Government of Minas Gerais. However, a household survey for the Great North Region was supported and report on poverty data for this region was prepared, feeding the policy making process. Number of secondary schools in the Great 59 (2009) 90 (2010) 90 (2010) Employment North that offer initial work preparation Program courses for the assembling and discontinued maintenance of computers in 2011. Proportion of live births from mothers 46.1% 48% 57.8% 66.6% with 7 or more pre-natal consultations in (2007) (2011) (2010) (2013) the Great North of Minas Gerais 29. The original project addressed poverty reduction in an indirect manner, by supporting measures expected to promote growth through improvements in public sector management, the business environment, and the quality of public service delivery. The Addition Financing provided additional direct support to the Government’s poverty reduction strategy by supporting the application of the results-based management approach to programs aimed at poverty reduction, and including the Rural Poverty Reduction Project (RPRP) as a new EEP. 30. Three DLIs supported the government’s Rural Poverty Reduction Program. The first was the number of families to be benefitted by the program, which was not only achieved before the end target date, but also surpassed. A second DLI called for intensive monitoring of the RPRP by SEPLAG and the Instituto de Desenvolvimento do Norte e Nordeste de Minas (IDENE). This was achieved by holding a total of six monitoring meetings a year in 2010 and 2011. The third DLI also targeted the impoverished Great North Region of the State. The Bank had assisted the State in the formulation of a Household Survey, the first being carried out in 2009. The 2010 target called for publishing the results of this Survey and preparing the 2011 survey. In 2011 a report was prepared which analyzed the statewide and Great North data and contributed to improved policy design and monitoring. 31. The two additional poverty DLIs were related to the health and education sectors. One called for the State Secretariat of Education to develop 13 courses for initial job training in the area of information technology. The courses lasted from 40 to 60 hours and were offered to secondary school students. The target for 2010 for this indicator (90 schools in the Great North offering initial work preparation courses for the assembling and maintenance of computers) was achieved, but the Program was discontinued in 2011. The final DLI measured the proportion of live births from mothers with seven or more pre-natal consultations in the Great North. The target for this indicator was not only achieved, but the proportion of live births has increased from 46.1 percent in 2007 to 66.6 percent in 2013. As a comparison, the proportion of live births in the State of Minas Gerais was 61.4 percent in 2007 and 71.5 percent in 2013. 38 2. Technical Assistance Component 32. Technical Assistance (TA) activities were introduced to help achieve the stated management for results objectives. Those activities included strengthening procurement and improving the knowledge base on which sector strategies, budgets and targets were based. Some of the activities carried out were studies including a household survey, impact evaluations in education, health and transport, a quality assurance survey, and a study of output indicators and links to outcomes (see list below for all TA activities). 33. The TA activities were directly linked to support the achievements of DLIs. For instance, a study measuring the quality of service provision to businesses can be directly linked to achievement of the DLI that measured the time necessary to open a business in the State capital. Similarly, an assessment of the state capacity to handle road safety and the identification of a set of actions to improve capacity informed the efforts of the GoMG toward the achievement of DLIs in the transport sector. 34. In the transport sector, the project provided technical assistances to the DER and the SETOP in strengthening institutional capacities mainly on road safety and logistics/transport management. One of the main achievements was the creation of a logistics planning and management unit under the SETOP, which the project assisted in through the hiring of consultants. The objective of the unit is to evaluate strategies on the State transport network in consideration of territorial integration and supply chain improvement. The unit has capacity of technical analysis on transport planning and has conducted economic feasibility studies on the concession/PPP roads (MG-050 and ring road), micro traffic simulation, demand analysis of agricultural products, and freight Origin-Destination (OD) survey in the metropolitan area. Furthermore, the unit plans to revise the State highway concession program and update the mid-term road investment plan. The project also financed the feasibility study on the Northern Minas railway to improve mineral transport. These analyses provide a solid grounding for decision-making in transport/logistic policies in the State. 35. The Economic Master Plan for the Metropolitan Area of Belo Horizonte has resulted in being a policy guiding document. The purpose of the Economic Master Plan, prepared by a consultancy firm hired through Technical Assistance, was to serve as a guide for the development of the Metropolitan Region of Belo Horizonte in the next 20 years. The Plan’s objective is to improve the State’s ability to attract foreign investments to the region in the medium and long term. This plan is now the basis for an integral strategy of economic and regional development for the Metropolitan Area of Belo Horizonte, streamlining economic and social needs and the required infrastructure for a sustainable economic and social development; it also integrates the participation of state and municipal level administrations. 36. The State Secretariat of Health prioritized use of the TA funds to finance the process of change management in the implementation of electronic records systems in two areas: (a) family health and (b) urgent and emergency health care. Specifically, an Organizational Change Management plan resulted in the successful implementation of a Hospital Management System in the urgent and emergency care units of 19 selected hospitals. Entity Project IDENE Rural Poverty Reduction Program 39 Roads Structuring Project related to access to the Administrative Center of the SEDE Government of Minas Gerais, its surrounding areas and their areas of influence SEDE Economic Master Plan for the Metropolitan Region of Belo Horizonte SEDE ExportaMinas Central Projects – Minas Gerais Strategy for Foreign Trade SEDE Best Practice Guide for Structuring an Independent Verifier SEDE ExportaMinas Central Projects – “Perishables” Project SEDE Studies on regulation of PPP concession contracts Training, courses and development of best practices in regulation for civil servants SEDE of the State of Minas Gerais SEDE ExportaMinas Central Projects – “Food Clusters” Project ExportaMinas Central Projects – Coaching on Exporting to the Middle East and SEDE North Africa Region (MENA Project) SEDE Study of Railway Infrastructure Feasibility in the Great North of Minas Gerais SEE Study on the quality of education in schools in the State of Minas Gerais SEMAD Implementation of the mobile SISEMA module SEMAD Capacity building (management of SISEMA) through distance education Development of a project portfolio for the State of Minas Gerais under the SEPLAG Descomplicar Structural Program SEPLAG Procurement Planning Project SEPLAG PAD-MG (Pesquisa por Amostra Domiciliar) SEPLAG Project for the Implementation of a Real Estate Management System SEPLAG Establishment of procedures for the operation of the State IT policy Procurement of 8 High Capacity Processing Servers to meet the expansion of SEPLAG services provided to State entrepreneurs by Minas Fácil (Descomplicar Structural Program). Evaluation of the Results-based Management in the Government of the State of SEPLAG Mians Gerais Development of the Inventory Management Model for the Public Administration SEPLAG of Minas Gerais Implementation and Technical Solutions Plan for the Governmental Management SEPLAG Vision Integrated System – GRP MINAS SEPLAG Training on Socio-Economic Assessment of Projects SEPLAG Technical assistance on the strategic planning for the Delegacia Virtual – PCMG SES Project for the SES/MG Change Management Team Implementation of the Office of Strategic Planning for Transport Logistics SETOP (Escritório do Plano Estratégico de Logística de Transportes – EPELT) 40 Annex III. Economic and Financial Analysis 1. No comprehensive economic assessment was undertaken at preparation. The original Project Appraisal Document (PAD) states, that as a multi-sectorial operation supporting government programs and activities in different areas and of different natures, a comprehensive economic assessment did not apply and therefore was not undertaken. However, in the cases of education and transport a more detailed ex-ante economic analysis was carried out and included in the original PAD. 2. Although economic rates of return are difficult to compute in public sector reform projects, efficiency gains are likely to have occurred in the areas targeted by the project components. At project completion, due to a lack of information no extensive ex-post economic assessment was carried out to measure whether the assumptions of the ex-ante analysis had materialized. The effectiveness of the interventions is difficult to assess due to the lack of a true counterfactual. However, drawing on different sources and methodologies (e.g., estimating the counterfactual by averaging pre-intervention data), and widening the scope to additional sectors, evidence suggests that the project had an overall positive impact. 3. A recent World Bank study 19 on results-based management in several Brazilian states shows improvements in several sectors. An analysis comparing current with pre-intervention outcomes in education, health, and the security sector in Minas and other Brazilian States was carried out by a World Bank team in late 2013. The analysis examined changes in relation with regional trends to determine if the improvements depart in meaningful ways from the overall trend. In addition, the study applied a truncated time- series cross section to control for a number of additional factors influencing service delivery outcomes in the examined sectors. Education Sector 4. In the original PAD, two kinds of empirical analysis were conducted for the education sector. First, an exercise to calculate rates of return to each additional year of education in the State of Minas Gerais indicated a rate of return of 4.46 percent for the Primary level (Grades 1 to 8), 10.3 percent for the Secondary level (Grades 9 to 11), and 19.74 percent for the tertiary level. Second, a production function of test scores using data from Portuguese and Mathematics test scores aggregated at the municipal level was used to provide insight into the determinants of learning achievement in Minas Gerais. Results pointed to the importance of socio-economic conditions, indicating how social inequalities were reproduced by the existing educational system. 5. For the education sector, the above-mentioned World Bank study finds that positive results. Especially after the introduction of the second-level agreements in 2007, Minas Gerais recorded a statistically significant improvement in the IDEB score from 3.40 in 2007 to 3.60 in 2010. A quasi-experimental before-and-after comparison of outcomes in Health, Public Security, and Education revealed that when comparing Minas Gerais to 19 Viñuela and Zoratto (2013): Performance Gains in Results-Based Management in Brazilian States, Report No. 82592-BR. 41 other states in the Southeastern region only improvements in education outcomes go beyond the general trend (see Annex 10 for details). Transportation Sector 6. Cost-benefit Analysis in Transportation Component in original PAD. For transport, a cost-benefit analysis was conducted to evaluate the two following programs: 1) paved highways rehabilitation and maintenance under the Pro MG-Pleno program, and 2) road surfacing under the Pro-Acesso program. The net present value (NPV) of investments under the Pro MG-Pleno program was calculated as R$2.245 billion at a 12 percent discount rate over 20 years evaluation period. The internal rate of return (IRR) of the program was estimated at 35.60 percent. The road surfacing component under the Pro- Acesso program was estimated to have a NPV of R$998 million at a 12 percent discount rate and yield an IRR of 26.42 percent. A sensitivity analysis switching values under different scenarios showed that the economic returns of the Pro-Acesso and Pro-MG Pleno programs were robust. 7. The ex-post economic evaluation of the both Pro-Acesso and Pro-MG was conducted through a simplified way due to the limited data availability, but it confirmed that the both programs remain economically viable. While the ex-ante economic evaluation model was developed with the HDM-4, software commonly used for appraisal of Bank-financed road projects, most of data required to update to the ex-post model are not available. Accordingly, the posteriori economic viability was examined through assessing the two most important factors; traffic volume and costs, which significantly affect the economic indicators. 8. The ex-post evaluation for Pro-MG was conducted for the five regions (CRG)20 of the total extension of 1,825km. The total cost spent for the first four-year CREMA contracts in these 5 regions was R$222 million (excluding inflation adjustments), which is 1.8 percent higher than the cost estimations used for the ex-ante HDM-model. Meanwhile, based on the traffic count data 21, the annual traffic growth is 6 percent on average, which is twice as high as the assumption used in the ex-ante evaluation (3 percent). Assuming these cost and traffic growths in all regions, it is confirmed that IRR and NPV are better than those 22 in the ex-ante evaluation 23. 20 CRG1, 3, 16, 20 and 24. 21 Available for only two regions: CRG4 and 17, both of which were eligible to Bank financing. The traffic count surveys were conducted in 2009 and 2014. 22 The IRR and NPV at appraisal were 38.77 percent and R$423million (a 12 percent discount rate) for the first 5 regions. 23 The ex-ante evaluation was conducted for 9 categories of roads with different pavement conditions and traffic levels. For example, the category with the lowest IRR among all categories at appraisal was a group of roads with medium traffic (730-1,700 per day) and good pavement conditions before the interventions. It’s IRR has been improved from 11.54 to 15.62 percent after applying the revised cost (1.8 percent higher) and annual traffic growth (6 percent). Due to the similarity of the model, applying the same cost and traffic growth rates, the results of all other categories will be better than the original ones. 42 9. Ex-post evaluation for Pro-Acesso. While the unit costs for civil works are on average 9.2 percent more expensive 24 than ones used for the ex-ante evaluation after adjusting inflation 25, the traffic volume has increased much more rapidly than expected at appraisal. Based on the traffic count data on the three sections under the Pro-Acesso, the traffic volume is on average 90 percentage higher than estimated in the ex-ante model. Assuming that the same cost and traffic growth rates are applied to all Pro-Acesso roads, the ex-post IRR and NPV of the Pro-Acceso program are further improved compared to the ex-ante evaluation 2627. Health Sector 10. In the health sector, no assessment was undertaken in the original PAD, evidence suggests a positive impact in the health sector. The 2013 World Bank study finds no discontinuity in the trend that could be linked to the introduction of performance agreements that was supported by the loan. While the rate decreased by 1.2 points between 2007 and 2010, child mortality levels remain above the regional average and the distance between the two has been fairly constant. Minas Gerais’ trend is similar to that of the Southeast region and that of Rio de Janeiro, a State not using performance-related pay in the health sector. Other impacts 11. Positive Behavioral Changes. Despite those concrete improvements, it should be mentioned that other outcomes of the project such as the introduction of a “culture of achieving results” among civil servants are likely to have occurred but are difficult to measure quantitatively. To capture civil servants’ perceptions of the management reforms, a survey was conducted among Minas’ civil servants. The majority of respondents believe that the introduction of results agreements helped them better understand their role and objectives at work (see Annex X). 12. High overall efficiency. To sum up, conducting a rigorous economic analysis is not viable due to the lack of an appropriate counterfactual. However, considering the achievement of most of the PDO and Intermediate indicators, the overall efficiency of the project can be considered rather high. 24 Based on the comparison of average unit costs for civil works used for the ex-ante evaluation with the actual contract values with adjustment of inflation. 25 Índice Nacional de Custo da Construçăo (INCC) was used to eliminate inflation effects from the costs. 26 IRR and NPV of the ex-ante evaluation were 26.42 percent and R$998 million at a 12 percent discount rate respectively for the 90 road sections of the Pro-Acesso. 27 As an example, one road section in Santanta Deserto, which had the lowest IRR of 10.4 percent among all road sections at appraisal, was re-assessed with applying the revised cost (9 percent higher) and traffic (90 percent higher). The updated IRR and NPV are 14.1 percent and R$0.67 million which are better than the ex-ante evaluation. Due to the similarity of the model, applying the same cost and traffic growth rates, the results of all other sections will be better than the original ones. 43 Annex IV. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Names Title Unit Lending Fernando Andres Blanco Cossio Senior Economist AFTP4 Pablo Fajnzylber Sector Manager, PREM AFTP2 Jose L. Guasch Consultant LCSPF Gerard Martin La Forgia Lead Health Specialist EASHH Aymeric-Albin Meyer Operations Adviser OPSPQ Isabella Micali Drossos Senior Counsel LEGES John Lincoln Newman Lead Poverty Specialist SASEP Pedro Olinto Senior Economist PRMPR Alexandre Borges de Oliveira Senior Procurement Specialist EASR1 Zelia Brandt de Oliveira Program Assistant LCC5C Suhas D. Parandekar Senior Education Economist EASHE Chris Parel Consultant LCSPS Sr Financial Management Rajeev Kumar Swami ECSO3 Specialist Carlos E. Velez Lead Economist LCSUW-HIS Ethan Weisman Lead Economist and Sector Lead LCSPR Deborah L. Wetzel Country Director LCC5C Christine de Mariz Rozeira Senior Economist LCSPS Supervision/ICR Joao Pedro Wagner De Azevedo Senior Economist ECSP3 Fernando Andres Blanco Cossio Senior Economist AFTP4 Edward William Bresnyan Senior Rural Development Speci LCSAR Barbara Bruns Lead Education Economist LCSHE Flavia F. Carbonari de Almeida Consultant LCSSO Flavio Chaves Natural Resources Mgmt. Spec. AFTN3 Roland N. Clarke Program Leader LCC5C Regis Thomas Cunningham Sr Financial Management Specialist EASFM William R. Dillinger Consultant LCSPS David Evans Senior Economist AFRCE Estanislao Gacitua-Mario Lead Social Development Specialist LCSSO Flavia Nahmias da Silva Gomes Program Assistant LCC5C Jose L. Guasch Consultant LCSPF Till J. Hartmann Consultant LCSPS Eric R. Lancelot Sr Transport. Engr. LCSTR Bernadete Lange Senior Environmental Specialist LCSEN Andre C. Medici Sr Economist (Health) LCSHH Aymeric-Albin Meyer Operations Adviser OPSPQ Isabella Micali Drossos Senior Counsel LEGES John Lincoln Newman Lead Poverty Specialist SASEP 44 Alberto Ninio Chief Counsel LEGEN Satoshi Ogita Transport Specialist LCSTR Pedro Olinto Senior Economist PRMPR Alexandre Borges de Oliveira Senior Procurement Specialist EASR1 Zelia Brandt de Oliveira Program Assistant LCC5C Jason Jacques Paiement Social Development Specialist LCSSO Chris Parel Consultant LCSPS Ezau Pontes Senior Health Specialist LCSHS Angela Nieves Marques Porto Consultant LCSPS Paul Procee Lead Urban Specialist EASCS Jose Guilherme Reis Country Sector Coordinator ECSPF Jean-Claude Sallier Consultant LCSTR Graciela Sanchez Senior Development Specialist LCSSO Katherine M. Scott Senior Economist LCSPP Tarsila Ortenzio Velloso Public Sector Specialist LCSPR Adrien J. Veron Consultant LCSTR Elisabet Vila Jorda Junior Professional Associate LCSTR Fanny Weiner Public Sector Management Specialist LCSPS Deborah L. Wetzel Country Director LCC5C Luciano Wuerzius Procurement Specialist LCSPT Laure de Castro Zoratto Economist LCSPS (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY08 99.52 862.29 Total: 99.52 862.29 Supervision/ICR FY09 89.92 524.00 FY10 56.03 232.77 FY11 37.22 184.73 FY12 9.18 73.44 FY13 11.77 56.29 FY14 6.23 65.24 FY15 12.43 93.80 Total: 222.78 1,230.30 45 Annex V. Beneficiary Survey Results See Annex X for the results of the survey that was conducted among public servants in Minas Gerais directly affected by the implementation of results agreements. The survey was carried out in mid-December 2014 among public servants to capture their perception resulting from the introduction and expansion of the results-based public management system since 2003. Annex VI. Stakeholder Workshop Report and Results N/A 46 Annex VII. Summary of Borrower's ICR and/or Comments on Draft ICR 1. This Annex summarizes the full version of the Borrower’s Completion Report which was made available to the Bank on October 17, 2014 and was 91 pages in length. The full version (in Portuguese) can be accessed online. 2. The Borrower’s report is divided into five sections. The first section provides a brief introduction to the project and its context. The second section presents the objectives, design, components, stakeholders, and indicators of the project. Subsequently, the third section deals with the implementation of the operation. This section includes a detailed account of eligible expenditure programs (EEPs), technical assistance (TA), disbursements, and safeguards. Section four displays the results of the operation by presenting achievements in the monitored areas. The results show a significant improvement of most of the indicators monitored. With regard to the observed improvements, it is highlighted that no evidence exists that links the improvement of the indicators with the operation. However, it is believed that the World Bank’s support had a positive impact on the performance of the indicators monitored. The last section provides an evaluation of the Borrower’s own performance as well as an evaluation of the performance of the Bank. This summary focuses mainly on the last section of the borrower’s ICR because its content is rather evaluative, as opposed to descriptive, and therefore more relevant for the Bank’s ICR. Evaluation of the Borrower’s own performance 3. The Borrower’s evaluation of its own performance is primarily based on interviews with public servants directly involved in the operation. Public servants stated that the Bank provided the operation with a stamp of legitimacy which was vital to cope with internal opposition. In addition, the Borrower benefited from the Bank’s thorough monitoring of indicators in that it helped create a new organizational culture focused on achieving results. 4. Technical assistance generated valuable knowledge. Another important condition that had a positive impact on the Borrower’s performance was the high amount of financial resources reserved for contracting technical assistance. It allowed the State to invest in the acquisition of knowledge in a way that would not have been possible without the Loan. Additionally, the experience with coordinating such a large operation and the frequent discussions between public servants and Bank staff generated knowledge that could also be applied in future operations. 5. However, the government encountered difficulties in defining which activities to include. These difficulties eventually contributed to the delay in implementing the TA component. For future operations involving TA, it is believed that a more centralized communication between the State and the Bank would be more effective. 6. In general, however, the TA component, despite making up a small part of the resources available, was considered highly relevant to the success of the operation. The contact with the hired consultants generated a strong learning experience for the teams involved. The Bank’s contracting practice, i.e. placing greater emphasis on skills than on price, helped recruiting highly qualified personnel capable of leveraging their skills to 47 public servants. Several projects have resulted in the development of innovative processes that have been effectively incorporated into routine public sector practices. 7. Important lessons learned emerged from the TA component. It was found that the success of the TA depends on clearly defining the goals of the activities. Furthermore, it is crucial that the team is willing and able to adapt, replicate and improve in order to internalize the knowledge produced by the consultants. Finally, a more thorough and systematic evaluation of the TA portfolio should have been conducted to eliminate those projects that have the highest risk of failure and discontinuity. 8. At the same time, the size of the operation and the high number of technical personnel involved generated inefficiencies and difficulties. Especially at the beginning, auditing practices of the Bank had to be aligned with those of the State, and there was a lack of clarity about financial reporting and responsibilities. Those difficulties were exacerbated by two critical factors. One factor was the high turnover of technical personnel involved on the Borrower side. The other was the missing of a structured knowledge management process for the State Government. As a consequence, in several cases, outgoing Government staff did not pass on knowledge about the operation adequately to their successors, creating errors and inefficiencies. As a case in point, the report mentions the Tribunal de Contas do Estado (TCE). The TCE’s audit process was significantly hampered by the high turnover rate of State employees. In addition, the report laments a delay between the start of the execution of the project and the start of the audit. 9. Design issues recognized by the Borrower included misalignment of incentives. The report highlights the fact that, in some cases, financial incentives were not targeting the sector teams that were actually responsible for achieving certain pre-determined goals. For example, the Health Secretary would receive financial compensation for fulfilling indigenous safeguards despite the fact that there are only few opportunities to implement indigenous safeguards in the health sector. 10. In sum, the Borrower’s performance would have benefited from improved knowledge management processes that are particularly important in an operation of this size and length. A smaller operation over a shorter time span that focuses on only a few areas might generate more profound results and suffer less of the difficulties associated with high staff turnover. Evaluation of the performance of the Bank 11. The feedback of the surveyed public servants on the Bank’s contribution to the Second Partnership Project with Minas Gerais is generally very positive. The respondents indicated that the Bank’s team was very well qualified and extremely proactive in finding solutions to the difficulties encountered during preparation and implementation. It was noted that Bank staff added value by bringing in international experience and best practices. Furthermore, it was stated that the Bank was very helpful in refining project ideas and providing a critical eye from a broader perspective. 12. Despite the generally fruitful cooperation, some initial difficulties occurred. For example, financial experts from the Bank were not used to the rules of Brazilian public accounting. However, those difficulties were overcome and successfully addressed by the Third Partnership Program (DPL II). Another area that caused some confusion concerned 48 the Eligible Expenditure Programs (EEPs). The Borrower was unsure about which expenditures counted towards the EEPs, which resulted in delays in the submission of the reports. With regard to the TA, the report states that, at times, it took the Bank an unnecessary long time to send letters of no objection, delaying the hiring of consultants. Another issue with the TA was that the Bank often criticized the design of activities without giving concrete indications for improvement, thus further delaying approval processes. 13. All in all, it shall be noted that the performance of the Bank was considered to be highly satisfactory by the Borrower, and that the difficulties encountered had only a punctual character. Sustainability and continuation of EEPs 14. The sustainability of the improvements is key to evaluating the success of the operation. Therefore, the Borrower’s report includes a table listing the EEPs supported under the SWAp operation that were kept in the Plano Plurianual de Ação Governamental (PPAG) for the period 2012-2015. The table shows that the EEPs were continued in the PPAG, even though in many cases under different names. For example, the “Network of Technological Innovation” program became the “Technology and Innovation towards a Knowledge Economy” program. The PPAG clearly reflects the continuity of programs initially implemented under the Minas Gerais Second Partnership Program. 49 Annex VIII. Comments of Cofinanciers and Other Partners/Stakeholders N/A 50 Annex IX. List of Supporting Documents Government of Minas Gerais (2014): Relatório de Conclusão (Borrower’s Completion Report). Draft. Joao Filocre (Undated) Mais tempo para ensinar, mais tempo para aprender. Blog. Viñuela, L and Zoratto, L. (2013): Performance Gains in Results-Based Management in Brazilian States. Report No. 82592-BR. World Bank (2008): Country Partnership Strategy Brazil 2008-2011. Report N. 42677-BR. World Bank (2008): Project Appraisal Document for a Second Minas Gerais Development Partnership Project. Report No. 43232-BR. World Bank (2008-2014): SWAp II Project Supervision Reports (ISRs). World Bank (2010): Project Paper on a Proposed Restructuring and a Proposed Additional Loan for the Second Minas Gerais Development Partnership Project. Report No. 53292-BR. World Bank (2012): Program Document for the Third Minas Gerais Development Partnership. Report No. 62267-BR. 51 Annex X: An Overview of Minas Gerais’ Management Reforms 1. This annex provides a more in-depth look into the management reforms implemented by the State since 2003, and the role of the Bank in making it possible. The analysis here presented follow three complementary methodologies: i) interviews with selected government officials; ii) data analysis, and iii) a perception survey with Minas’ public sector employees regarding the introduction of performance agreements. Management reforms carried out by Minas involved profound and multi-dimensional institutional changes which have evolved over time; as such, the team does not expect to draw any definite conclusion regarding its results, but simply to shed light into a couple of aspects that emerged from the interviews, survey and data analysis. All of these methodologies are limited in one way or another, and such limitations are highlighted in the respective sections. 2. Following this logic, this Annex is divided in three sections: Section (I) presents an overview of Minas’ performance agreements with a focus on the education sector, based on interviews with selected government officials. Section (II) presents a comparison of results in the health, education and citizen security sectors achieved by Minas vis- à-vis other Brazilian states. In section (III) we share the results from the questionnaire sent to Minas’ civil servants in December 2014 regarding their perception of those reforms. 3. While the data analysis points to a positive effect of results agreements on service delivery outcomes – particularly in Education - it is important to note that their adoption has not been implemented in isolation. Fiscal reforms and efforts to strengthen planning and budgeting functions were important preconditions for their adoption. In the education and health sector, the introduction of common metrics and monitoring systems capturing the relative performance of every school/hospital, municipality, and state has facilitated the measurement of results and distribution of rewards. These common metrics have not only been effective in providing a basis for comparison, but also have been important in motivating reform. Minas has also taken other complementary steps, such as creating special careers for planning and management professionals with a differentiated pay scale. In some cases, these specialists have been deployed in front line units where performance is being measured to provide technical assistance and support managers. In general, the survey of Minas’ staff is indicative of a positive perception of the introduction of the results-based management model. Results also indicate that the second-stage results agreements brought more improvements in the way staff understand their role and objectives at work than the first-stage RAs. A qualitative analysis of the open question answers shed light on the main critiques raised by staff: wages remained low despite the productivity bonus, RAs added additional bureaucracy to daily work, some targets were set low on purpose and, in some cases, staff believe they were already working towards targets and as such the RAs were not necessary. 4. To some extent, these critiques reflect lessons learned raised by Minas’ government officials themselves. They acknowledge that the management reform became too big to be manageable: internal political pressures to expand the RAs to all secretariats brought additional challenges to the process, such as having to create and negotiate targets for results areas that are not as measurable (e.g. culture) as others (e.g. education). Some officials believe that, in order to be effective, the reforms should focus in a selected group 52 of sectors considered of high-priority. This what the path followed by Pernambuco, for example, which focuses on the health, education, and citizen security sectors. I) The Performance Agreements in Minas Gerais - the case of Education 28 Background 5. Over the last eleven years, the Government of Minas Gerais has introduced a wide range of reforms in the way the State is managed. While the details have evolved over time (and continue to evolve) one core element of the reform has been the introduction of results agreements (acordos de resultados); signed agreements between the Governor and the heads of each state secretariat setting out specific results to be achieved in the year ahead, as well as specific programs and management reforms to be implemented. Since 2008, these have expanded to include similar agreements between each secretariat and its subordinate departments and implementing organizations (schools, hospitals, etc.). 6. The reform process began in response to a desperate fiscal situation. At the beginning of the last decade, Minas found itself in a difficult fiscal situation. In 2002, the state was running an overall deficit equal to three percent of revenues. Long-term debt totaled R$35.3 billion29 or twice gross revenues and funds for capital investment had disappeared. Although 2002 was an election year, expenditures on investment totaled only R$755 million, or four percent of total expenditure. 7. A new governor, Aécio Neves, was elected in 2002. Shortly after taking office, he set about putting the State’s fiscal situation in order, cutting non-salary recurrent costs, abolishing three thousand appointive positions, and centralizing the management of the wage bill in order to improve control over payments and reduce fraud. To increase revenues, he stepped up enforcement of the state’s principal revenue source: the value added tax (ICMS). 8. Since then, the fiscal results have been impressive. Neves was reelected by an overwhelming majority of 80 percent in 2006. Although he resigned to run for the Federal Senate in 2010, he was replaced by his vice governor, who was then elected in his own right shortly thereafter. As shown in Figure 1, the state’s tax revenues have nearly doubled in real terms since 2002. While this in part reflects a growing economy-- the state’s GDP grew by 90 percent over the same period—it also reflects an increasingly effective system of tax administration. Tax revenues were equal to 8.7 percent of GDP in 2002. That ratio increased to 9.3 percent of GDP in 2005 and has remained roughly at that level ever since. In absolute real terms, tax revenues dipped after the onset of the global economic crisis in 2009 but then made a swift recovery. 28 This section was written by William Dillinger (consultant) based on interviews with government officials in September 2014. 29 Secretaria Estadual da Fazenda, Balanco Orçamentário, 2002 53 Management Reforms 9. The newly-elected administration in 2002 recognized that fiscal adjustment alone would not solve the problems and launched a major program of management reforms, called ‘management shock’ (choque de gestão). The first phase consisted of: (i) defining a set of priority projects and programs (the list of the original priority projects is shown at the end of this annex), (ii) setting targets for their implementation, and (iii) monitoring them closely. Inter alia, the Governor met every month with the heads of the responsible secretariats to review progress and discuss problems with each activity. The list of the original priority projects is shown at the end of this annex. It demonstrates a wide range of activities across a variety of sectors. Three were reportedly ‘top priority’: construction of the new government administrative center, renovation of the central historic square in Belo Horizonte (Praça da Liberdade), and a program aimed at paving access roads to remote municipalities. Spending on all 31 priority projects constituted seven percent of total state expenditures in 2007. 30 This was targeted to increase to 9.5 percent of the total by 2010. 10. Another element of the reform was the introduction of results agreements (acordos de resultado). These set out targets to be achieved by certain state organizations and provided for salary bonuses if the targets were achieved. According to interviews with current and former government officials, the reform started small, with only the Secretariat of Finance and the Forestry Institute signing agreements in 2004; both organizations raising their own revenues. Salary bonuses were granted on the basis of the revenue increases achieved by each organization. In 2005/2006, the State extended the reform to the Secretariat of Planning (SEPLAG) and State Attorney’s Office (AGE) with rewards based on reductions in expenditures. The reform was then rolled out to all state secretariats in 2008. This expansion was prompted by the desire of other state secretariats to give performance bonuses to their own employees. As discussed below, performance targeting turned out to be more suited to some secretariats (e.g., education) than others (e.g. culture). 11. First Level Agreements. The 2008 reforms also brought a major expansion in the content and coverage of the results agreements (Acordos). The Acordos now had two levels. The first level was between the governor and the managers of the organizations within each of 26 sector secretariats. (Although these agreements are often referred to as taking place between the governor and each secretariat, the agreements in fact consolidate all the organizations within a given sector (‘system’, in Minas parlance). Thus the agreement with the health system was signed by the Secretary of Health, but also the director of FHEMIG (Fundação Hospitalar do Estado de Minas Gerais) (which manages the state hospitals and has the same organizational rank as a secretariat), the Director of the Hematology Institute, and so forth. The second level was an agreement between the manager of each secretariat, institute, foundation, etc. and its subordinate departments. Thus FHEMIG has second level agreements with each of its hospitals; the Secretariat of Education has agreements with each state school, etc. 30 Source: World Bank. 2010. Additional Financing - Second Minas Gerais Development Partnership Project Appraisal Document 54 12. This section focuses on the Agreement with the education sector. This is partly because education is a key function of the state, and a sector that lends itself to performance measurement: a desired outcome (performance levels on standardized tests) can be measured and a relatively direct line of causation can be drawn from the actions of the State to that outcome. 13. Outcome Indicators (Indicadores Finalísticos). The 2008 Acordo with the education system had three parts. The first consisted of a table of outcome indicators and targets (indicadores finalísticos e metas). The 2008 Acordo specified 14 of them: • the proportion of third graders meeting recommended levels of literacy; • the average proficiency score for literacy in the third grade of ensino fundamental (1st to 9th grade, equivalent to 6 to 14 year old) (EF); • the average proficiency scores for Portuguese and mathematics in the fifth and ninth grades of ensino fundamental and in the third grade of ensino médio (15 to 17 years old) (EM); • the level of age/grade distortion in ensino fundamental and ensino médio; and • the graduation rates for ensino fundamental and ensino médio. The Acordo also includes separate targets for age/grade distortions in the northern part of Minas. Table 1: Educational Achievement Targets for Educational Achievement (1st Level Agreement of 2008) Year Grade Level 2007 2008 2009 Average Proficiency Scores Literacy 3rd EF 536 558 590 th 5 EF 195 202 209 9th EF 241 247 253 rd Portuguese 3 EM 274 282 291 5th EF 205 210 216 9th EF 251 257 264 rd Math 3 EM 282 292 303 Percent of Students Meeting Recommended Levels Literacy 3rd EF 66 73 81 Age-Grade Distortion Rate Statewide EF 25 21 17 EM 39 35 29 Age-Grade Distortion Rate Northern Minas EF 28 25 19 EM 48 53 58 Graduation Rates EF 63 67 71 EM 48 53 58 EF: ensino fundamental EM: ensino medio 55 Note: Proficiency is measured on the basis of standardized tests designed by the national institute for educational research (Instituto Nacional de Estudos e Pesquisas Educacionais - INEP) and reported in the system of educational statistics (Sistema de Avaliacao de Educacao Basica - SAEB). Scales range by subject and grade. For the ninth grade of ensino fundamental, for example, the scale for Portuguese ranges from zero to 350. The scale for math ranges from zero to 300. Age-grade distortion is defined as the percentage of students who are more than one year behind the grade appropriate for their age. Source: Ministry of Education of Minas Gerais 14. As shown in Table 1, the Acordo called for modest but steady progress in most of the indicators. The Agreement was methodical in setting out exactly how progress against each target was to be measured, including the sources of data that were to be used. For instance, the proficiency of third-graders in literacy was to be based on the state’s annual PROALFA literacy test. Average scores in Portuguese and mathematics were to be based on the annual state PROEB test. Graduation rates were to be taken from the annual school census (censo escolar). At the outset, the achievement of each target was measured in binary terms. No credit was given for coming close. Either the target was met or it was not. 15. To consolidate the indicators into a single rating for this component of the Acordo, each of the indicadores finalísticos was given a weight. The indicator ‘percent of third graders reaching the recommended level of literacy was given the largest weight with 18 percent. The reading proficiency score for third graders in ensino fundamental had a weight of ten percent, while all the other targets had weights of six percent. 16. Priority Projects (Projetos Estruturadores). The second part of the 2008 Acordo with the education system required progress on the following eight priority projects and programs. • Acelerar Para Vencer, a program aimed, in this case, at reducing repetition rates in northern Minas and other poorer parts of the state; • SIMAVE, an electronic reference library for teachers; • expansion of full-day education to additional schools (as opposed to the current prevailing practice of short days with double shifts); • introduction of ‘new standards of management for basic education’ which in turn included a program entitled ‘improvements in physical infrastructure, furnishings, and educational facilities in basic education’; • Projeto Escola Viva/Comunidade Ativa, a program which encourages schools in underprivileged neighborhoods to offer courses to the community on weekends (According to its website, the state school Celso Machado, for example, offers courses in computers, taekwondo, dance, and residential electrical repair every Saturday); • Promedio, a program intended to make ensino médio (grades 10-12) more attractive to recent and prospective dropouts. It includes the expansion of adult education programs, free evening meals for students attending night school, and courses providing basic work qualifications; • Programa de Educação Profissionalizante de Minas Gerais (PEP/MG), which provides free slots in private technical schools for students in the second and third years of ensino médio, as well as recent graduates; and teacher training. 56 17. In the 2008 Acordo, the target for progress on these eight priority projects was broadly defined with 100 percent of the planned activities to be completed. The Agreement nevertheless set out a fairly detailed description of how such progress was to be evaluated. Each priority project was to be detailed according to the project management methodology adopted by SEPLAG. These plans were to include a set of actions and products to be achieved in the current year. The Central Superintendency for the Strategic Management of Resources and Actions of the State (Superintendência Central de Gestão. Estratégica de Recursos e Ações do Estado – SCGERAES) was required to intensively monitor the rate of execution of these projects against the planned actions and targets. 18. The system for scoring each project and consolidating them into a single score for this component was complex. Essentially, each project was evaluated on the basis of three criteria: the proportion of the target that was achieved, the timeliness of its achievement, and the proportion of the original budget allocation actually spent. If the target for the project was fully achieved on time and on budget, it received its full weight. If the target was only partly achieved, it received a reduced weight. And if it were delayed or subject to cost overruns its weight would be further reduced. Thus, for example, a project that achieved only 80 percent of its target with a delay of 30 days (but remained within its budget) would receive only 88 percent of the weight to which it would otherwise be entitled. To consolidate the rating for all eight projects, each project was assigned a weight – or more precisely two weights; one based on the relative cost of the project (its original budget allocation for the current year, divided by the budget allocations for all structural projects in the current year) and the other, an equal weight (in this case, one-eighth of the total). The total score for the component was calculated as the average of the two weighted scores. 19. The third section of the Agreement concerned management reforms. There were nine of them in the 2008 Acordo with the education system. They consisted of: (i) the identification of ‘critical macro-processes’ and (ii) the redesign of at least three of them; (iii) the development and testing of the new version of the state school management data base (Sistema Integrado de Gestão – SIGESP) and (iv) its implantation in 695 schools; (v) the development of a system for monitoring school construction; (vi) implementation of the pedagogic program Programa de Intervenção Pedagógica (PIP) in 2600 schools; (vii) increased use of SIMAVE (the target was 360,000 visits) and finally (viii,ix) two measures aimed at processing of retirement applications and staff dismissals. 20. These were also weighted to produce a consolidated score. The development of the system for monitoring school construction was given the heaviest weight (30 percent). The implementation of the SIGESP system in 695 schools was given a weight of 20 percent. In principle, no credit was to be given for partial achievement of a target, although the Acordo authorized the Evaluation Commission (see below) to award partial credit on an exceptional basis. 31 Penalties were also imposed for delay. A project that was delayed by more than 60 days, for example, would receive only half the weight to which it was otherwise entitled. No penalty was imposed for cost overruns. In order to yield a single consolidated score for the education system as a whole, the score of each three components were then combined and 31 The Acordo also explicitly granted credit for partial achievement of the seventh target. If the number of visits to the virtual reference center for teachers was between 252,000 and 279,000 (rather than the targeted 360,000) for example, 60 percent of the credit for achievement of the task would be awarded. 57 weighted. The score on indicadores finalísticos was given a weight of 70 percent; the priority projects, 20 percent, and the management reforms, ten percent. 21. The content of first-level Acordos has changed somewhat over the subsequent six years. There were five indicadores finalísticos in the 2013 Agreements with the education system, down from the former 14 32, while the number of structural projects had increased, from eight to eleven (of these, four were also in the 2008 Acordo). The number of management reforms had decreased to eight. However, a new component has been added aimed at encouraging improvements in expenditure management. The 2013 Acordo includes eight such measures, including: the percent of items acquired through the state’s electronic procurement system, the proportion of the state’s procurement plan actually executed, the percentage of operational expenditures as a percent of total expenditures, and the index of waste of supplies in stock. 22. Second Level Acordos. The 2008 reform required that the targets specified in each first level Acordo be broken down into individual results and targets for each working group (equipe de trabalho) within the organization (secretariat, foundation, etc.). These second level Acordos were to be signed by the head (dirigente maximo) of the organization and the head of each working group. In the case of the education system, there were separate agreements for each school, as well as agreements with each of the departments in the central administration of the Secretariat of Education. 23. The Acordos with individual schools contained only quantitative indicators. According to the former secretary of Education of Minas, Joao Filocre 33, the 2008 Agreement tracked student performance (average proficiency levels and percentages of students meeting recommended proficiency levels at various grades), drop-out and graduation rates, age-grade distortions, and the proportion of students attending full day programs (tempo integral). The 2013 second-level Acordos with individual schools tracked 16 indicators. These included ten of the 14 indicators tracked in the 2008 first-level Acordo (see Table 1) dropping only the graduation rates for ensino fundamental and ensino medio and the separate indicators of age/grade distortions in Northern Minas, while adding six new ones. In addition to tracking average proficiency scores in Portuguese and math at the fifth and ninth grade levels of ensino fundamental and the third grade of ensino médio, the Acordo tracked the percentage of students performing at the recommended level in each subject at each of the three grade levels. 24. The initial targets for individual schools are based on a somewhat sophisticated trend analysis (the Winters method). This established the starting point for negotiations between the SEE and the individual school directors. To avoid the incentive to set easy targets for each school in order to maximize its chance of success the performance score for the Secretariat as a whole was largely determined by the aggregate performance of the individual schools. Thus if a challenging target was set on the Secretariat-level was automatically passed down to the level of individual schools. 32 They are: the proportion of students in 3rd grade of EF meeting recommended literacy levels and the proportion of students in ninth grade of EF and third grade of EM meeting recommended proficiency levels in Portuguese and math. 33 Joao Filocre. Minas Educação: Remuneração Variável e Qualidade de Educação. 58 25. The 2013 second level Agreement with the SEE also included separate agreements for various administrative units within the central administration. The Acordo with the public relations department, for example, tracks four indicators: (i) the number of articles published in the state government news magazine (Agência Minas) or the Diário Oficial, (295, down from 250 in 2012); (ii) the number of press reports submitted on time (eight, up from seven); (iii) the percent of staff with personal staff development plans (planos de gestão do desempenho individual—PGDIs) prepared on time (92 percent, down from 100 percent in 2012) and (iv) the ‘timeliness and quality of data provided to the state strategic management monitoring system. In addition to quantitative indicators, the Acordo requires the public relations department to produce three ‘products’: 15 good practice videos; 16 televised round table discussions, and five new preparation courses for the national high school (ensino médio) exam. 26. Rewards. As noted earlier, the staff of organizations meeting their targets is eligible for bonuses. At the end of a calendar year, (or more precisely, in the January following the close of the fiscal year) each organization prepares a performance report (relatório de execução) comparing actual results against targets. This is submitted to the respective Comissão de Avaliação for each agreement. In the case of a first level agreement between the Governor and the education system, the evaluation commission is comprised of representatives of the Governor, the Secretariats of Finance, Planning, and Education, and a representative of the staff of the SEE. In the case of a second level Acordo between the SEE and an individual school, it is comprised of: (i) a representative of the Governor, nominated by SEPLAG, (ii) a representative of the school, nominated by the school director; (iii) a representative of the school employees, nominated by their union and the employees themselves, and (iv) a representative of SEPLAG. 27. The evaluation commissions review progress against targets and conclude with a consolidated score for each school and a score for the secretariat itself. The score is weighted. Surprisingly, only half of the bonus is awarded on the basis of the school’s performance; the other half is based on the performance of the secretariat as a whole. 34 No part of the bonus is based on the performance of individual staff—although staff are, in theory, subject to individual performance evaluations (see below). Because the bonus is calculated as percent of each individual’s salary, it is larger (in terms of Reais) for people in higher paying positions. Thus a poorly performing staff in a highly-paid position would receive a larger bonus than a high-performing staff in a lower paid position. And even if the whole school failed to achieve its targets, staff would still receive a bonus—provided the Secretariat as a whole achieved its targets for the State as a whole. 35 28. Results. At first glance, the Acordos would appear to have had a remarkable effect on student performance—particularly in the earlier grades. As shown in Figure 4, the percentage of third grade students meeting the recommended level of literacy nearly 34 In the earlier Acordos, 65 percent of the bonus was based on the performance of the Secretariat. Ten percent was based on the performance of the regional office of the education secretariat and only 25 percent on the basis of the performance of the individual school. 35 Note that the bonus is a one time-payment, rather than a permanent increase in salary. According to mission interviews, the state has not yet paid the bonus for 2012 and owes no payment for 2013 because targets were not met. It is reported that the state has promised to pay 2012 bonus later in 2014. 59 doubled between 2006 and 2013, reaching 93 percent by the end of the period. 36 (All data are for the rede estadual only.) There have also been impressive gains in the proportion of students meeting recommended levels of proficiency in Portuguese and mathematics in the lower grades. The proportion of children in the fifth grade meeting recommended proficiency levels in Portuguese and math doubled over the eight year period. Progress, however, tapers off somewhat at the higher grades. The proportion of ninth graders meeting proficiency levels increased by a (still impressive) 55 percent over the period. But gains in the third year of ensino médio were not as good. Proficiency levels in Portuguese and mathematics both increased by about 33 percent. And at the end of the period, only one- third of students in the final year of ensino médio were reading at the recommended level and less than four percent of students were meeting recommended levels in mathematics. 37 Figure 1: Trends in Performance on Education Indicators 100 Percent at Recommended Level 3rd EF Literacy 80 5th EF Portuguese 60 5th EF Math 9th EF Portuguese 40 9th EF Math 20 3rd EM Portuguese 3rd EM Math 0 2006 2007 2008 2009 2010 2011 2012 2013 Source: Ministry of Education of Minas Gerais 29. However, correlation does not prove causation. To derive lessons from the Minas experience, one has to determine the chain of causality: the connection between the actions taken by the State and the actual changes in literacy rates and student performance in Portuguese and mathematics at the lower grades. Furthermore, identifying the role and impact of the Agreements and their design, i.e. effectiveness of targets, choice of the priority projects, and the monetary rewards. And most importantly, if and how any of these actions affected the behavior of teachers in the classroom and the learning experience of students. 30. According to interviews, the targets did help by defining explicit quantitative goals gave a clear sense of the Governor’s priorities, from the Secretary of Education to individual school directors and teachers. However, the impact of the other components of the Acordos appeared to have varied. Most of the sectoral staff interviewed seemed to regard the structural projects and expenditure management monitoring as unwarranted micromanagement. The salary bonuses were also viewed as not particularly important 36 Due to changes in state policy, this improvement may be less impressive than it appears. In 2006 (Minas discontinued its policy of holding under-performing children back during the first three years of ensino fundamental. Children are now automatically promoted to the next grade, regardless of performance. This may, in part, explain the improvements in literacy rates among third graders. 37 It also bears noting that most of the gains occurred in the early years of the reform: 2006-2010. After 2010, there was virtually no gain (9th grade Portuguese being the one exception). In the last year of ensino médio, proficiency levels actually declined in both Portuguese and math. 60 motivating factors, although this may be because they were only remotely related to individual teacher performance (or even individual school performance). 31. Even if the targets were effective in setting out the government’s priorities, how did this affect the behavior of teachers in the classroom? In many respects, school directors are hamstrung in their ability to respond to performance targets as, for example, they are not involved in staff recruitment, rotation or dismissal. However, reforms introduced around the same time as the Agreements, gave school directors some ability to reward individual performance. Prior to 2003, all teachers received automatic salary increases after five years of service, regardless of performance. As of 2003, the automatic increases were abolished for teachers hired after that date. 38 Instead, salaries now had two components: a fixed component and a variable component based on performance. Performance was evaluated on the basis of four criteria: professional development (15 points), interpersonal relations (15 points), professional and institutional commitment (20 points), and technical and professional ability (50 points). With respect to the last of these, performance was evaluated on the basis of a teacher’s effectiveness in planning the work, making use of student evaluations, teaching methods, management efficiency, and commitment to teaching all students, including those with learning difficulties. 32. The school director, however, did not have the authority to evaluate teacher performance on his/her own. Instead, the evaluations were conducted by a commission (Comissão de Avaliação) consisting of five members: the director of the school, two teachers selected by their peers, and two members elected by the community. The evaluations, moreover, were not particularly selective. In 2009, 42 percent of evaluated staff received a rating of over 90 points and 51 percent received a rating between 80 and 90 points. As a result, individual performance bonuses may have little effect on teacher behavior. 39 33. In theory, performance evaluations also affect a teacher’s prospects for promotion. To advance within grade, a teacher must have five positive evaluations (defined as at least 70 points) along with the required academic credentials. Negative evaluations can result in dismissals. The process for dismissal can be initiated after two negative evaluations in three consecutive years, three negative evaluations in five, or four negative evaluations in ten. Since the vast majority of teachers receive positive evaluations, however, this presumably has little effect on teacher behavior. (The process of firing new teachers who are still on probation is much simpler and has resulted in 600 dismissals (out of a teacher corps of 160,000) according to Filocre. Although the number is small, Filocre reports that the demonstration effect is powerful.) Box 1. Human Resource Management Reforms In addition to introducing bonuses for individual performance and bonuses for performance at the work unit and secretariat level (through the Acordos), the Minas government undertook several other reforms in human resource management in the period 2003-2014. Inter alia, it consolidated the number of career streams in the state civil service structure, reducing the number from 893 to 168. This considerably increased the State’s ability to 38 Staff hired before 2003 have the option of switching to this system but most did not. 39 In principle, this problem could be addressed by imposing a normal distribution on performance ratings. According to mission interviews, such an approach would not be accepted in Minas at the present. 61 allocate staff efficiently. By broadening the definition of each career stream, staff can now shift from one position to another without having to restart their careers from the beginning. Human resources reforms included the introduction of a modern system for human resource management (SISAP – Sistema de Administração de Pessoal) and a system for performance evaluation (SISAD – Sistema de Avaliação de Desempenho), in 2005 and 2007 respectively. As such, information available before the introduction of the systems is limited and doesn’t allow for a complete analysis over time. 34. But perhaps the most important measure that accompanied the Acordos was a program aimed at ensuring that all children were reading and writing by age eight. The Programa de Intervencão Pedagógica (PIP) 40 was introduced in 2007, and provides intensive on-site assistance in literacy training. Organizationally, PIP consists of a central team of 46 literacy training specialists—one for each region. The central team has three functions: (i) disseminating key concepts of the program in a form that is readily understandable; (ii) monitoring the situation in each school in order to identify needs and problems and (iii) training regional teams. Each specialist spends two weeks per month visiting the regional office of the education secretariat (SRE) and its schools. The regional teams spend four days a week visiting schools, focusing on the ones with the most problems and lowest performance. At its peak, the regional teams had 1,800 staff, making 87,000 PIP visits per year. 35. The PIP arguably played an important role for the remarkable increase in literacy rates from 2006 to 2013. The targets in the Agreements prompted school directors and decision-makers in the Secretariat of Education to focus on improving literacy rates and focused attention on student performance, while holding them accountable for achieving the targets. However, the managers tasked with achieving the goals needed the authority and the technical support to do so, such as the PIP. However, giving school directors more control over staff recruitment, assignment and promotion might have given additional room to further improve student performance. Similarly, performance bonuses granted under the Acordos could have given greater weight to the performance of individual schools and less weight to aggregate performance at the state-wide level. 36. Design issues on relevance of targets. While schools are just one of the targets of Minas’ system of performance monitoring, they may be uniquely well suited to it. In some sectors, it is more difficult to translate policy goals into quantifiable targets for specific government organizations, which might result in targets, which are unrelated to policy goals or targets that require organizations to achieve policy goals that they have little control over. 37. One example of the former is the set of targets for the public relations Department of the Secretariat of Education. In this case, the targets required the Department to produce a number of videos without referring to the video’s content or objectives. Another example is one of the targets for the Gabinete of the Secretariat of Economic Development. It required the Gabinete to increase the ratio of Minas’ economic 40 According to Filocre, PIP has its own set of results agreements (Termos de Termos de Pactuação de Metas) signed by the school director, the regional office of the SEE and the parent-student-teacher board (Colegiado da Escola). As described by Filocre, the Termos de Pactuacão set out proficiency goals for 2008, 2009, and 2010. See Joao Filocre. Undated blog. Mais tempo para ensinar, mais tempo para aprender. 62 growth rate to the economic growth rate of Brazil as a whole, a target which is beyond the Gabinete’s control. 38. Similar causality issues are apparent in the health sector. The first-level Acordo for the Secretariat of Health requires the Secretariat to reduce the rate of infant mortality from 13.1 per thousand births in 2010 to 12.6 in 2013. But many of the factors that account for infant mortality (including contaminated drinking water) also lie beyond the purview of the Secretariat. 39. However, many targets reflected actions that the Secretariat has some influence over. An example is the target of a percentage of births, where the mothers have had at least seven pre-natal checkups. By the same token, the indicadores finalisticos for individual hospitals (in the second level agreement with FHMIG) include some of the standard measures of hospital efficiency, including the average length of stay and the hospital occupancy rate 40. The Agreements also resulted in a high number of indictors to be tracked across sectors. There are presumably 66,300 targets for state schools alone (17 indicators for each one of the 3,900 state schools). There are 165 indicators for the state hospitals (15 indicators for each of the eleven state hospitals) as well indicators for the casas de saúde (14 indicators for each of the four of them). 41. Timing of implementation. The preparation for an annual Acordo begins after the budget for that year has been submitted to the State Assembly, which generally happens by September 30. The final agreement on an Acordo is typically not reached until March of the year to come. As a result, the process starts before the results of the previous year are available, which could serve as a basis to set targets. By the same token, the process ends after the budget year is well underway and managers have time to influence the results. One alternative would be to start the Acordo preparation process much earlier in the budget cycle and rely on indicators from the previous calendar year or to include the preparation of Acordos in the budget preparation cycle. 42. Overall, judged solely on the basis of the focus of this review on the Agreements in the education sector, this operation appears to have been worthwhile. In retrospect, however, the Bank could have urged SEPLAG to be more conservative in defining the scope of the Acordos—confining them to actions that are both meaningful and under the control of the responsible organization. It could also have encouraged the state to take complementary actions (beyond those included in other DLIs). These might have included giving greater management authority to school directors – to the extent that the federal law allows - and adjusting the performance bonuses to give greater weight to individual school performance. 43. In the next section we turn to the data to analyze Minas’ results in selected sectors and those of the other Brazilian States. If Minas’ management reforms over the decade were transformational, one would expect that Minas would have over performed other States in crucial sectors such as education, health and citizen security. 63 II) Results in Minas Gerais vis-à-vis other Brazilian States 41 44. This section compares the experiences of Minas Gerais, states implementing results agreements (e.g. Pernambuco, Rio de Janeiro) and states that did not implement any form of results-based management 42. We use nationally comparable outcome indicators such as child mortality, homicide rates, and proficiency in Mathematics and Portuguese as measured by IDEB scores. The analysis explores the relationship between performance management and changes in sectoral outcomes using a variety of methods. The results of the empirical analysis support the expectation that the implementation of team-level results agreements is associated with significant and positive changes in outcomes in the security and education sectors, at least in the short and medium term (not in health). 45. Minas Gerais registered positive results in the education sector, surpassing the average of the southeastern region. The state also has a lower homicide rate than other states with the same GDP per capita. Similarly, Pernambuco has reversed a rising homicide rate and recorded considerable gains in the education sector through the implementation of performance agreements. In the case of Rio de Janeiro, the introduction of performance agreements and a teacher bonus corresponded with the first positive change recorded in educational achievements since 2005. Although Ceará is not using performance-related pay, its monitoring system and results-based management model has translated into sizable and consistent gains in the education and health sectors. Health sector 46. In the case of Minas Gerais, there is no discontinuity in the trend that could be linked to the introduction of performance agreements in the health sector in 2004 or their refinement in 2008 (Figure 4). While the rate decreased by 1.2 points between 2007 and 2010, child mortality levels remain above the regional average and the distance between the two has been fairly constant. Minas Gerais’ trend is similar to that of the southeast region and that of Rio de Janeiro, a State which does not use result agreements in the health sector. 41 This section is based on Vinuela and Zoratto (2013). 42 It is important to mention the limitations of this analysis: the reforms under consideration have not been designed ex ante as randomized controlled treatments, with clear treatment and control groups. To mitigate problems of causal inference created by the lack of a control group, a regression discontinuity and difference- in-difference designs are used. Moreover, responsibilities for the provision of health, education, and security services are shared across levels of governments and attributing results to state interventions can in some cases be difficult. One exception is secondary education that is the sole responsibility of state governments. 64 Figure 2: Child Mortality in Minas Gerais and Rio de Janeiro (per 1,000 born alive), 2002-2010 40 MG: MG: Chil Mortality (per 1,000 born alive) Performance Performance Agreements Agreements – 1 stage – 2 stage 20 10 30 2002 2004 2006 2008 2010 Year Minas Gerais Rio de Janeiro Southeast Source: Ministry of Health, SVS 2013. Security Sector 47. After three consecutive years of decline, the national homicide rate per 100,000 inhabitants started increasing again in 2007. In the following three years, the rate rose by 9 percent. During that period of time, only 8 states recorded reductions (Figure 5). The largest drop was observed in Pernambuco with a reduction of 25 percent of its homicide rate, followed by Rio de Janeiro, Mato Grosso do Sul, and Minas Gerais with 15, 12, and 11 percent respectively. Yet it is important to bear in mind that Rio de Janeiro and Pernambuco had some of the highest homicide rates in the country and their rate remain higher than the value predicted by GDP (Figure 6). Figure 3: Homicide Rate Change, 2007-2010 Figure 4: Homicide Rate in Minas Gerais and Rio de Janeiro (per 100,000 inhabitants), 2002- 2010 RJ: PBM 60 20 Homicide Rate (per 100,000 inhabitants) Agreements PA MG: Performance PB BA - Bonus Agreements– 1st stage 50 AP MG: 2nd stage 10 AM CE RO AL 40 TO RN SE GO MA PR SC AC DFMT 30 PI 0 SP RS RR MG ES MS 20 RJ -10 10 PE -20 0 10 20 30 40 50 60 2002 2004 2006 2008 2010 Homicide Rate 2007 Year 95% CI Fitted values Minas Gerais Rio de Janeiro Homicide Rate Change 2007-2010 Southeast Source: Ministry of Health, SVS 2013. Source: Ministry of Health, SVS 2013. 65 48. Minas Gerais’ homicide rate has remained below the regional average in the southeast. Currently, the State has the second lowest rate in the Southeastern region after São Paulo and the fourth lowest in the country. The State achieved considerable reductions between 2004 and 2010, though it was surpassed by São Paulo that currently has the lowest rate in the region. It is, nonetheless, important to highlight that the introduction of performance-related pay in 2004 coincided with the change in trend in homicides. 49. While displaying a declining trend since 2004, reductions were on average higher after 2007 (2.7 percent for 2004-2006 versus 4 percent for 2007-2010). Between 1999 and 2004, the homicide rate had more than doubled in the State of Minas Gerais, reaching the highest point in the states contemporary history. During that period, the State trend was going in the opposite direction than that of the other three states that had started to see declining levels of homicide since 2002. Education Sector 50. The national average IDEB score for public secondary schools have improved at an average of 4 percent every two years since the index was adopted in 2005. However, the improvement observed in the period 2005 to 2011 was more modest than made in primary education. The change in the national average grade was of 13 percent, compared with 31 percent in the initial years of primary education and 17 percent in the final years of primary education. Between 2007 and 2011, the states of Goiás, Rio de Janeiro, São Paulo, Santa Catarina, Roraima, and Tocantins registered much larger changes than those for states that had the same score in 2007 (Figure 4). Many observed no change or declining scores. Still, among others, Ceará and Minas Gerais have scores above the predicted value for their GDP, while Rio de Janeiro falls much below what the score for its income level. Figure 5: IDEB Public Secondary School Score Figure 6: IDEB for Public Secondary School Change 2007-2011 and IDEB Score in 2007 Score in Minas Gerais and Rio de Janeiro (0- 10), 2005-2011 .6 4 GO MG: Performance MG: Performance RJ: Performance Agreements – 1 stage Agreements – 2 stage Bonus Schools .4 RJ IDEB Secondary School SP SC 3.5 .2 PI AP MT AM TO PE MG 0 RN SE MA CE RR MS 3 PB BA DF ES -.2 AL PA AC RS PR 2.5 -.4 RO 2.5 3 3.5 4 2004 2006 2008 2010 201 IDEB Secondary School 2009 Year 95% CI Fitted values Minas Gerais Rio de Janeiro IDEB Secondary School Score Change 2009-2011 Southeast Source: INEP 2013. Source: INEP 2013. 66 51. In the southeast, Minas Gerais has shown steady improvement in the IDEB grade for secondary education and remains above the regional average. The introduction of performance agreements does not appear, on the aggregate level, to have generated any clear discontinuity. And, the distance between the score for the state and the average for the region seems to be closing. On the other hand, the first generation of these agreements was already in place when the IDEB began to be measured in 2005. Before-and-after Comparison of Outcomes Table 2: Before-and-After Comparison of Means Other States in their Region Before After Difference Northeast without Pernambuco Child Mortality (per 1,000 born alive), 2007-2010 34.27 22.71 -11.57*** Homicide Rate (per 100,000 inhabitants), 2007-2008 20.62 30.09 9.47*** IDEB Public Secondary School Score, 2007-2011 2.65 2.88 0.23*** IDEB Public Secondary School Score, 2009-2011 2.69 2.95 0.26*** Southeast without Minas Gerais Child Mortality (per 1,000 born alive), 2004-2010 16.68 14.37 -2.32*** Homicide Rate (per 100,000 inhabitants), 2004-2010 45.98 36.48 -9.49* Child Mortality (per 1,000 born alive), 2007-2010 15.86 13.66 -2.20*** Homicide Rate (per 100,000 inhabitants), 2007-2010 41.93 34.42 -7.52* IDEB Public Secondary School Score, 2007-2010 3.09 3.27 0.18 Southeast without Rio de Janeiro Homicide Rate (per 100,000 inhabitants), 2009-2010 33.21 29.59 3.62 IDEB Public Secondary School Score, 2009-2011 3.36 3.61 0.25** Source: Authors, based on data from IBGE. Note: *** Significant at the 0.01 level (2-tailed) ** Significant at the 0.05 level (2-tailed) 52. When comparing the changes in Minas Gerais to other states in the southeast region, it is possible to conclude that only improvements in education outcomes go beyond the general trend. In child mortality, the change is very similar and the difference between the state and the region remains fairly constant. The homicide rate for the period after the introduction of the “Management Shock” policy is lower than the rest of the region, but on average other states have had larger reductions. Conversely, the State recorded a statistically significant improvement in the IDEB score, but not the average for the rest of the region. 53. Regression analysis suggest that, at least in the short and medium term, the implementation of results-based management is associated with significant and positive changes in outcomes in the security and education sectors. On average, states using team level targets have 6.6 fewer homicides per 100,000 inhabitants than those that do not, all else equal. Similarly, states that have introduced performance agreements and a bonus for teachers and school staff have improved their Index of Basic Education Development (IDEB) score for public secondary schools by 0.26 additional points. 67 With the implementation of the second generation performance agreements Minas Gerais has achieved positive results in the education sector that go beyond the gains of the southeastern region. Pernambuco has reversed a rising homicide rate and recorded considerable gains in the education sector. In the case of Rio de Janeiro, the introduction of the performance agreements and teacher bonus in 2010 corresponded with the first positive change recorded in educational achievements in 6 years. While the analysis points to a positive effect of results agreements on service delivery outcomes, it is important to note that the adoption of these models has not been implemented in isolation. They require clear planning and the disaggregation of goals to a level that can be linked to the action of teams and individual members. Fiscal reforms and efforts to strengthen planning and budgeting functions were important preconditions for their adoption. In the education and health sector, the introduction of common metrics and monitoring systems capturing the relative performance of every school/hospital, municipality, and state has facilitated the measurement of results and distribution of rewards. These common metrics have not only been effective in providing a basis for comparison, but also have been important in motivating reform. III) Survey with public servants of Minas Gerais 43 54. During supervision missions, the Bank team often heard enthusiastic reports of a “change in culture” or a “behavior change” resulting from the introduction of performance agreements. According to those views, the government and its staff had moved in the direction of working for results, instead of working to “fulfill processes”. This anecdotal evidence of a change in culture inside the institution motivated the elaboration of a broader perception survey that could reach a larger number of public servants. The survey was conducted in mid-December 2014 among public servants of Minas Gerais to capture changes in behavior and motivation that might have resulted from the introduction and expansion of a results-based public management system since 2003. A link to the online survey was sent via email to 17,000 public servants based in the Administrative Center (Cidade Administrativa) out of which 1,926 (11.33 percent) completed the questionnaire 44. The full list of questions and answers is presented below. 55. Profile of respondents: approximately 70 percent of those who answered the survey started working for the Government after 2003. As such, only 30 percent of respondents witnessed the period pre and post management shock. Among those 70 percent, 26 percent started working between 2003 and 2008 and thus witnessed both the introduction of 1st and 2nd –level performance agreements, in 2003 and 2008 respectively. The remaining 44 percent started working between 2009 and 2014, when 2nd level agreements were already in place. 68 percent of all respondents are in the 18-45 year old bracket, and the majority (57 percent) of respondents are women. 41 percent of respondents 43 This section was prepared by Laura Zoratto (GGODR), Till Hartmann (GGODR) and Sheheryar Banuri (DECMG). We are grateful to Pedro Olinto (GPVDR), Philip Keefer (IADB) and Renos Vakis (DECSM) for their comments and suggestions for this questionnaire. 44 The total number of active State employees is 400,203, which includes, among others, teachers, police men and firemen. 68 occupy a technical/analyst position, and 11 percent are senior management staff (director level and above, up to Secretary). The majority of respondents – 44 percent – work in line secretariats such as Health and Education, and 21 percent are in central secretariats such as Planning, Finance and the Governor’s cabinet. 56. The vast majority of respondents stated that they know their role and goals at work but a smaller percentage attributes this to performance agreements. 93 percent of the respondents that work for the Minas Government for 5 years or more indicate that they have a clear understanding of their role and goals at work (question 6). Moreover, 75 percent also assert that they understand their role and goals better than they did five years ago (question 7). However, only 44 percent of respondents agrees or strongly agrees with the assertion that the introduction of results agreements helped them understand what was expected of them in their job (question 10); 29 percent neither agree nor disagree with the statement, which seems to indicate that, although the vast majority of respondents have a clear understanding of their role and goals, not everyone attributes this clarity to the introduction of results agreements. 57. 50 percent of respondents think that, in general, the introduction of results agreements has improved the functioning of the public service in Minas; 27 percent neither agrees or disagrees (question 11). Similarly, 55 percent believes that the results agreements established are relevant to the overall mission of the respective institutions, and 25 percent neither agree nor disagree (question 12). 58. 65 percent of respondents that work for the Government of Minas Gerais for one or more years have received a performance bonus of any sort in the past five years, and 5 percent said they do not know what a performance bonus is. The remaining did not receive a bonus. 54 percent of respondents are dissatisfied with the professional growth opportunities available in their jobs, in contrast to only 27 percent who are satisfied (19 percent are neutral). Interestingly, 39 percent think that the opportunities for professional growth have improved in the last five years, while 42 percent disagree. 59. Open Question: The final question of the survey (question 13) “In what way did the introduction of the results agreements affect your work?” was answered by 986 respondents. A qualitative analysis of the individual answers revealed mixed answers ranging from strong positive to strong negative perceptions of the reform. In general, many statements praised the fact that the reform introduced clear goals and objectives. On the negative side, many statements point out that the reform’s effectiveness was hindered by insufficient economic incentives (i.e., low bonus payments, low salaries) and political influence. Several respondents indicated that they have always tried to do their work well, regardless of having results agreements or not. A few respondents mentioned that they had already worked with targets before and did not need the agreements to incentivize their work. Other recurring criticisms are that the results agreements only added bureaucracy to existing work and that the bonus payments had been often delayed. 60. Regression analysis: To further disentangle the survey results, we divided the respondents in three groups: 1 = respondents joining before 2003; 2 = respondents joining between 2003 and 2008; 3 = respondents joining between 2009 and 2013. We control for gender and age range, and have occupation and secretariat fixed effects. Regressions are probit specification with dependent variable: respondent agree/strongly agrees with the 69 statement. The interpretation is based on comparing the first intervention group (before 2003) with the second intervention group (2003-2008) and with the no intervention group (2009-2013). The data should be interpreted with caution however, as tenure is likely to blurr the results, since these are not true comparison groups. They do provide some indication, however, of whether the interventions had any effect. 61. Q6: Comprehension of goals and objectives: Workers exposed to both interventions (in 2003 and 2008) are more likely to agree with the statement, indicating that both interventions had incremental effects on comprehension of goals and objects. 62. Q7: Comprehension is better than 5 years ago?: Workers exposed to the first intervention are less likely to agree that comprehension has improved, relative to the second intervention. In fact, workers exposed to the second intervention are significantly more likely to report improvement relative to the control (workers not exposed to the intervention) and relative to the first intervention. Results are supported by the regressions as well (workers between 2003 and 2008 are significantly higher than either of the other two groups: p<0.01 and p<0.05). This suggests that the second intervention brought about more improvements than the first. 63. Q9: Opportunities improved in the last 5 years: Workers exposed to the first intervention and second intervention are not different from each other (p=0.27) but are significantly more likely to agree that opportunities have improved in the last 5 years relative to the control (p<0.05 and p<0.10 respectively). This indicates that both the first and second interventions improved perceptions of changes in career opportunities. 64. The survey was conducted after the introduction of results agreements; the lack of a baseline and thus reliance on recall questions that are potentially subject to recall bias can limit the validity of the findings. In general, however, the survey is indicative of a positive perception regarding the introduction of a results-based management model. Survey results also indicate that the second-stage results agreements brought more improvements in the way staff understand their role and objectives at work than the first-stage RAs. 70 Appendix 1: Questionnaire with Minas’ public employees and Results: Q1: When did you start working for the Government of Minas Gerais? In 2014 12.8% Between 2009 and 2013 32.5% Between 2003 and 2008 25.9% Before 2003 28.8% Q2: How old are you? Over 66 0.9% 46-65 31.0% 36-45 27.8% 26-35 33.6% 18-25 6.6% Q3: What is your gender? Male 43.4% Female 56.6% 71 Q4: 0.7% What is your job role? 0.3% 0.1% 2.6% Secretary (Secretário) 7.2% Assistant Secretary (Secretário- Adjunto) Undersecretary (Subsecretário) 31.1% 17.3% Superintendent (Superintendente) Director (Diretor) Advisor (Assessor) Analyst (Técnico) 40.6% Other 45 Q5: What department do you work at? Line Secretariats (e.g., Education, Health) Ministry of Finance (Fazenda) 33.2% 44.3% Ministry of Planning (Planejamento) Secretary for Economic Development (Desenvolvimento Econômico) Governor's Office (Gabinete do 0.7% Governador) 11.9% 2.7% Other 7.1% 46 45 Respondents who chose the option “Other” in Q4 listed themselves as: managers, technical assistants, auditors, administrative assistants, HR analysts, lawyers. The vast majority would fall in the category of analyst. 46 Respondents who chose the option “Other” in Q5 wrote that they work at: civil or military police, civil defense (Bombeiros) communications office, ombudsman office, and a few autarchies. Many respondents mentioned that they work at the “environment secretariat”, “science and technology secretariat”, “culture secretariat”, which would fall in the survey’s category of “line secretariats”. 72 Q6: Do you agree or disagree with the following statement? I have a clear understanding of my role and goals at work. 45% 47% 1% 1% 6% Strongly Disagree Neither Agree Strongly disagree agree nor agree disagree Q7: Do you agree or disagree with the following statement? I understand my role and goals better than 5 years ago. 42% 33% 14% 3% 7% Strongly Disagree Neither Agree Strongly disagree agree nor agree disagree 73 Q8: Do you agree or disagree with the following statement? I am satisfied with the professional growth opportunities available in my job. 30% 23% 21% 19% 6% Strongly Disagree Neither Agree Strongly disagree agree nor agree disagree Q9: Do you agree or disagree with the following statement? Opportunities for professional growth for public servants in Minas have improved in the last 5 years. 29% 25% 19% 17% 10% Strongly Disagree Neither Agree Strongly disagree agree nor agree disagree 74 Q10: Do you agree or disagree with the following statement? The introduction of results agreements helped me understand what was expected of me on my job. 36% 29% 16% 11% 8% Strongly Disagree Neither Agree Strongly disagree agree nor agree disagree Q11: Do you agree or disagree with the following statement? In general, the introduction of results agreements has improved the functioning of the public service in Minas. 41% 27% 8% 14% 9% Strongly Disagree Neither Agree Strongly disagree agree nor agree disagree 75 Q12: Do you agree or disagree with the following statement? The established results agreements are relevant to the overall mission of my institution. 46% 25% 6% 14% 9% Strongly Disagree Neither Agree Strongly disagree agree nor agree disagree Q13: Did you receive a performance bonus of any sort in the past five years? 4.9% Yes 30.6% No I do not know what a 64.5% performance bonus is 76 77