Document of The World Bank Report No: ICR2648 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-76750) ON A LOAN IN THE AMOUNT OF US$130 MILLION TO THE FEDERAL DISTRICT WITH THE GUARANTEE OF THE FEDERATIVE REPUBLIC OF BRAZIL FOR A FEDERAL DISTRICT MULTISECTOR PUBLIC MANAGEMENT PROJECT June 18, 2013 Brazil Country Management Unit Human Development Sector Management Unit Latin American and the Caribbean Regional Office CURRENCY EQUIVALENTS (Exchange Rate Effective December 31, 2012) Currency Unit = Real (R$) R$ 1.00 = US$ 0.4893 US$ 1.00 = R$ 2.0435 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS CBC Central Bidding Committee (Central de Compras) CCO Control Center for Operations (Centro de Controle Operacional) CPS Country Partnership Strategy DF Federal District (Distrito Federal) DFTRANS Federal District Transport Authority (Departamento de Transporte Urbano do Distrito Federal) DLI Disbursement-linked Indicator DO Development Objective ECD Early Childhood Development EEPs Eligible Expenditure Programs FHP Family Health Program (Programa de Saúde da Família) FM Financial Management GDF Government of the Federal District (Governo do Distrito Federal) ICR Implementation Completion and Results Report IDEB Basic Education Development Index (Indice de Desenvolvimento da Educação Básica) IEG Independent Evaluation Group IFR Interim Unaudited Financial Reports LFR Law of Fiscal Responsibility (Lei de Responsabilidade Fiscal) M&E Monitoring and Evaluation MTR Mid-term Review OECD Organization for Economic Co-operation and Development OM Operational Manual OSCIP Civil Society Organization of Public Interest PAD Project Appraisal Document PAF Fiscal Restructuring and Adjustment Program (Programa de Reestruturação e Ajuste Fiscal) PCU Project Coordination Unit (Unidade de Coordenação do Projeto) PDIs Project Development Indicators PDOs Project Development Objectives PDTU Master Plan for Urban Transportation (Plano Diretor de Transporte Urbano) PEFA Public Expenditure and Financial Accountability PPA Multi-year Financing Plan (Plano Plurianual) PSM Public Sector Management QER Quality Enhancement Review RA Results Agreement (Termos de Compromisso) RBM Results-based Management SEDEST State Secretariat of Social Development (Secretaria de Estado de Desenvolvimento Social) SEE State Secretariat of Education (Secretaria de Estado de Educação) SEFAZ State Secretariat of Finance (Secretaria da Fazenda) SEPLAG State Secretariat of Planning and Management (Secretaria de Estado de Planejamento e Gestão) SEPLAN State Secretariat of Planning and Budget (Secretaria de Estado de Planejamento e Orçamento) SES State Secretariat of Health (Secretaria de Estado de Saúde) SET State Secretariat of Transport (Secretaria de Estado de Transporte) SETC State Secretariat of Transparency and Control (Secretaria Estadual de Transparência e Controle) SIL Specific Investment Loan SWAp Sector-Wide Approach TA Technical Assistance TCDF Federal District Court of Accounts (Tribunal de Contas do Distrito Federal) Vice President: Hasan A. Tuluy Country Director: Deborah L. Wetzel Sector Manager: Joana Godinho Project Team Leader: Ricardo Silveira ICR Team Leader: Daniela Pena de Lima ICR Primary Author: Sandra Rosenhouse BRAZIL FEDERAL DISTRICT MULTISECTOR PUBLIC MANAGEMENT PROJECT CONTENTS A. Basic Information ............................................................................................................ i  B. Key Dates ........................................................................................................................ i  C. Ratings Summary ............................................................................................................ i  D. Sector and Theme Codes................................................................................................ ii  E. Bank Staff ....................................................................................................................... ii  F. Results Framework Analysis ......................................................................................... iii  G. Ratings of Project Performance in ISRs ....................................................................... ix  H. Disbursement Profile..................................................................................................... ix 1. Project Context, Development Objectives and Design ................................................... 1  3. Assessment of Outcomes .............................................................................................. 11  4. Assessment of Risk to Development Outcome ............................................................. 14  5. Assessment of Bank and Borrower Performance ......................................................... 15  6. Lessons Learned............................................................................................................ 17 Annex 1. Project Costs and Financing .............................................................................. 19  Annex 2. Outputs by Component...................................................................................... 20  Annex 3. Economic and Financial Analysis ..................................................................... 26  Annex 4. Bank Lending and Implementation Support/Supervision Processes................. 27  Annex 5. Beneficiary Survey Results ............................................................................... 29  Annex 6. Stakeholder Workshop Report and Results ....................................................... 30  Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR ......................... 31  Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ........................... 51  Annex 9. List of Supporting Documents .......................................................................... 52  Map IBRD 36809 .............................................................................................................. 53  A. Basic Information Federal District Country: Brazil Project Name: Multisector Management Project ID: P107843 L/C/TF Number(s): IBRD-76750 ICR Date: 06/18/2013 ICR Type: Core ICR GOVERNMENT OF Lending Instrument: SIL Borrower: FEDERAL DISTRICT Original Total USD 130.00M Disbursed Amount: USD 26.06M Commitment: Revised Amount: USD 130.00M Environmental Category: C Implementing Agencies: State Secretariat of Planning and Management (Secretaria de Estado de Planejamento e Gestão -- SEPLAG), State Secretariat of Education (Secretaria de Estado de Educação -- SEE), State Secretariat of Health (Secretaria de Estado de Saúde -- SES), State Secretariat of Transport (Secretaria de Estado de Transporte -- SET), and Federal District Transport Authority (Departamento de Transporte Urbano do Distrito Federal – DFTRANS) Cofinanciers and Other External Partners: Not applicable B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 01/29/2008 Effectiveness: 10/15/2009 10/15/2009 Appraisal: 07/01/2008 Restructuring(s): - - Approval: 04/09/2009 Mid-term Review: 01/17/2011 - Closing: 12/31/2012 12/31/2012 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Unsatisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Unsatisfactory Borrower Performance: Highly Unsatisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Moderately Moderately Quality at Entry: Government: Unsatisfactory Unsatisfactory Implementing Quality of Supervision: Unsatisfactory Highly Unsatisfactory Agency/Agencies: Overall Bank Overall Borrower Unsatisfactory Highly Unsatisfactory Performance: Performance: i C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Project Quality at Entry No None at any time (Yes/No): (QEA): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Unsatisfactory Closing/Inactive status: D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) General education sector 16 17 Health 15 23 Public administration- Education 18 19 Public administration- Health 18 24 Public administration- Transportation 33 17 Theme Code (as % of total Bank financing) Child health 10 5 City-wide Infrastructure and Service Delivery 32 26 Education for all 34 34 Health system performance 13 10 Population and reproductive health 11 25 E. Bank Staff Positions At ICR At Approval Vice President: Hasan A. Tuluy Pamela Cox Country Director: Deborah L. Wetzel Makhtar Diop Sector Manager: Joana Godinho Keith E. Hansen Project Team Leader: Daniela Pena de Lima Joana Godinho ICR Team Leader: Daniela Pena de Lima ICR Primary Author: Sandra Rosenhouse ii F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The proposed Project Development Objectives (PDOs) are the following: (a) improve public sector management and accountability by establishing results-based management practices and improving fiduciary oversight; and (b) increase access, quality and efficiency of the education, health and public transport services by modernizing the education system; modernizing, decentralizing and integrating various levels of health care; and strengthening the institutional and operational capacity of the public transport sector. Revised Project Development Objectives (as approved by original approving authority) Objectives were not revised. PDO Indicator(s) The Project Appraisal Document (PAD) identified three types of Project Development Indicators (PDIs): Global Disbursement-linked Indicators (DLIs), sector specific DLIs and unlinked PDIs. (a) Global Indicators Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Comply with the Fiscal Restructuring and Adjustment Program (Programa de Indicator 1: Reestruturação e Ajuste Fiscal -- PAF as measured by (i) ratio financial debt/real net income; and (ii) expenditures on personnel (DLI) Value Complies with (Quantitative or NA Complies with PAF PAF Qualitative) Date achieved 12/31/2008 12/31/2011 06/30/2011 Comments Target achieved every semester. (incl. % (i) the ratio should be less than one. Latest: 0.24. achievement) (ii) expenditure on personnel should be less than 60 percent. Latest: 52 percent. Execute at least 70 percent of the budget of each Eligible Expenditure Program Indicator 2: (EEP) included in the Project (DLI) Value Education: 56% (Quantitative or 0% 70% Health: 53% Qualitative) Transport: 55% Date achieved 12/31/2008 12/31/2011 12/31/2011 Comments Target generally not achieved by any of the sectors. Health and transport met the (incl. % target in 2010. achievement) iii (b) Sector-specific Project Development Indicators: Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Public Sector Management Secretariats/GDF agencies sign annual Results Agreements (Termos de Indicator 3: Compromisso – RA) defining performance targets (DLI) Value (Quantitative or 1 26 26 Qualitative) Date achieved 12/31/2008 12/31/2011 12/31/2011 Comments Target achieved. (incl. % achievement) Secretariats/GDF agencies issue annual performance reports, which are then Indicator 4: vetted and published by SEPLAG (DLI) Value (Quantitative 0 21 20 or Qualitative) Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments Target partially achieved. (incl. % achievement) Financial Management Reform Action Plan based on Public Expenditure and Indicator 5: Financial Accountability (PEFA) diagnostic implemented Value PEFA Action Plan (Quantitative Action Plan agreed. Not done fully implemented or Qualitative) Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments Target not achieved in either State Secretariat of Finance (Secretaria da (incl. % Fazenda – SEFAZ) or the Controladoria of GDF. achievement) Procurement Action Plan based on Organization for Economic Co-operation Indicator 6: and Development (OECD) Indicators diagnostic implemented Value OECD indicators (Quantitative Action Plan agreed. Action Plan fully Not done or Qualitative) implemented Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments Target not achieved. (incl. % achievement) Indicator 7: Performance incentives in the Education Sector Value (quantitative No Yes Not done or Qualitative) Date achieved 12/31/2008 12/31/2011 12/31/2012 iv Comments Target not achieved. This was subject to authorization by the Legislative (incl. % Assembly. A decree was approved and later (by June 2010) cancelled. achievement) Indicator 8: Performance incentives in Health Sector Value (quantitative No Yes Not done or Qualitative) Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments Target not achieved. This was subject to authorization by the Legislative (incl. % Assembly. A decree was approved but was not implemented. achievement) Education Increase the efficiency of school management strategies, as measured by Indicator 9: selected indicators Value Data not available Quantitative 0 90% or Qualitative) Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments With the changes in SEE, the program was modified and the monitoring (incl. % system was dropped. achievement) Improve Basic Education Development Index (Indice de Desenvolvimento da Indicator 10: Educação Básica – IDEB) score in 125 selected primary schools located in poor urban areas by two percent each Value 125 schools 66 schools 2005 IDEB of 125 (Quantitative increase IDEB by increased IDEB by selected schools or Qualitative) 2% 2% Date achieved 12/31/2007 12/31/2011 12/31/2011 Comments Target not achieved. (incl. % achievement) Increase access (enrollments) to early childhood development (ECD) programs Indicator 11: in low-income Regional Areas for children aged 4-5 by 13 percent (DLI) Value (Quantitative 77% 90% 68% or Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2011 Comments Target not achieved. (incl. % achievement) Implement full-time basic education activities (Escola de tempo integral) in Indicator 12: low-income areas of DF in 100 schools (DLI) Value (Quantitative 0 100 0 or Qualitative) Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments Target not achieved. New administrations reduced the expansion of the (incl. % program, and the few full time schools established were not in low-income v achievement) areas, as defined in the Project. Indicator 13: Improve completion rates in Primary Education Value (Quantitative 63% 80% 85% or Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2011 Comments Target achieved. (incl. % achievement) Indicator 14: Improve completion rates in Secondary Education Value (Quantitative 75% 85% 66% or Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2011 Comments Target not achieved. (incl. % achievement) Decrease number of students with overage distortion rate in Basic Education Indicator 15: (DLI) Value (quantitative 92,104 17,000 69,693 or Qualitative) Date achieved 12/31/2007 12/31/2011 12/31/2011 Comments Target not achieved. Entities implementing programs to reduce distortion were (incl. % discontinued with change in Government authorities. achievement) Decrease the number of students with overage distortion rate in Secondary Indicator 16: Education (DLI) Value (Quantitative 34,805 3,000 23,733 or Qualitative) Date achieved 12/31/2007 12/31/2011 10/08/2012 Comments Target not achieved. Entities implementing programs to reduce distortion were (incl. % discontinued with change in Government authorities. achievement) Health Indicator 17: Provide the Cartão de Saúde to 50 percent of the DF population (DLI) Value (Quantitative 10% 50% 75% or Qualitative) Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments Target surpassed. (incl. % achievement) Indicator 18: Cover 40 percent population with Family Health Program (DLI) Value (Quantitative 7% 40% 20% or Qualitative) vi Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments Target not achieved. (incl. % achievement) Cover 80 percent pregnant women with at least four prenatal visits in Indicator 19: Brazlândia, Paranoá, Riacho Fundo and Samambaia (DLI) Value (Quantitative 72% 80% 88% or Qualitative) Date achieved 12/31/2008 12/31/2011 07/31/2012 Comments Target achieved. Average of all locations. (incl. % achievement) Indicator 20: Decrease hospitalization due to complications of diabetes Value (Quantitative 1.9% 1.7% 1.5% or Qualitative) Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments Target achieved but data are unreliable. (incl. % achievement) Indicator 21: Increase the proportion of infant deaths investigated Value (Quantitative 40% 90% 64% or Qualitative) Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments Target not achieved. Number of investigated deaths under 1 year olds/ total (incl. % number of deaths under 1 year olds X 100. achievement) Indicator 22: Increase pap smear rate among women 25-59 years old Value (Quantitative 13.4% 20% 12% or Qualitative) Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments Target not achieved. (incl. % achievement) Establish management of the Hospital St. Maria by a civil society organization Indicator 23: (OSCIP) 70% activation Value Cost Reporting OSCIP contract (Quantitative OSCIP Selected User survey >70% cancelled or Qualitative) good evaluation Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments Target not achieved. (incl. % achievement) vii Transport SET submits the master plan for urban transport (PDTU) for approval to the Indicator 24: Federal District Legislature Final report approved by SET, Value Plan approved by draft law (Quantitative Plan does not exist Legislature on submitted to State or Qualitative) 05/04/2011 Legislature for approval Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments Target achieved. (incl. % achievement) DFTRANS controls operational and fare revenue data of the basic bus service Indicator 25: (DLI) DFTRANS has partial DFTRANS Value access to secondary processes primary (Quantitative operational & fare operational & fare Data not available or Qualitative) revenue data revenue data on a daily basis Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments Target not achieved. Hired bus companies withheld control of information and (incl. % data are not available. achievement) DFTRANS remotely monitors 10 percent of the fleet of the basic bus service Indicator 26: through the Control Center for Operations (CCO) (DLI) 10 percent of fleet Value of basic bus CCO not (Quantitative No CCO exists service remotely established or Qualitative) monitored through the CCO Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments Target not achieved. (incl. % achievement) Increase regularity index for trip compliance (timetables and itineraries) by bus Indicator 27: operators of the basic bus service Value Data not available (Quantitative 77% 90% or Qualitative) Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments Target not achieved. (incl. % achievement) Reduce number of complaints received by DFTRANS regarding non- Indicator 28: compliance with timetables and itineraries Value 9,000 7,000 Data not available (Quantitative viii or Qualitative) Date achieved 12/31/2008 12/31/2011 12/31/2012 Comments The methodology to measure this indicator was changed midstream and thus (incl. % even when available are not comparable with the baseline. achievement) (c) Intermediate Outcome Indicator(s) The PAD does not distinguish between PDO indicators and Intermediate Outcome Indicators, so all indicators in this document are referred to as Project Development Indicators (PDIs). G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 06/25/2009 Satisfactory Satisfactory 0.00 2 12/18/2009 Moderately Satisfactory Moderately Satisfactory 0.00 3 03/04/2010 Moderately Satisfactory Moderately Unsatisfactory 0.00 4 01/13/2011 Moderately Satisfactory Moderately Unsatisfactory 0.00 5 07/15/2011 Moderately Satisfactory Moderately Satisfactory 15.23 6 03/13/2012 Moderately Unsatisfactory Moderately Unsatisfactory 15.23 7 11/09/2012 Moderately Unsatisfactory Moderately Unsatisfactory 24.32 8 12/26/2012 Unsatisfactory Unsatisfactory 24.32 H. Disbursement Profile ix 1. Project Context, Development Objectives and Design 1.1 Context at Appraisal 1. The Country Partnership Strategy (Report No. 42677-BR, May 6, 2008, CPS) for FY 2008-2011, completed just before Project appraisal, noted Brazil’s solid progress with respect to improving macroeconomic stability and equity. Poverty decreased, as did income inequality. However, the global financial crisis was beginning to have a negative impact on economic growth. Some states responded to the situation by introducing reforms to improve the efficiency and quality of services in sectors that accounting for a significant share of Government expenditures. The first states to introduce some of these reforms were Minas Gerais, Ceará, Pernambuco and the Federal District (Distrito Federal -- DF). 2. At the time of Project preparation, the DF was the third largest urban area in Brazil after São Paulo and Rio de Janeiro. Its macroeconomic prospects were generally favorable, with growth in fiscal income of 11 percent in 2007. Population growth was greatly overburdening public sector services, which was serving a population 5-6 times higher than originally envisaged. Expenditures on human resources for education and health had been increasing substantially since 2002, to the point that they were squeezing out public investment. As a result, existing public services were deteriorating and the Government of the Federal District (Governo do Distrito Federal -- GDF) was unable to expand to underserved areas mostly located outside the central urban area. 3. In May 2007, in response to growing inequality and pressure on public funds, the incoming Governor, Mr. Arruda, launched an ambitious reform program to improve the overall efficiency and effectiveness of public expenditures. The public sector management (PSM) reforms launched by GDF centered on the development of Strategic Plans and Results Agreements (Termos de Compromisso -- RA) to be signed with line secretariats and specific agencies focusing on priority areas. This was expected to free up funds to enable an expansion of services in response to growing demands, and an improvement existing services. 4. GDF approached the World Bank (Bank) because it had insufficient funds to implement its ambitious reforms, and a Specific Investment Loan (SIL) with a Sector Wide Approach (SWAp) 1 was very compatible with Government’s own Results-based Management (RBM) approach. GDF originally planned for a series of four SWAps over a period of 10 years, beginning with the three sectors responsible for most of GDF expenditures: education, health, and transport. All three sectors had a poor and deteriorating coverage and quality of services, with markedly lower coverage and 1 Support to Government’s Eligible Expenditures Programs (EEPs); disbursements triggered by the execution of budgeted programs and achievement of specific results; and use of Government’s systems, procedures and staff. 1 quality in the periphery. Efficiency was also deemed to be low. In education, coverage of early child development (ECD) education and secondary/vocational education were found to be insufficient particularly in peripheral areas, and student performance, while higher than the national average (but declining) in central DF, was fifty percent lower in some of the more distant administrative units. High rates of age-grade distortion contributed to low educational outcomes. In health, population pressure combined with an increasing prevalence of non-communicable diseases due to population aging and changing life styles, greatly increased demand, particularly for hospital level care. Limited access to basic health care made the referral network more dysfunctional. An absence of an adequate information system to manage care was seen as a barrier to improved efficiency and effectiveness. In transport, problems were both institutional and operational. Key entities such as the State Secretariat of Transport (Secretaria de Estado de Transporte – SET) and the Federal District Transport Authority (Departamento de Transporte Urbano do Distrito Federal -- DFTRANS) lacked staff with the needed technical skills, information systems and regulatory framework, and had difficulties supervising and managing the existing services. The population, on the other hand, faced long trips due to the urban sprawl, and expensive and inefficient transport (infrequent off peak hours, irregular schedules) encouraging illegal services. 5. For the Bank, the provision of assistance to the GDF’s reform program fit well with the CPS for 2008-2011 agreed with Brazil. The overarching pillar of the CPS was to support growth, good governance and public sector management as a foundation to address equity, competitiveness, and sustainability. The new CPS shifted the focus of the World Bank assistance from federal to state level operations, and from increasing coverage to improving quality of services and public expenditure. 6. As for the Federal Government it was interested in state-level financing, as it would allow it to drill down federal programs to the state and municipal levels and enable it to align federal and state interventions. The Bank and the Federal Government were also interested in limiting the proliferation of multiple small loans and sought the aggregation of several programs into single loans to leverage results and reduce transactions costs. The Brazilian SWAp was the product of close collaboration between federal and subnational Governments and the World Bank. This period witnessed the development of several state multi-sectoral SWAps, including Ceará, Minas Gerais, and Pernambuco. 1.2 Original Project Development Objectives (PDO) and Key Indicators 7. The Project had two PDOs: (a) improve public sector management and accountability by expanding results-based management practices and improving fiduciary oversight; and (b) increase access, quality and efficiency of the public education, health and transport services through the modernization of the education system; the modernization, decentralization and integration of various levels of health care; and the strengthening of the institutional and operational capacity of the public transport sector. 2 8. Progress towards achievement of the PDOs was to be measured through a set of 28 Project Development Indicators (PDIs). Thirteen of the PDIs were selected to be also Disbursement-linked Indicators (DLIs), out of which two were defined as global indicators to be measured every six months. The other eleven DLIs were to be measured annually. Failure to comply with the first global indicator would block disbursements until the target was met. The second global indicator would determine the percentage allocation that each sector would receive every six months. No or partial compliance with the other DLIs would result in proportionally lower disbursements based on an agreed algorithm. Overachievement on all DLIs would result in a “bonus�. In addition, there were 15 un-linked PDIs. The Project Appraisal Document (PAD) did not distinguish between key outcome indicators and intermediate outcome and output indicators. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 9. The PDO remained valid and unchanged throughout the Project’s implementation. 1.4 Main Beneficiaries 10. The Project’s main beneficiaries were users of public education, health and transport services in GDF, and particularly in low-income areas. Service users would see an improvement in access and quality of services. Efficiency gains and a focus on results were expected to improve resource allocation, slowly reducing service inequalities between Brasilia and the peripheral regions. The 26 agencies and secretariats participating in the management reforms would benefit from the improvement of their management skills. 1.5 Original Components (as approved) 11. The Project included two components. The first followed a SWAp approach to support GDF’s public sector management reform, and the second consisted of a technical assistance (TA) component designed to support the fulfillment of targets in the first. 12. Component 1. Strengthening Results-based Management in the Public Sector (US$390 million, total Project cost; US$120 million, loan financing). This component would support the adoption of RBM tools, including the signing of annual RAs between the Governor and each of the line secretariats and agencies of the DF, and the submission of annual reports on performance with respect to agreed targets. Subject to legislature approval, the GDF intended to introduce rewards (e.g., additional budget, bonus payments) for those entities that met or exceeded their performance targets in health and education. 13. EEPs selected for financing were designed to introduce or expand reforms to improve the effectiveness, quality and equity of services in these sectors. In education, it was expected that the expansion of the Shared Management program (Gestão Compartilhada), whereby principals would be selected on the basis of qualifications rather than political appointment, and where incentives would be given to schools based 3 on teacher and student’s increased performance would improve efficiency, quality and performance. Expanding access to ECD programs and the Full-time Schools program (Escola de Tempo Integral), which provides longer and better learning opportunities than traditional schools, would improve access to education (reducing inequity) and in the latter case, also improve quality. The focus of these investments was to be low-income areas and areas with low Basic Education Development Index (Indice de Desenvolvimento da Educação Básica – IDEB) scores. In health, improvement in access and quality focused on the expansion of primary care through the Family Health Program (Programa de Saúde da Família – FHP). The expansion of FHP would strengthen prevention and resolve less complicated health problems at a lower level of care, decreasing patient overload in hospitals while improving service quality and health outcomes. The latter would also reflect a more efficient use of health resources. Efficiency would also be improved through the introduction of patient electronic health cards (Cartão de Saúde), through the piloting of the subcontracting of the management of the Hospital Sta. Maria to a non-profit social organization with the purpose of improving hospital management, and through the establishment of RBM practices in the sector. In transport, effectiveness and efficiency of SET and DFTRANS would be increased through the establishment of a transport Control Center for Operations (Centro de Controle Operacional – CCO), the preparation of an urban transport Master Plan, the review and improvement of internal processes, the modernization of management information systems and equipment, and building capacity of staff. 14. Component 2. Building Capacity in the Public Sector (US$10 million total Project cost and loan financing). In addition to financing TA and training to support the implementation of RBM and the fulfillment of agreed performance targets, this component was to finance preparatory studies for a possible future operation, and TA to strengthen fiduciary oversight (of the Project). Specific TA was included for: (a) the design and implementation of cost-accounting systems for education and health to permit an assessment of the cost-effectiveness of specific services; (b) TA for assessments of management programs in the education and health sectors (the education “Gestão Compartilhada� program and the public-private partnership to be established to manage the Hospital Sta. Maria); (c) an evaluation of outsourcing of some services in these two sectors (“Merenda Escolar� and hospital transport); (d) inputs to efforts of SET and DFTRANS to integrate the different public transport modes and to regulate the provision of public transport services in the DF; (e) an impact evaluation of the Government’s Management Modernization program; (f) training on Strategic Planning, M&E, and the definition of indicators for the State Secretariat of Planning and Management (Secretaria de Estado de Planejamento e Gestão – SEPLAG), Health, Education, SET and DFTRANS; and (g) TA to strengthen Financial Management (FM) and Procurement capacity including the development of Financial Management and Procurement Action Plans, a Public Expenditure and Financial Accountability (PEFA) exercise, and the modernization of the procurement processes of the Central Bidding Committee (Central de Compras – CBC) and the participating secretariats. 1.6 Revised Components 15. Project components were not revised. 4 1.7 Other significant changes 16. Not applicable. 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 17. Project design contained several elements that are considered best practice. The Project used Government procedures and staff to the extent possible, seeking to foster ownership, reduce transactions costs and strengthen local Government systems. It also linked disbursements to the achievement of results, which has been shown to lead to improved performance and management (IEG, 2009 p. xii). Besides the traditional assessments,2 the Public Expenditure and Financial Accountability Assessment (PEFA) informed financial management arrangements and defined an action plan to address weak areas. Training in procurement using OECD indicators was carried out during project preparation. The procurement action plan to strengthen institutional capacity in GDF was to be finalized during the first semester of implementation. 18. The Government demonstrated a high degree of commitment to the operation at the highest levels. In addition, the administration was perceived as truly reformist and ready to implement an innovative project. Project preparation was led by the SEPLAG, and an effective preparation team was assembled. The Governor and SEPLAG’s Secretary took the lead in many of the discussions on the scope of the Project. Together, they convened the other sectors and obtained their commitment and participation. The Government’s team prepared a detailed Project proposal (Carta Consulta) that served as the basis for the development of the PAD. Focal points from the three sectors (education, health and transport/DFTRANS) and SEPLAG worked closely in the definition of Project indicators and the selection of EEPs to be supported by the operation. Overall, GDF’s preparation team was very responsive and proactive in its work with the Bank team, all of which contributed to a timely Project preparation. 19. The lending instrument and approach adopted was in line with the CPS of 2008-2011. Bank management viewed the SWAp as a cutting-edge approach and encouraged risk taking with regards to RBM-based reforms. Partly inspired by the success of the Minas Gerais “Partnership for Development� Development Policy Loan (DPL) that introduced RBM in all state Government secretariats, SWAps adopted the RBM aspects and tried to introduce more flexible disbursement mechanisms, while also supporting local procurement and financial management procedures, whenever acceptable to the World Bank. 2 A social assessment was carried out, which provided useful inputs on the service delivery and local management aspects of the programs financed. A Quality Enhancement Review (QER) was carried out yielding important feedback to the preparation team. Financial Management (FM) and Procurement capacity assessments were carried out. 5 20. The adoption of RBM in PSM was the cornerstone of the Project. The signing and evaluation of RAs were included as DLI to give them priority. But, there was no discussion in the PAD as to how RAs would be evaluated or how they would be tracked. M&E capacity, crucial for the monitoring of results, was not assessed. Given this Project linked disbursements to results, the assessment and design of an M&E system should have been given greater attention. As no outcome indicator was included to measure changes in public sector management derived from the signing of RAs, it is difficult to determine if management practices changed even if the indicators are met. 21. At the time of Project preparation only one multi-sector state-level SWAp had already been completed (Ceará) and thus lessons learned from implementation were limited.3 Some of the fiduciary arrangements had not been ironed out within the Bank, which led to shortcomings in Project design. For instance, the algorithm linking disbursements to results was too complex, and somehow failed to effectively link disbursements to results - the Project would disburse once a year regardless of whether any sectoral indicators (other than global indicators) were met. There was lack of clarity on whether the Bank procurement rules should apply to the full program or only to the financed share. On the legal side, while the Project called for the hiring of an independent annual technical audit to verify the data submitted by the Government, particularly for DLIs, the Loan Agreement did not make disbursements contingent on the review by such independent technical audit. Thus, disbursements took place without validation from a detailed independent review. 22. Finally, insufficient caution was exercised in Project design to protect it from possible threats to good governance. Much was made of the administration’s improved fiscal prudence and reformist approach. While this seemed an important positive break from the previous administration known to have had a history of corruption, the recentness of the change called for the team to make more in-depth assessments of implementation and fiduciary risk. The only related risk identified in the risk matrix was the possibility that a new administration might not follow up on the reforms. 2.2 Implementation 23. One month after Project effectiveness, 4 there were serious allegations of corruption involving the executive, legislative and judicial branches of Government. The scandal led to the arrest of the Governor. Soon afterwards the Deputy Governor, who took his place in February 2010, resigned. By January 2011, when a newly elected Government came on board, there had been five different Governors since the Project had been approved. The corruption allegations were not related with this or another Bank- financed project in the DF, and the Bank’s thorough fiduciary review following the scandal did not uncover any wrongdoing on Bank-financed contracts in the DF. 3 The main lesson from the Ceará project was that indicators needed to be carefully selected, ensuring they are clearly defined and of high priority to the Government to affect the Project’s success. 4 The Project was approved on April 9, 2009, signed on September 10, 2009 and declared effective on October 15, 2009. 6 24. These events5 essentially paralyzed the Project until about May 2011, and caused a major blow to implementation from which the Project would never recover. Each change in Governor led to changes in secretaries, and to changes in both staffing and administrative structure in each of the sectors, and often to some changes in policy. The State Secretariat of Education (Secretaria de Estado de Educação -- SEE) had six different Secretaries since the Project was prepared, Health had five, Transport had four, and DFTRANS had five. More importantly, the Secretariat leading the reform, SEPLAG, had seven.6 All these changes resulted in delays given new staff had to become acquainted with the Project, and to a total loss of ownership. 25. The Project Coordination Unit (Unidade de Coordenação do Projeto -- PCU) team also underwent changes, and its isolation under subsequent administrations reduced the Project’s potential to succeed. The team’s composition and leadership changed in March 2010 and its location within SEPLAN changed as well. The PCU went from reporting directly to the Secretary of SEPLAN to reporting to the Assistant Secretary of Management Modernization within SEPLAN. It had limited access to decision makers within SEPLAN and reduced convening power vis-à-vis the other implementing partners. A subset of the March 2010 PCU team remained in the team until Project closing. 26. In addition, the PCU lacked capacity to assume all of its responsibilities as defined in the Project’s Operational Manual (OM). The PCU team was to monitor and evaluate Project activities, both technical and financial, including following up on the achievement of agreed targets. Regular supervision meetings did not take place outside of Bank missions. PCU did not review data submitted by sectors, which were inconsistent over time and some were inconsistent with protocols in the OM. All these factors did not help to rebuild a sense of ownership amongst sectoral implementers. 27. Deficiencies were found in Project management, FM, procurement and M&E. There was an inability to effect procurement of goods and services under the Bank guidelines and delays in carrying out the financial audit. The needed procurement specialist for the Project was never hired, nor was the required independent technical audit consultancy. 28. The seriousness of the situation was reflected in the Project’s poor performance as measured by the PDIs in Component 1, and the lack of any progress in Component 2. After the first year, when a total of seven out of 25 indicators were met (28 percent), achievement of targets declined to five (20 percent) from 2010-2012. None of the activities identified in the PAD under Component 2 were carried out. There were long delays in preparing an acceptable procurement plan (approved in September 2010 conditional on some adjustments). Loan funds were not advanced until April 2011, and it took four more months (August 2011) before the team was able to utilize the funds. The 5 Refer to timeline of events in Annex 2. 6 SEPLAG’s name also changed, becoming the State Secretariat of Planning and Budget (Secretaria de Estado de Planejamento e Orçamento – SEPLAN) in January 2011, which is how this ICR will refer to it from this point on. 7 law authorizing the Bank loan only allowed the use of funds for investment, excluding technical assistance, and the Government team did not make the necessary amendments to the law prior to the arrival of the funds. Further delays ensued in the procurement of consultancies because of lack of adequate Terms of Reference (TORs). And, when TORs had been approved, there were often further delays as procurement processes required the approval of the DF Attorney General’s office (Procuradoria Geral do DF), generally not acquainted with Bank procurement procedures. 29. Some progress was made on Component 2 after GDF requested Bank support to develop programs and capacity in its State Secretariat of Transparency and Control (Secretaria Estadual de Transparencia e Controle -- SETC), created in May 2011. The Bank agreed and some training for auditors was financed, as were 15 individual consultancies mainly designed to evaluate options, develop methodologies, and develop training materials. On request by SEPLAN, SETC carried out an ex-post technical audit of Project indicators in the end of 2012. In 2012, the number of implementing secretariats under this component was expanded, although many never carried out the planned studies. Those that did included the State Secretariat of Government (Secretaria de Estado de Governo) and the State Secretariat of Social Development and Income Transfer (Secretaria de Estado de Desenvolvimento Social e Transferencia de Renda -- SEDEST). The State Secretariat of Government carried out 13 individual consultancies, 10 of which focused on social participation. Some of the studies are not related to Project objectives. 30. By Project closing, only 19.7 percent of the loan was utilized. GDF sent its first request for retroactive financing for US$26 million in December 2009 soon after the corruption scandal. Given the risk involved, the Bank obtained agreement from the newly elected administration in 2011 to not claim reimbursements for the years 2008 and 2009, as well as the first semester of 2010, in order to completely de-link the Project with the past administration. Thus, there was no retroactive financing. Total disbursements for the Project amounted to US$ 26,064,934. (Refer to Annex 1 for costs by component). 31. The Bank moved quickly to assess to what extent its portfolio with GDF was affected, carrying out a detailed fiduciary assessment that included a complete review of all procurement processes and financial management reports, including 11 contracts with five companies that had been named in the corruption case. Although the reviews related to the Project were not qualified, the Bank did not consider any disbursements until after the newly elected administration came on board in January 2011. 32. The Bank also considered triggering the cross-suspension remedy in the General Conditions in May of 2010, when the Brasilia Sustentável Project was suspended, because of the numerous changes in Government authorities, and the lack of implementation progress. However, after some discussion it was decided that the Government taking office after elections should be given a chance to move the Project forward. Afterwards, lack of evidence of non-compliance with legal covenants due to insufficient recording of Project issues in Project files impeded any legally-related action. 8 33. A mission visited Brasilia in February 2011 to meet with new authorities and implementing teams, and assess to what extent the Project was still relevant to GDF’s priorities. They also set out to work with the PCU to establish a minimum operating environment, meeting with new staff in participating sectors to disseminate information about the Project, and providing training for GDF staff on FM and procurement. 34. On the technical side, the mission found that objectives and most activities were still relevant, and the incoming Government was interested in re-launching the Project, extending it by 18 months and adjusting some of the indicators and their targets. During that mission, the Bank team worked with the GDF team to review adjustments needed to Project indicators. A subsequent mission in June 2011 reviewed the proposed Project indicators and their targets and found them to be in line with Project objectives. It also worked to include other secretariats under Component 2. It was agreed the formal request for restructuring should reach the Bank for its review and approval soon after the mission. 35. Despite the fact that a new Government was in place, implementation did not improve and important shortcomings on Bank supervision were also observed. 7 Even under a new administration, delays in the preparation of progress reports and IFRs continued for the duration of the Project. The request for restructuring was not submitted until October 20, 2011, after the Bank had held an internal review of the Project and decided not to restructure it due to its poor performance.8 In 2012, there was minimal supervision activity, other than procurement approvals for studies under Component 2, which is understandable given the decision made in December 2011 to not extend the Project. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 36. The Project utilized data already available to GDF. Although indicator protocols were generally fairly well defined, there were several indicators that lacked sufficient detail, and ended up being improperly calculated in progress reports. A results framework identifying the levels of the various indicators (key outcomes, and intermediate outcomes and outputs) and their links to each other and to the PDOs was not prepared. The first PDO lacks outcome measures and the second PDO lacks outcome or output measures for some sectors, and those included are not necessarily linked. A system to track compliance with RAs was needed to assess the gains resulting from the use of RBM, but this was not contemplated in the design. 37. The PCU was tasked with monitoring Project progress, ensuring the quality of the data submitted, and building the capacity of the different sectors for M&E, but overall M&E capacity was insufficient. Each secretariat assigned a focal point to oversee Project activities, and the data were put together by each secretariat’s statistical unit. Monitoring of implementation was limited, and the quality of the data was not 7 Refer to section 5.1 (b). 8 A formal letter was sent to GDF in December 2011 to that effect. 9 reviewed. Given the frequent changes in staffing that took place at the participating secretariats, many incoming staff did not understand the protocols for many indicators, and the PCU did not ensure they did. The Project allocated funds for training in strategic planning, definition of indicators, and M&E for staff of SEPLAN and the participating secretariats through Component 2, but no training was provided. 2.4 Safeguard and Fiduciary Compliance 38. None of the Safeguard Policies are triggered by the Project design, as envisioned construction was expected to consist of minor repairs to schools and health clinics. 39. The Project operated in a high fiduciary risk environment, difficult to predict, soon after effectiveness, although financial management risk was assessed to be moderate and procurement risk as average at the time of appraisal. 40. Weaknesses identified during Project preparation and the PEFA assessment were to be addressed through an agreed FM Action Plan to be financed through Component 2.9 The plan left key actions related to improving internal controls to the first and second years of implementation in a three-year project. Given that the Bank had limited experience working with GDF this might have been overly optimistic. 41. Weaknesses in GDF’s financial management capabilities resulted in important delays. By March 2011, the first audit report (2008-2009) had yet to be carried out. The Bank agreed to waive that audit report and for the first six months of 2010, as there were no Bank-financed expenditures during the period. The Bank agreed to have the Federal District Court of Accounts (Tribunal de Contas do Distrito Federal -- TCDF) carry out the external audits. The audit for the second semester of 2010 was furnished with a significant delay, mainly because the Bank had delayed more than four months in giving feedback to TCDF on the audit TORs, which were not approved until November 2011. The 2011 audit was submitted in November 2012, and the report noted that several weaknesses detected in 2010 had not been addressed, most related to weaknesses in the PCUs ability to manage the finances of the Project. The PEFA to be carried out under the Project, to complement the one from the design phase, was never executed. 42. A draft Procurement Action Plan, which was to be finalized during the first semester of implementation and financed through Component 2, was prepared on the basis of the Procurement Assessment. The assessment included only SEPLAG’s Central Bidding Committee (Central de Compras -- CBC) because it would carry out all procurement processes under the Project. The CBC also suffered significant staff rotation 9 The FM assessment carried out during preparation noted that GDF had satisfactory FM arrangements to meet Bank’s requirements, and indicated that some aspects of FM arrangements were still under development. It noted weak internal controls in the four implementing entities, and the need to strengthen SEPLAG’s coordination and monitoring and evaluation capacity. The use of multiple non-integrated software systems weakened the reliability of reporting. 10 due to the corruption scandal, and loss of staff (and later a restructuring). No procurement was carried out under Component 1 until its final year because of lack of procurement capacity. Delays in the approval of procurement plan delayed access to Component 2 funds. In addition, DF’s Attorney General’s office had to approve all procurement and they were unaware of agreed procedures. 43. Neither FM nor Procurement teams at the Bank carried out a reassessment of FM and procurement capacity after the significant change in staff following the corruption scandal. 2.5 Post-completion Operation/Next Phase 44. Although originally envisioned as the first in a series of multisector SWAps, no follow-on operation will take place. GDF, however, through the Sub-Secretariat of Management Modernization of SEPLAN, is continuing with its RBM program, using a modified instrument. Although several of the previous RAs have not been renewed, some new entities have been included. According to a report from the Coordination of Institutional Planning unit (Coordenação de Planejamento Institucional -- COPLAN) who is leading this effort within the Sub-Secretariat of Management Modernization, RAs are monitored every three months and evaluated annually. They will soon begin piloting software used in the Federal Government to monitor RAs. Of the second generation of RAs to be signed, health will be the first to use the new software. 45. Financing for the three sectors continues as before the Project was implemented, and thus sectors will not have the additional funds needed to continue to expand coverage. Given the limited expansion achieved in most programs, recurrent costs will not be much higher than before. Most programs supported in education, many of which also received federal funding, were modified and or discontinued under subsequent Secretaries of Education. Given federal funding is tied to the implementation of federal programs, some of the funding for GDF may have been lost. Most programs financed in Health receive federal funding and thus support for FHP and the implementation of the health card will continue. Transport is introducing a different subcontracting approach with bus companies that is expected to give it the control of operations and revenue it needs, and is now preparing bidding documents for a CCO that will function under the new system. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 46. Overall relevance of objectives, design and implementation is rated as moderate. Relevance of objectives is assessed as high, as the objectives of the Project continue to be relevant today and consistent with GDF’s stated priorities. The objectives are also consistent with the 2012-2015 CPS, which includes improving the efficiency of public spending and the increased coverage and quality of public services for low-income households as two of its four strategic objectives. Relevance of design is assessed as moderate given the insufficient focus of the Project on overall management improvement 11 beyond the signing of RAs, insufficient focus on M&E capacity and arrangements, complicated disbursement arrangements, and lack of alignment between Project design and current education sector priorities where policies are still in flux. Implementation relevance is low given the limited relevance the Project had in SEPLAN’s overall development plans and objectives after the initial administration departed. 3.2 Achievement of Project Development Objectives 47. The Project had two objectives. The table in Annex 2 contains annual measurements for all PDIs. PDO 1: Improve public sector management and accountability: Not achieved. 48. There are six output measures linked to PDO 1. The first two have to do with the signing and annual evaluation of RAs signed between the Governor and the various secretariats and agencies implementing Government policy. While the number of RAs to be signed did meet its expected target of 26, and the number assessed annually reached 20, close to the target of 21, only 20 of the original 26 RAs were in active status by the end of the Project. There was no measure of the management impact of signing and/or evaluating RAs nor were any additional linked activities included to get a better sense of any management changes taking place and thus their impact is unknown. Moreover, it is unclear that the Project had any influence in the RA program. None of the other outputs related to PDO 1 (the implementation of the FM and Procurement action plans, and the introduction of incentives for good performance in education and health sectors) were carried out. PDO 2: increase access, quality and efficiency of the education, health and public transport services): Not achieved for any of the three dimensions. 49. Access: Overall, the PDO on access was not achieved as only two out of seven indicators were achieved. Increasing access was an objective for both education and health. In education, access was increased as measured by primary school completion rates, but declined as measured by secondary school completion rates. The latter were lower at the end of the Project (66 percent) than when it began (75 percent). Increased enrollment in ECD programs and greater availability of full time schools were also to reflect increased access. ECD enrollments also decreased over the life of the Project, and no full time schools were established in the areas targeted by the Project. In health, increased access would be reflected in FHP coverage expansion, and greater coverage of prenatal care and Pap smear tests. FHP only reached half of the coverage expected and coverage of Pap smear did not change. Coverage of prenatal care did increase coverage even more than planned. 50. Quality: The PDO on quality was not achieved, as only one out six indicators was achieved (but the data are unreliable). Improved quality of education was measured by an improvement in the Basic Education Development Index (Indice de 12 Desenvolvimento da in Educação Básica – IDEB)10 of two percent each of the 125 low performing primary schools selected in low-income areas. Of the 125, 66 schools met this target. Increased primary and secondary school completion rates would also reflect improved quality of education but as noted above, the later did not increase. The programs included in the Project to improve quality (i.e. the Gestão Compartilhada program and the full-time school program) were either discontinued or modified and thus did not have the expected results. In health, quality of FHP services was to be measured by a decrease in hospitalization rates for diabetes mellitus 11 and increase in infant death investigations. For the first indicator, data varied considerably over time as the measurement was based on a different set of hospitals in each measurement, some of which did not provide services to treat diabetes and thus is considered unreliable. The second one did increase from 40 percent to 64 percent but still significantly lower than its 90 percent target. In transport data from the indicator measuring improved quality (reduction of complaints) is unreliable because DFTRANS changed the way it handled complaints mid-project, and thus the data are not comparable over time. 51. Efficiency: The PDO on efficiency was not met, as only one out of seven indicators was achieved. In education, improved efficiency was measured with an index based on four indicators12 that were regularly monitored in the Gestão Compartilhada program, and by decreases in age-grade distortion (which would result in increased efficiency by moving students through the system faster reducing the overall costs per student). The Gestão Compartilhada program was modified in subsequent administrations and the monitoring system was dropped, and both primary and secondary school age-grade distortion programs were dropped as well. Thus, there are no data measuring efficiency of school management and data on age-grade distortion in both primary and secondary school decreased only about a third of what it was supposed to have decreased. In health, the indicators for improved efficiency included the decrease in hospitalization rates for diabetes mellitus, and the establishment of management the Santa Maria Hospital by an OSC, which was cancelled after the change in administration. A third indicator, the expansion of coverage of the health card, was met (surpassed) and while it is an important input to a system to improve efficiency, it does not reflect improved efficiency (it is an output measure). And in transport, the indicator on efficiency, the trip and itinerary regularity index, is unavailable as the contract with the firm managing the information system was cancelled after the corruption scandal and the CCO was never contracted. An output indicator which would contribute to improving efficiency, i.e. preparing a transport master plan, was met. 10 IDEB is composed of test results of student performance in schools and grades, as measured by the National Assessment Tests (Prova Brasil) and indicators of student flow (promotion, repetition and dropout rates). It is carried out every two years by the Ministry of Education. 11 The indicator is measured as the number of hospitalizations due to diabetes mellitus complications divided by the total number of hospitalizations (excluding child birth) in the same place and period. 12 The indicators were: (i) decrease in student absenteeism; (ii) decrease in teacher absenteeism; (iii) decrease in student dropout rates; and (iv) full participation of School Council as informed by the school monitoring system. 13 3.3 Efficiency 52. An updated cost-benefit analysis was not prepared after Project completion because so many activities were discontinued or were never introduced. Efficiency is rated as low. 3.4 Justification of Overall Outcome Rating Rating: Unsatisfactory 53. There were major shortcomings in the operation’s achievement of objectives. Its relevance is moderate and its efficiency low. 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development – not applicable (b) Institutional Change/Strengthening – not applicable (c) Other Unintended Outcomes and Impacts (positive or negative) – not applicable 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops Not applicable 4. Assessment of Risk to Development Outcome Rating: Moderate 54. Risk to development outcome differs by sector. In health the risk to development outcome is low. In transport it is moderate. In PSM and education the risks are significant. 55. In PSM, as implemented through RBM, the most important risk to development outcome is political. The institutionalization of RBM is weak and thus very vulnerable to changes in commitment caused by changing authorities. Although new entities have been included in the process, and new software will be used to monitor RAs, not all who signed agreements are being evaluated. 56. In education, most programs supported by the Project were modified and or discontinued under subsequent administrations. The deterioration of many of the education indicators over the lifetime of the Project clearly shows this. The most important risk to development outcome in the education sector is also political. Every new administration has made significant changes to education policy. 57. In health the programs were all institutionalized before the Project began as they are Ministry of Health initiatives and receive some federal funding. Thus the focus on increasing access, quality and efficiency will continue, even if expansion is slower. 58. In transport, although little was achieved through the Project, their focus remains on improving quality and efficiency but some of the initiatives have now been rethought. The GDF is introducing a different subcontracting approach with bus companies that are expected to give them the control of operations and revenue that they 14 need, and are now preparing bidding documents for a CCO that will function under the new system. However, lobbying by powerful bus companies could derail their attempt to gain control of the operations of public urban transport. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Unsatisfactory 59. Bank performance is rated as Moderately Unsatisfactory. Shortcomings in Project design were mainly in fiduciary and M&E arrangements, both of which constitute crucial elements of well-functioning SWAps. The disbursement algorithm linking disbursements to results was complex. Moreover, disbursements were not made conditional on the contracting of the external technical audit. M&E capacity, crucial for the monitoring of results, was not assessed. Given this Project linked disbursements to results, the assessment and design of an M&E system to track results should have been given greater attention. And, given the importance that the process of evaluation of RAs may have in generating management change, greater focus should have been given to it in the design. Finally, governance risks were underestimated. The team took some risks adopting an innovative approach that had been insufficiently tested, which at the time both the Federal Government and the Bank encouraged. However, the shortcomings proved to have a significant detrimental effect on Project performance. (b) Quality of Supervision Rating: Unsatisfactory 60. Handling the corruption scandal posed a significant challenge for the supervision team. The continuous changes in Government authorities from that moment until new elections were held meant there was no effective counterpart to deal with, making it difficult to supervise. Nevertheless, there were several aspects of supervision that had major shortcomings even after the new Government came on board in 2011. 61. Despite the fact disbursement was linked to results, supervision visits after June 2010 did not report on PDIs. With the exception of supervision of the transport sector, which prepared separate Aide Memoires, it is unclear when and how data submitted were reviewed. Given that no technical audit was contracted, and that disbursements were conditioned on results, the supervision team should have given this issue greater attention. None of the Aide Memoires discuss the content of either the RAs or their evaluation (although the first Aide Memoire notes it should be reviewed), when they were a key focus of the Project. 62. An important shortcoming of supervision was the insufficient documentation on Project progress and issues. There are only three Project aide memoires available for 15 the 3.3 years of Project implementation 13 . A partial, unfinished Aide Memoire was available for a fourth mission (June 2011). Although interviews with the PCU revealed brief two-day visits by the Bank had been made, no record of them exists in Project files. Some visits did not include sectoral team members, except for transport. Because there is only a limited amount of information in Project files for the last two years of supervision, it is not clear what was addressed by the Bank team. 63. Response times with respect to FM issues were slow. There were significant delays in the approval of the final TORs for Project audits. FM also did not always record important decisions and sometimes did not communicate them to the team. 64. The Bank team did not carry out a full reassessment of capacity in procurement and FM once the Government was in place. FM did carry out a desk review to monitor the continuing adequacy of the Project’s FM arrangements. In hindsight, this would have been a good entry point for discussions with GDF on setting up a minimum environment for Project implementation. (c) Justification of Rating for Overall Bank Performance Rating: Unsatisfactory 65. Since Bank performance in ensuring quality at entry is Moderately Unsatisfactory and quality of supervision is Unsatisfactory, overall Bank performance is rated as Unsatisfactory. 5.2 Borrower Performance (a) Government Performance Rating: Moderately Unsatisfactory 66. The Government includes the Governor’s office, SEFAZ and TCDF. While the Government in office during Project preparation was very committed to the Project, subsequent authorities did not share its enthusiasm and did not prioritize the Project. SEFAZ was well staffed and had generally good performance, respecting the segregation of duties, but with no satisfactory internal controls. Two Project audits were carried out by the TCDF (a third one is under preparation), and were well prepared. Both were delayed, although the first one was due to the Bank’s delay in approval of TORs. (b) Implementing Agency or Agencies Performance Rating: Highly Unsatisfactory 67. Implementing Agencies performance is rated as Highly Unsatisfactory because of severe shortcomings in the implementation of the Project. Implementing 13 October 2009, June 2010 and February 2011. 16 entities include the SEPLAN and its PCU, and the three sector State Secretariats (SEE, SES, and SET/DFTRANS). The loss of priority of the Project under the current Government, and SEPLAN in particular, is a key reason for its poor performance. The Project was not on SEPLAN decision-makers’ radar screen. Project implementation was severely affected by the constant change of authorities and staff after the corruption scandal. The loss of ownership amongst all implementers was significant. 68. Project ownership on the part of implementing secretariats was low, particularly in Education where political differences with other authorities continued to interfere with policies and programs. 69. Sectoral secretariats had difficulties in submitting consistent monitoring indicator data and financial data to the PCU. Procurement capacity was also weak and the only procurement carried out by the sectors was in Health during the final year of the Project. Given the PCU did not provide the needed support to them, as defined in the Project Operational Manual, problems with data persisted to the end. 70. Project management was deficient, and the planning function at the PCU was weak. Coordination meetings with the sector state secretariats only took place at very early stages of Project implementation, and thus the only forum available to build ownership, encourage cross-fertilization, and generate cohesion was not used. Project progress reports prepared by the PCU were frequently delayed, as were IFRs. No progress reports were prepared for the period January to June 2011 and for July- December 2012. Poor planning also caused delays in access to funds from Component 2 once they were available. Follow-up of project issues with the Bank frequently experienced significant delays, as was evident with the submission of the request for restructuring – eight months after discussions with the Bank had been initiated. Financial management also had some problems although many of them were due to deficiencies in the data submitted by implementing secretariats. Issues noted in the first audit were not addressed in the next audit (only two audits have been carried out to date). Although the PCU had no procurement capacity it never hired the necessary support. (c) Justification of Rating for Overall Borrower Performance Rating: Highly Unsatisfactory 71. Given Government performance is rated as Moderately Unsatisfactory and implementing agency performance is rated as Highly Unsatisfactory, Overall Borrower Performance is rated as Highly Unsatisfactory. 6. Lessons Learned 72. Strong Government and implementing entity ownership and capacity are critical for effective design and implementation. GDF was very proactive in the design stage, but ownership was lost when implementation began, due to changes in administrations. 73. Project design needs to introduce safeguards to protect the Project in case of a change in political situation, particularly when taking risks with a new instrument 17 and new counterparts. Specific conditionalities can be introduced to shield aspects of the Project that could significantly affect Project outcomes if not implemented as agreed. Preparation teams tend to be overly optimistic, particularly when faced with effective and reformist counterparts, and forget that Government administrations change periodically, bringing with them changes in priorities and staffing. 74. Assessing changes in management practices has to go beyond the signing and evaluation of RAs. RBM was a cornerstone of this operation. There is little if any involvement in the monitoring of those RAs or their evaluation. And if there are no outcome measures to assess change in management practices, DLIs can be met without having an impact on the PDO. 75. M&E should be part of fiduciary review package, particularly if disbursing against results. This would include preparing a solid log frame; developing measurable indicators; and establishing viable M&E arrangements. Moreover, the hiring of the technical audit should carry the same weight as the hiring of the financial audit. And, it should be made conditional on disbursements. 76. The “flexibility� expected through the Sector Wide Approach added more complexity and limited the ability of implementing secretariats to make the necessary investments. Bank fiduciary rules and reporting systems remained a challenge even though the Project design intended to introduce more flexible arrangements. Considerable difficulty was experienced in the identification of EEPs for reimbursement. At first, difficulties arose because the Bank demanded that contracts to be reimbursed must contain the anti-fraud and anti-corruption clause that ongoing contracts did not have. Later, only non-procurable expenditures could be reimbursed because of the GDF inability to carry out procurement under Bank rules. 77. Consistent recording of project events and issues is necessary for management to be able to make informed decisions regarding projects. The Project team explored avenues to suspend disbursements when it was evident Project performance would not improve, but was unable to do so because it needed evidence on non-compliance of covenants, which was not recorded. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies – not applicable (b) Cofinanciers – not applicable (c) Other partners and stakeholders – not applicable 18 Annex 1. Project Costs and Financing (a) Project Cost by Component (in USD Million equivalent) Actual/Latest Appraisal Estimate Percentage of Components Estimate (USD (USD millions) Appraisal millions) Component 1: Strengthening Results-Based Management in the Public Sector (Total) 130.0 24.48 18.8% Education 48.0 8.82 18.4% Health 48.0 11.79 24.6% Transport 24.0 3.88 16.1% Component 2: Building capacity in the Public Sector (Total) 9.675 1.255 * 13.0% State Secretariat of Transparency 0.61 State Secretariat of Government 0.20 State Secretariat of Social Dev. 0.03 State Secretariat of Planning & Budget (SEPLAN) 0.004 Total Project Costs 129.675 25.735 19.8% Front-end fee IBRD 0.325 0.325 100.0% Total Financing Required 130.0 26.06** 20.0% *The subtotals under Component 2 do not add up because final expenditures by individual State Secretariats were never submitted. **Because disbursement transactions had not been completed yet as of the date of this ICR, this amount may not be the final one. (b) Financing Appraisal Actual/Latest Type of Estimate Estimate Percentage of Source of Funds Cofinancing (USD (USD Appraisal millions) millions) Borrower 270.00 940.00 348.0% International Bank for Reconstruction and Development 130.00 26.06 20.0% 19 Annex 2. Outputs by Component Achievement of Targets 1. The Project’s poor performance was reflected in its limited achievement of Project Development Indicators (PDI) targets. The PAF indicator (global indicator 1) was met each semester throughout implementation. Excluding the EEPs for the period prior to effectiveness, the second global indicator (executing at least 70 percent of the EEP budget) was met only once by two sectors (health and transport). Education never met this target. There were three annual measurements of all PDIs (including Disbursement-linked Indicators - DLIs) (2009, 2010 and 2011) and an additional measurement for 2012 was available for all, except some education indicators that are based on the education census only published in May for the previous year. The number of DLIs achieved per year (2009-2011) out of a total of 11 (excluding global indicators), was five, three, and three, respectively. Regarding the other PDIs, this excluding an indicator on hospitalization for diabetes mellitus with unreliable data (see M&E section), the number of other PDIs met, out of a total of 14, was two, two and two, respectively, for the three years. Thus, after the first year when a total of seven out of 25 indicators were met (28%), achievement of targets declined to 20 percent in 2011. Component 1: Strengthening Results-Based Management (RBM) in the Public Sector 2. Component 1 sought to institutionalize RBM in public administration overall, and more directly by supporting critical public sector management reforms in education, health and transport sectors. The component was structured as a Sector Wide Approach (SWAp), financing a percentage of the selected programs (EEPs) in the three sectors. To receive their full disbursement, sectors had to execute 70 percent of budgeted EEPs and comply with all their DLIs. Other than in 2009 when EEP expenditures reported included retroactive expenditures (in the disbursement that was effected), education never reached the 70 percent and health and transport only reached it once. Moreover, a total of seven PDIs met their targets on the first year and only five thereafter. This included only three of 11 DLIs. 3. Public sector management and accountability were to be strengthened by expanding RBM practices and improving fiduciary oversight. The latter was to be achieved through training in strategic planning and project management, and M&E, establishing cost-account systems (for the participating sectors), and strengthening procurement and financial management all through Component 2. None of the activities under Component 2 was carried out. 4. The adoption of RBM, including the signing of annual Results Agreements (RAs) between the Governor and each of the line secretariats and agencies of the DF, and the submission of annual reports on performance with respect to agreed targets, were expected to improve the overall efficiency and effectiveness of public spending, and increase accountability in all sectors. The expected outputs for this sector were the signing of 26 RAs with the various secretariats and Government entities, and the annual 20 evaluation of signed RAs at the end of each year of implementation. The expected 26 RAs were signed by 2011, and an additional seven (new entities) were signed in 2012 covering the new Plano pluri-annual (PPA) for 2012-2015. RAs were signed for a three- year period, and thus those that signed in 2008 (health) and 2009 (10 others) should have signed a second agreement in 2012. None of the overdue RAs were renewed. In fact, an annual report by the Institutional Planning Coordination (Coordenação de Planejamento Institucional, COPLAN ) notes that in 2012 there were only 20 active RAs. 5. Progress with regards to the evaluation of RAs was less than expected. By December 2011, 21 were to have been reviewed. Only 20 had been reviewed in 2012. The evaluation of RAs was not the subject of supervision missions and thus there is no information as to whether or not the implementation of RAs, and the manner in which they have been applied have improved management. 6. GDF intended to introduce rewards (e.g., additional budget, bonus payments) for those entities that met or exceeded their performance targets in health and education. In education, it was linked to the Gestão Compartilhada program but that program was dropped. In health, it was linked to performance on individual RAs signed between the health Secretary and health units. However, neither of the incentive programs was implemented. 7. In education, it was expected that the expansion of the Gestão Compartilhada program would improve efficiency, quality and performance. Expanding access to early child development (ECD) programs and the Escola de Tempo Integral program for basic education (full-time schools which provide longer and better learning opportunities that traditional schools) were also expected to improve access to education (reducing inequity) and improve quality. The focus of these investments was to be low-income areas and areas with low Basic Education Development Index (Indice de Desenvolvimento da Educação Básica – IDEB) scores. And programs designed to reduce age-grade distortion were to improve efficiency of the education system, by moving students through the system faster, thus reducing costs). However, ECD enrollments declined over the lifetime of the Project, the full time education program was modified so by the end no schools were located in low-income areas, the focus of Gestão Compartilhada was modified and the monitoring instrument was discontinued, and the programs designed to reduce age grade distortion were also discontinued. The number of primary schools students with age-grade distortion was supposed to decline from over 92,000 to 17,000 but in fact only declined to about 70,000. Secondary age-grade distortion declined from 34,805 to 28,556, far above the 3000 expected. These events contributed to the decrease in secondary school completion rates and the leveling of primary school completion rates. Overall performance of activities under the State Secretary of Education (Secretaria de Estado de Educação – SEE) was poor. The SEE’s new administration, which has rejected the programs developed by the previous administration that have been well tested-throughout the country, is still in the process of defining several of its programs. 21 8. In health, a RA had been signed with the Governor, and subordinate RAs were signed with health units in January 2008. Improvement in access and quality focused on the expansion of primary care through the Family Health Program (FHP) was an integral part of the RA. The expansion of FHP would strengthen prevention and help resolve health problems before they reached the hospital, thus decreasing patient overload in hospitals while improving service quality and health outcomes. The latter would also reflect a more efficient use of health resources. Efficiency would also be improved through the introduction of a computerized managerial and clinical information system fed by information contained in patient’s electronic health cards (Cartão de Saúde) and through the piloting of the subcontracting of the management of the Hospital Sta. Maria to a non-profit social organization with the purpose of improving hospital management. 9. In order to expand the FHP, GDF needed to establish more health units and equip them. Because the project did not finance infrastructure, the program could not expand. Instead, it planned to use reimbursed funds to then build the needed centers. But, because each time they did not meet their targets, they received even less funding, expansion only became more difficult over time. In the last year they did finally procure equipment for some new centers and thus some expansion will be possible. During the lifetime of the Project, however, FHP did not expand sufficiently. It was supposed to reach 40 percent coverage and only reached half as much. Coverage of prenatal care, Pap smear tests and investigations of infant deaths are all carried out as part of the FHP. Coverage of prenatal care reached its target, but it had already reached that value at Project launch. Coverage of Pap smears and investigation of infant deaths did not increase sufficiently because they are linked to FHP expansion. Coverage of the population with the Cartão de Saúde surpassed its target. And, while use of the cartão is an important input to improve efficiency in the sector, improved efficiency itself was not measured. The other measure of improved efficiency, the subcontracting of management of the Santa Maria Hospital, was not met because, although the hospital was up and running under management by a non-profit, the contract was rescinded in 2011. And finally, the decrease in hospitalization due to diabetes mellitus was incorrectly measured and cannot be used. 10. In transport, effectiveness and efficiency of State Secretariat of Transport (Secretaria de Estado de Transporte – SET) and Federal District Transport Authority (Departamento de Transporte Urbano do Distrito Federal – DFTRANS) would be increased through the establishment of a transport Control Center for Operations (Centro de Controle Operacional –CCO), the preparation of an urban transport Master Plan, the review and improvement of internal processes, the modernization of management information systems and equipment, and building capacity of staff. The Master Plan was to define policy and future investments for the sector. This was already under preparation when the Project began and was actually completed and approved by May 2011. The other two key areas of focus in transport were the establishment of a CCO and of an integrated transport database that would allow DFTRANS to manage operational and revenue data. Although a law on the new regulatory framework for public transport services required operators to provide a data collection and transmission subsystem as part of the automated fare collection system and 22 the automated ticket readers in the buses, bus operators retained control of the ticketing data. A CCO was to be installed to allow DFTRANS to regain control over the operation of public transport services and ensure that operators comply with the operational programs and their other obligations. 11. The transport sector was one of the most affected by the corruption scandal as several activities relied on information technology (IT) consultancies, which was the sector where much of the corruption was found. The CCO was never contracted, and until end of 2011, they never had the control of the ticketing data. The data, however, were not available in 2012 as there are delays in submission. Data on the regularity index also ceased to be available. The regularity index, which compares planned services with services effectively provided by operators, was calculated using data provided by bus operators. A private firm was contracted to manage and maintain the transport information system, but the contract was cancelled after the corruption scandal. Thus, the data were not available after this. It was expected that by Project end the data would be derived from the CCO. And, the indicator on complaints, which was based on a database that registered all complaints, cannot be used as DFTRANS changed the way it handled complaints, not registering all of them and thus the fall in number of complaints is due to lower reporting rather than fewer complaints. 12. Although not under the Project, the transport sector has made some important advances. A new master plan was developed that incorporates both the CCO and the fare and ticketing data base under one integrated system. Given the bus company lobbying, they decided to subcontract bus services by requesting bids for coverage of specific areas rather than bidding for bus fleets, and employing contracts specifying quality and coverage parameters. Two out of five contracts have now been signed. 23 Indicators are shaded when they met the target for the year. DLIs are in italics. Values for global indicators include those measured in the semester following annual DLI measurement, as those presented with annual measurements were projections and not actual EEP execution, as the data were not available until May of each year. *Data on this indicator are unreliable so they are not included in the assessment of achievement. 24 Component 2: Building Capacity in the Public Sector 13. Significant delays in utilizing funds under this component were experienced due mainly to delays in finalizing a procurement plan. None of the technical assistance that had been planned under this component was ever contracted. Instead, several consultancies and training were carried out to strengthen the newly established State Secretariat of Transparency and Control (SETC). SETC carried out some training both in Brazil and abroad for its auditors, on auditing techniques, leadership and management, and attendance to various conferences on corruption, data privacy, and on internal controls for Government contractors. They carried out a number of individual consultancies aiming to develop and improve its auditing function and develop policies with respect to access to information, information security and other areas. It also assessed IT needs and the improvement of its new transparency portal. They also hired TA to develop training materials. The State Secretariat of Planning and Budget (Secretaria de Estado de Planejamento e Orçamento -- SEPLAN) requested them to carry out an ex-post technical audit of Project data in 2012. 14. Later, consultancies with numerous other entities were agreed, including SEDEST, Secretaria de Governo, Defensoría Pública (Public Defense), Health, DFTRANS and others, for a wide variety of subjects including many which, although approved, were never carried out for lack of time. Studies included: an assessment of the quality of asphalt, IT platform design, approaches to social participation observatories, assessing social participation among subpopulations including youth, nonwhites, lesbians, gays, transgenders and bisexuals, agrarian reform policies, etc. Some training was financed for three people from SEPLAN/SES to visit a project in Uruguay. The majority of studies were carried out in the second half of 2012 and thus insufficient time had transpired to see if the study results are being utilized. Several studies are not linked to Project objectives. 25 Timeline of Events for the Project GDF Date World Bank April 9, 2009 Board Approval Sept 28� Oct 2, 2009 Project Launch Mission Oct. 15, 2009 Declaration of Effectiveness Allegations  of Corruption Nov.  2009 Governor Arruda arrested Feb. 11, 2010 Interim Gov Paulo O. Alves  Pereira departs Feb. 23, 2009 New PCU team formed March 2010 April 5, 2011 First disbursement made ($15.6 m) Interim Gov Wilson Ferreira Lima departs April 17, 2010 May 7, 2010 Brasilia Sustentável Project suspended June 7�16, 2010 Supervision mission Sept 2010 Procurement Plan approved w/ conditions National and State elections  held October 2010 Interim Gov Rogerio Rosso departs Jan. 01, 2011 Elected Governor Agnello Queiroz begins  term Feb. 21�25, 2011 Mission to discuss  restructuring June 13�17, 2011 Mission to review restructuring  proposal Draft of audit TOR submitted for comments July 2011 Request for restructuring submitted Oct. 20, 2011 Oct 31, 2011 2nd Semester 2010 audit due Nov. 2011 Audit TORs  approved 2008�2009 audit waived Dec. 19, 2011 Restructuring  not approved June 28, 2012 Second disbursement made ($9.1 m) Dec. 31, 2012 Project closing date Feb 6, 2013 Third disbursement ($2.2 m) April 30, 2013 End of Grace Period  Annex 3. Economic and Financial Analysis Not Applicable. 26 Annex 4. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members Responsibility/ Names Title Unit Specialty Lending Joana Godinho Task Team Leader LCSHH Health Jeffrey James Rinne Senior Public Sector Mgmt LCSTR Public Sector Mgmt Chris Parel Consultant Public Sector Mgmt Maria Madalena R. dos Santos Consultant LCSPE Education Elizabeth Goller Senior Transport. Specialist LCSTR Transport Rodrigo Briceño Consultant, Economist Economic Analysis Mariana Montiel Senior Counsel LEGLA Legal Isabella Micali Drossos Lawyer LEGLA Legal Ademildes Dantas Senior Operations Officer LCSHH Operations Daniela Pena de Lima Senior Operations Officer LCSHH Operations Luis R. Prada Villalobos Senior Procurement Specialist MNAPR Procurement Nicolas Drossos Consultant LCSFM Financial Mgmt Gunars H. Platais Senior Environmental Economist LCSEN Env. Safeguards Maria Valeria Pena Consultant Soc. Development Cecilia Lorena Brady Consultant Carla Zardo Program Assistant LCC5C ACS Veronica Jarrin Senior Program Assistant LCSHH ACS Supervision/ICR Joana Godinho Task Team Leader LCSHH Health Ricardo Silveira Task Team Leader LCSHE Education Jeffrey James Rinne Consultant, Public Sector Mgmt LCSPF Public Sector Mgmt Tarsila Ortenzio Velloso Public Sector Management Spec LCSPS Public Sector Mgmt Maria Madalena R. dos Santos Consultant, Education LCSHE Education Laudo Bernardes Consultant, Management LCSHH Institut. Analysis Elisabeth Goller Senior Transport. Spec. LCSTR Transport Mariana Margarita Montiel Senior Counsel LEGLE Legal Catarina Portelo Senior Counsel LEGLE Legal Ademildes Dantas Senior Operations Officer LCSHE Operations Daniela Pena De Lima Senior Operations Officer LCSHH Operations Luis R. Prada Villalobos Senior Procurement Specialist MNAPR Procurement Etel Patricia Bereslawski Senior Procurement Specialist LCSPT Procurement Regis Thomas Cunningham Senior Financial Mgmt. Specialist LCSFM Financial Mgmt Nicolas Drossos Consultant, FM LCSFM Financial Mgmt Susana Amaral Financial Management Specialist LCSFM Financial Mgmt Joseph Kizito Senior Financial Mgmt. Specialist LCSFM Financial Mgmt Gunars H. Platais Senior Environmental Economist LCSEN Env. Safeguards Marize de Fatima Santos Program Assistant LCSHH ACS Judith Marcano Williams Program Assistant, Temp LCSHE ACS 27 (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) Lending FY08 260.06 FY09 125.20 Total: 385.26 Supervision/ICR 298.75 Total: 684.01 28 Annex 5. Beneficiary Survey Results Not applicable 29 Annex 6. Stakeholder Workshop Report and Results Not applicable 30 Annex 7. Summary of Borrower's ICR and/or Comments on Draft ICR The Borrower submitted its Project Completion Report accompanied by an Executive Summary. Both documents are dated March 27, 2013, and are available in the Project files. The Executive Summary in Portuguese followed by a translation to English provided by GDF were included in this Annex. The Government also submitted comments on this ICR on June 10, 2013. 31 Programa de Moderniz.a,vio da Gestio PUblica do Oistrito Federal 1 SWAp Resumo Este relat6rio tern por objetivo detalhar o desempenho no ambito do Programa, tanto do Govemo do Distrito Federal quanto do Banco Mundial, sob a 6tica de li~Oes aprendidas. atraves dos cumprimentos de suas metas e indicadores bern como dos aspectos financeiros que tazem parte do acordo, em urn horizonte onde futuras opera~6es de credito intemacional poderao ser utilizadas em novos projetos. A implementa~ao do Programa de Modemiza~ao da Gestao Publica do Distrito Federal previa a estrutura~ao em dois componentes bilsicos descritos a seguir, que englobavam a~6es e atividades para a melhoria da presta~ao dos servi~os publicos de Saude, Educa~ao e Transportes e tambem para a modemiza~ao da gestao publica. As a~6es previstas neste Programa foram consubstanciadas no Plano de Govemo GDF 2007-2010 e aplicadas nos Programas de Despesas Elegiveis inseridas no Plano Plurianual de 2008 a 2011, e em urn conjunto de atividades de apoio para a modemiza~ao do modelo de gestao publica do GDF. Componente 1 - Melhoria da Presta~o de Servicros Publicos em Saude, Educacrao e Transpones. Focado nos setores de Saude, de Educa~ao e de Transportes/DFTRANS, por intennedio do reembolso financeiro ao Tesouro Distrital das despesas realizadas por meio da execu~ao financeira do Or~amento do GDF, nos programas elegiveis que possibilitassem o cumprimento das metas setoriais de Govemo. 0 reembolso financeiro ao Tesouro Distrital estava limitado a 32,5% (trinta e dois virgula cinco por cento) da execu~ao dos Programas Elegiveis nos tres setores, condicionados ao cumprimento, por parte do Govemo do Distrito Federal, de metas nas areas do Programa. Alem do desempenho especifico em cada area do Programa, o reembolso financeiro tambem estaria condicionado a execu~ao de resultados globais como o cumprimento dos preceitos da Lei de Responsabilidade Fiscal, ao superavit primario e a execu~ao de pelo menos 70% (setenta por cento) do or~amento de cada programa inserido no projeto. Foi preparada uma matriz de compromissos para este Programa, que consistia em urn agrupamento de despesas elegiveis, acordados com o Banco Munclial, a partir dos programas prioritarios do Govemo do Distrito rederal, periodo este que antecedia a assinatura do acordo de emprestimo, o que chamamos de periodo retroativo. Componente 2- Modernizacrao da Gestao Publica - Assistencia Tecnica . Este componente, focado na melhoria da gestao da publica, visava atividades de desenvolvimento e fortalecimento dos instrumentos de monitoramento e avalia~ao para o Modelo de Gestao por Resultados, metodo para avalia~ao da melhoria da gestao publica. As aquisi~6es e desembolso no ambito deste componente deveriam seguir as regras de licita~ao e desembolso do Banco Mundial e excepcionalmente nao era exigida contrapartida. As principais atividades previstas para este componente contemplava aquisi~6es de servi~os consultoria individual e de empresas principalmente para a administra~ao 2 32 Programa de Moderniz.a,vio da Gestio PUblica do Oistrito Federal 1 SWAp direta. lnicialmente objetivou-se o desenvolvimento de urn sistema de avalia~ao de custos e desempenho e a elabora~ao de rnodelos de monitoramento e avalia~ao dos compromissos de resultados. o componente visaria tambem aumentar a economia, a eficiencia e a transpan§ncia do sistema de aquisi~iies do GOoF, alinhando-Q com os padr6es intemacionais por intermedio do fortalecimento institucional, da modemiza~ae dos processos de aquisi~6es, da capacita~ae dos tecnicos, no estabelecimento de mecanismos de planejamentos de aquisi~6es e no gerenciamento de contratos. lmplementa9ao e aspectos operacionais As secretarias setoriais de Saude, Educa~ao. Transportes e DFTRANS forarn responsaveis pela execu~ae direta das a~6es e atividades de contrapartida e inseridas neste Programa. Coube as Unidades de Gerenciarnento Local - UGL, de cada secretaria, infonmar sobre o andamento das atividades e a demonstra~ae dos resultados alcan~ados junto a Secretaria de Estado de Planejamento e Or~amento quando solicitado pela Coordena~ao Geral do Programa. o acompanhamento do projeto foi realizado pela Unidade de Coordena~ae Geral do Programa - UCP com base nos cronogramas e indicadores estai:Jelecidos no acordo de emprestimo. Miss6es de supervisae do Banco Mundial juntamente com a Coordena~ao do Programa fizeram o acompanhamento das metas pactuadas junto as secretarias executoras de forma sistematica, com reuni6es ordinarias e extraordinarias confonne a demanda de cada setor. Observa-se, no entanto que pelas caracteristicas multisetoriais deste Programa, e seus aspectos inovadores aplicados no modelo de gestae do Governo do Oistrito Federal aumentariam o risco de uma resistencia cultural, tomando-se urn desafio ainda maior na lase de execu~ae do que o esperado em sua fase de prepara~ae em 2008. A crise politica no final de 2009, que acarretou as frequentes mudan~as entre govemadores, secretaries e equipe t ecnica, afetando o projeto desde a sua lase inicial de execu~ao em outubro de 2009 ate o momento de encerramento em 2012, prejudicando significativamente a execu~ao de todas as atividades previstas na fase de prepara~ao do programa. o Programa de Mooemiza~ao da Gestae Publica do Oistrito Federal teve uma oportunidade de reestrutura~ao. confonne ja era prevista no acord:o de emprestimo, mas isto s6 poderia acontecer ap6s 18 meses de sua abertura, o que ocorreu em abril de 2011. Sendo assim o BIRD juntarnente com a UCP/SEPLAN iniciaram miss6es de revisae do acordo bern como reuni6es setoriais para revisao dos objetivos, metas e indicadores, objet ivando inclusive urn novo cronograma fisico e financeiro, prorrogando o Programa em 18 meses ap6s data prevista de encerrarnento e reaJocando desta forma os recursos nao reembolsados ate entae. Em abril de 2011 foi solicitada pelo Banco Mundial uma minuta da Proposta de Reestrutura~ao do Programa, os trabalhos junto aos setores envolvidos tiveram inicio no mesmo mes com a presen~a do BIRD e UCP, sendo conduida e encaminhada ao BIRD em setembro de 2011. No entanto o BIRD manifestou-se informando que nae havia mais temi)o habil e em dezembro de 2011 o Banco Mundial encaminhou oficialmente carta ao Governador solicitando Plano de A~ao para os indicadores nao atingidos e informando que o Prograrna nao seria prorrogado, permanecendo desta forma a data prevista inicialmente de 31/12/12 os mesrnos objetivos, estrutura, metas e indicadores. 3 33 Programa de Moderniz.a,vio da Gestio PUblica do Oistrito Federal 1 SWAp Para a execu~ao deste programa foi elaborado o Manual Operativo do Programa - MOP, o qual orientava a opera~ao do Programa. Este Manual Operativo foi atualizado e apelfei~oado, de comum acordo entre o Govemo do Distrito Federal e o Banco MundiaJ, segundo as necessidades surgidas e li~iies aprendidas ao Iongo da sua implementa~ao. Assim, foi recomendado que o usuario sempre fizesse consultas a edi~ao mais recente, que estava sempre disponivel no site da Secretaria de Estado de Planejamento e Or~amento atraves do portal do Programa (www.programagestaogdf.df.gov.br). Avaliacrao dos Resultados 0 Govemo do Distrito Federal passou por grandes reformula~6es no periodo de vigencia do contrato de emprestimo em tela, as quais provocaram grande instabilidade politica e administrativa na gestao publica nos exercicios de 2009 e 20 10 . Como advento do rnovo Govemo do Distrito Federal em 2011, a equipe teve que reformular as prioridades metas existentes as previstas no novo plano de govemo. Com isso, a equipe gestora da Secretaria de Educa~ao foi alterada diversas vezes, cada qual com os seus direcionamentos e com uma nova estrutura administrativa. Por isso, tivemos varias composi~6es d'a Unidade de Gerenciamento l ocal, nao possuindo infonma~6es hist6ricas quanto aos assuntos debatidos em cada reuniao. o ideal seria que o contrato em questao tivesse sido reformulado em 2011, sobretudo no que diz respeito aos indicadores, com metas mais factiveis e que melhor espeJhassem a nova realidade do DF, contudo nao foi possivel a celebra~ao desse ajuste de repactua~ao. Portanto, os numeros apresentados aqui, reftetem a tentativa de monitoramento de indices elaborados por outrem que tinham como objetivos especificos prioritarios outros que nao os mesmos da atual gestao. Dessa forma, destacamos a atua~ao da SEDF em cada a~ao elencada, mesrno que tenha ocorrido de forma substancialmente diferente do proposto de forma inicial pelo primeiro comite Gestor do Programa. Ressa~o que a Secretaria de Educa~ao tern agido pr6-ativamente com vistas a promover a melhoria na modemiza~ao da gestao das unidades escolares visando sempre· ao alcance de acrescimo sustentavel nos niveis de qualidade do ensino publico do Distrito Federal. lndicadores de Desembolso: i) Para o indicador de desembolso "Aumento da taxa de atendimento a educa~ao infantil de 4 e 5 anos em areas carentes do DF (Fonte: Telematricula). lnicialrnente a diminui~ao nas metas de atendimento da Educa~ao 1 nfantil, deve-se ao fato da entrada em vigor da Resolu~ao 0 112010 do Conselho de Educa~ao do DF que altera a data de corte para 3 1/03, para acesso a Rede Publica. A dificuldade encontrada para o aJcance das rnetas se deu por au sen cia de unidades escolares que atendem o publico-alvo na rede publica de ensino do Distrito FederaL ii) Para o indicador de desembolso "Atendimento de a'lunos com distor~ao idade/serie no ensino fundamental (1° ao 9" ano) em programas de corre~ao de 4 34 Programa de Moderniz.a,vio da Gestio PUblica do Oistrito Federal 1 SWAp nuxo escolar": A Secretaria de Educa~ao manteve em 2009 os programas de corr~ao de fluxo com o objetivo de corrigir a distor~ao idade/serie, implementando os Ires projetos (Se liga, Acelera e Veredas) atenderam urn total de 16.501, segundo dados do Censo Escolar. No entanto, o govemo atual avaliou que nao haveria interesse na continuidade dos programas que eram executados por entidades privadas. iii) Para o indicador de desembolso "Atendimento de alunos com distor~ao idade/serie no ensino media em programas de corr~ao de fluxo escolar": Este e indicador referente a corre~ao de fluxo escolar dos alunos matriculados na 1• e 2• series do ensino Media que se encontravam em series nao compativeis com a idade, em decorrencias de multiples fatores, tais como: reprova~ao escolar, evasao, ingresso tardio na escola, etc. Em rela~ao aos estudantes no ensino Media com distor~ao idade/serie, segundo dados do censo escolar de 2010, registrou-se um total de 24.017 nessa condi~ao em 2010. iv} Para o indicador de desembolso "Numero de escolas com educa~ao integral em areas prioritarias do DF.": Em 2009, 201o e 2011 nao houve nenhuma escola que desenvolveu, por compl eto, ou seja, minima de 07 (sete) horas de atendimento diario, em todos os dias da semana, com atendimento a todos os alunos na perspectiva da Educa~ao Integral, dentro das areas prioritarias. As escolas com educa~ao integral atendidas naquele periodo, segundo dados do EDUCACENSO, nao estao locaJizadas nas areas prioritarias, conforrne prescrito no indicador correspondente, sendo estas escolas localizadas nas demais regi6es do Distrito Federal, sendo atendidas 04 (quatro) escolas em 2009, 01 (uma) escola em 2010, 01 (uma) escola em 2011 e 02 (duas) escolas em 2012. Cabe ainda ressaltar que a Educa~ao Integral no Distrito Federal iniciou-se em 2007 com a cria~ao da Secretaria de Estado Extraordinaria para a Educa~ao Integral - Decreta n• 28.503, de 0411212007, extinta em 2010 com a cria~ao da Subsecretaria de Educa~ao Integral, Cidadania e Direitos Humanos [ pelo Decreta n• 31.613 de 23/4/2010. Em 12/12/2011, DODF n• 237, de 13113/2011, houve a extin~ao desta Subsecretaria e a cria~ao da Coordena~ao de Educa~ao Integral, vinculada a Subsecretaria de Educa~ao Basica. Com esta nova organiza~ao. fez-se o planejamento para o periodo 2012 a 2014, sendo que ap6s o periodo ·de cadastro no EDUCACENSO, ainda em 2012, foram inaugurados mais 03 Centros de Educa~ao lnfantil em tempo integral (01 em Sao Sebastiao, 01 em Brazlandia e 01 na Estrutural), sendo que estas escolas atenderam a sua totalidade de estudantes em periodo de 07 horas diarias. Para 2013, Iemos urn acrescimo de mais 22 Unidades escolares oferecendo educa~ao integral em periodo de 10 horas dianas a todos os estudantes. Destas escolas, Iemos 02 Unidades em Ceilandia, 03 Unidades em Planaltina, 07 Unidades no Plano Piloto/Cruzeiro, 04 Unidades em Nudeo Bandeirante/candangolandia!Riach.o Fundo I, 01 Unidade em Santa Maria, 02 Unidades em Brazlandia e 03 Unidades em Taguatinga. 5 35 Programa de Moderniz.a,vio da Gestio PUblica do Oistrito Federal 1 SWAp lndicadores de Monitoramento: i) Para o indicador de monitoramento "Aumentar as taxas de promo~ao do ensino fundamental (1° ao go ano)": Em 2011 foi alcan~ada a taxa de prom~ao com a mectia de aprova~ao do Ensino Fundamental de 84,79%. Contudo, a atual administra~ao da Secretaria de Estado de Educa~ao, identificando que as metas estipuladas eram timidas para os objetivos do Govemo do Oistrito Federal, esta implementando os Cidos de Aprendizagem na rede publica de ensino, com vistas a propiciar a aprendizagem dos alunos minimizando, ainda mais, os percentuais de reprova~ao escolar. ii) Para o indicador de monitoramento "MelhOrar a taxa de promo~ao do ensino Mectio.· Em 2011 a taxa de prorno~ao dos alunos do ensino Medio foi de 66,17%, segundo dados do Censo. As causas para o nao atingimento da meta estipulada nao foram identificadas tempestivamente·. contudo a atual administra~ao da Secretaria de Educa~ao, obseJVando os baixos resultados alcan~ados no periodo avaliado, detenminou a implementa~ao para exercicio de 2013 da Semestralidade do Ensino Mectio, com vistas a oportunizar a aprova~ao escolar e redu~ao dos indices de evasao. iii) Para o indicador de monitoramento "Melhorar o lOEB de 125 escolas selecionadas com mais baixo lOEB localizadas em RA's mais carentes. (Anexo IV)": A partir da media da Prova Brasil 2007 e 2009, verifica-se que as escolas que atendem o Ensino Fundamental Regular - Series lniciais - 4" serie, atingiram um percentual de 98% de melhoria nas taxas do lOEB. Em rela~ao ao ensino Fundamental Regula r - series finais: 5" e s• serie a varia~ao encontrada foi de 96,5% daquelas que alcan~aram melhorias no lOEB. iv) Para o indicador "Aumentar a eficiencia de estrategias de gestao nas escolas": o indicador foi instituido sob a vigencia da Lei 4.036, de 25 de outubro de 2007, e executada por meio do Programa de Trabalho destinado a execu~ao da descentraliza~ao da administra~ao financeira das escola (POAF}. Contudo, no decorrer do exercicio de 2011, foram realizadas plenarias com a comunidade escolar que culminaram com a realiza~ao da confen§ncia da Gestao Oemocratica da Educa~ao. Oiante desse marco, aprovou-se a Lei n• 4.751, de sete de fevereiro de 2012 que revogou a norma anterior e estabeleceu a gestao democratica na rede publica de ensino do Oistrito Federal. saude: il Populacao coberta oelo Proorama SaUde da Familia: o conceito de "Popula~ao Coberta" utilizado pelo Ministerio da Saude s6 reconhece equipes completas, com todos os profissionais previstos na regulamenta~ao sobre o tema, inclusive no que se refere ao quantitativa de cada categoria. Nos ultimos anos a SESOF encontrou uma serie de dificuldades para contrata~ao e fixa~ao de pessoal em alguns locais, o que resultou em um percentual expressivo de popula~ao coberta por equipes 6 36 Programa de Moderniz.a,vio da Gestio PUblica do Oistrito Federal 1 SWAp incompletas, nao consideradas no calculo. Registra-se tam!Jem, a impossibilidade de implanta~ao. aquisi~ao ou aluguel de im6vel para abriga-las. Alem do baixo or~amento govemamental para edifica~ao, tambem nao foi possivel a utiliza~ao de recursos do BIRD para esse fim. ii) Prooorcao de 6bitos intantis i nvestigados o nao alcance de:sta meta deveu-se em grande parte ao contexto politico vivenciado uma vez que a estrutura~ao de comiss6es/comites e levada a efeito a partir da nomea~ao , para a ocupa~ao de diversos cargos nas Regionais de Saude. Desta maneira nao e realista esperar que atividades que dependam de estrutura atinjam um nivel adequado de funcionamento no primeiro ano de um govemo, ap6s uma fase prolongada de desestrutura~ao. Porem, observa-se a recupera~ao expressiva ocorrida em 2011, embora fosse esperado que algum tempo decorresse antes que as comiss6es estivessem totalmente fomnadas e adaptadas as novas chef.as e orienta~6es tecnicas. iii) Razao entre exames citopato1 6gicos cervico-vaginais em mulheres entre 25 e 59 anos: o motivo principal para o nao atingimento desta meta foi a desistencia da SES na aquisicao dos insumos necessaries a realizacao desses exames pela area responsavel na SEPLAN. Tall deficiencia foi superada em 2012 em decorrencia da estrutura~ao da Central de Compras da SES em 2011 e do mapeamento e remodelagem de processes, em curso. Transpone: Ap6s a posse do novo govem o em 2011 foram tomadas decis6es de alto nivel no Setor Transporte que trouxeram novos fatores a serem considerados no desenvolvimento das a~6es em Transportes, dentre e'las: i) Homologa~ao do Plano d iretor de Transportes Urbanos, que pemnitiu a estrutura~ao de diversas a~iies em transportes de maneira prevista em lei; ii) Remodelagem do Sistema de Transportes Urbanos, com lan~amento de edital para novos contratos de opera~ao· do sistema de 6nibus de Distriito Federal e defini~ao de mudan~a do modelo de opera~ao. Este novo modelo mudou a contrata~ao de operadores de contrato por frota por bacias geograficas, integra~ao tarifaria e aberta mediante cart6es de transportes, concess6es por menor tarifa alem de diversas outras regras de opera~ao; iii) Constru~ao de corredores especiais de 6nibus, com destaque para Expresso OF, que liga a regiao sui do Distrito Federal a sua area central. Este corredor optou pela tecnologia operacionaJ BRT (Bus Rapid Transit), que envolve tecnologias veiculares especiais, infraestrutura de apoio diferenciada e sistemas de tecnologia como Centro de Controle Operacional; iv) lmplanta~ao nao apenas de um Centro de Controle Operaci'onal para a gestao do sistema, mas de todo um ambiente de Sistema lnteligente em Transportes (Intelligent Transport System - ITS), que e um ambiente mais amplo onde o Centro de Controle Operacional e apenas uma das grandes tecnologias envolvidas. 7 37 Programa de Moderniz.a,yio da Gestio PUblica do Oistrito Federal 1 SWAp v) Assun~ao do Sistema de Bilhetagem Automatica (SBA), com total transferencia da Gestao do SBA para o DFTRAN'S. lndicadores de Gestao Com rela~ao aos indicadores de gestao cabe informar que a Subsecretaria de Modemiza~ao da Gestao - SUMOG, unidade administrativa vinculada a SEPLAN, ficou responsavel pelos trabalhos relacionados aos Termos de Compromisso. num total de 33 ao Iongo dos ultimos anos, e principalmente por meio do Planejamento Estrategico lnstitucional (PEl) nao previsto inicialmente no Programa. Os trabalhos sao direcionados obedecendo as premissas e estrategias definidas pelo Plano Estrategico do Governo do Distrito Federal, conduido em agosto de 2011, com o apoio da Escola Nacional de Administra~ao Publica (Enap) e alinhados ao Plano Plurianual (PPA), Cicio 2012-2015. Alem do Plano Estrategico do Govemo. os PEls sao desenvolvidos em conformidade como Plano Plurianual (PPA), Cido 2012-2015, mais especificamente em atendimento ao Programa Tematico do Governo n•. 6203 - Apertei~oamento lnstitucional do Estado previsto no Plano Plurianual (PPA), Cido 2012-2015. Proposta de cont inuidade das a~oes previstas no Programa Gestao GDF Setor Ed uca~o: 0 Setor Educa~ao permanece trabalhando no sentido de desenvolver as a~6es que levarao ao atingimento de seus objetivos. Algumas corr~6es. altera~6es de metodos e procedimentos sao realizados no sentido de melhor atender a realiza~ao dessas a~6es. Setor Saude: 0 Setor Saude permanece realizando a~6es com loco nos objetivos pactuados no Programa Gestae GDF. Todos os objetivos pactuados fazem parte da agenda do Sistema Unico de Saude em nivel nacional, como politica de Estado onde o Distrito Federal nao e exce~ao. Assim, a amplia~ao da cobertura populacional pela estrategia Saude da Familia, a informatiza~ao de toda a rede de servi~os e dos processos administrativos, bem como a busca de identifica~ao e incorpora~ao de novos instrumentos gerenciais, continuam se configurando como objetivos estrategicos da SES para o periodo 2012 a 2015. Setor Transporte/DFTRANS: o Setor Transporte/DFTRANS (especialmente), ap6s superar as deficiencias internas provocadas por cancelamentos de convenios, lalla de acesso a dados essenciais ao controle das frotas de 6nibus, dentre outras dificuldades internas entrentadas esta conseguindo desenvolver as a~6es que permitirao o atingimento dos objetivos que foram pactuados no Programa e que fazem parte do Programa de Govemo. Para isso discuss6es tem sido realizadas no intuito de se evitar despesas duplicadas e maior 8 38 Programa de Moderniz.a,vio da Gestio PUblica do Oistrito Federal 1 SWAp otimiza~ao n of not only an Operational Control Center for managing the system, but an entire and completed Intelligent Transportation System (ITS - Intelligent Transport System), Which is a much broader system Where the Operational Control Center is j ust one of the maj or technologies involved. v) TakinQ over the control of Automatic TicketinQ System (SBA). with complete SBA management transfer to the OFTRANS. Management Indicators : Regarding the management ind icators it is important to inform that the Under secretariat for Modernization of the Management - SUMOG, WhiCh is an administrative untt subordinated to the Secretariat of Planning and Budget, was responsible for the Tenns of Commitment. It summed up a total of 33 over the past few years, and especially through the strategic lnstiM ional Planning (PEl) that was not foreseen in the originally program. The tasks are oriented to obey the premises and strategies set by the Government Strategic Plan of the OF that was concluded, in August 20 11, w ith the support of the National SChool of Public Administration (ENAP) as well as its alignment to the Multiyear Plan (PPA) 2012-2015. In addition to the Government Strat egic Plan, the PE ls are developed in accordance w ith the Multiyear Plan (PPA) 2012-2015, more specifically in compliance with the Government Thematic Program no. 6203 - state lnstiM ional Improvement predicted in the Multiyear Plan (PPA), 2012 -2015. 7 47 Programa de Moderniz.a,vio da Gestio PUblica do Oistrito Federal 1 SWAp Proposal for continuity of actions foreseen in the Management Program GDF Educatio n Sector The Education Sector remains working to develop actions that will lead to aChieVing their goals. Some of the corrections, the methods Changes and procedures are performed in order to better serve the realization of these actions. Healt h Sector: The health sector remains performing a ctions focused in the objectives that were settled in the Management Program GDF. All the settled goals are in the agenda of the nationwide Health System, as a State policy Where the OF has to follow. Thus, the expansion of population coverage by the Family Health Strategy, along with the computerization of all network services and the administrative processes, and the search for identification and incorporation of new management tools are continuing to be among the strategic goals of the Secretariat of Health for the period from 2012 to 2015. Transponati on Sector/DFTRANS: After overcoming domestic deficiencies caused by agreement cancellations, poor access into critical data to control bus fteets, and other internal difficulties faced, the Transportation Sector/DFTRANS is developing a ctions that will enable the achievements of the goals that were agreed in the Program Which are induded in the government Programs. For this reason, meetings have been held in order to avoid duplicate expenses and to better optimize the activities that should be implemented and that depend on definitions between the Secretariat of Transportation and DFTRANS. Lessons Learned Firstly, the Federal District Government recognizes the opportunity to assimilate the technical knoWledge and at the same time to improve its workforce With this experience regard ing foreign cred~ loans, in order to pursue better governmental management. Surely everyone, either from the GDF or the IBRO. Who participated in this project, made a big contribution with great effort and dedication. As lessons learned, we observed that it would have been very valuable to have into the structure of the coordination program, specialist consultants With not only international acquisitions or financial KnOWledge, but also a permanent procurement committee With pemnanent public servants With experience in a cquisitions, besides a legal advisory office. In this sense certainly the program would have much more quickness in government acquisitions. Also it was noticed that even if there was no pol~ical crisis weakening the program, some of the following inconsistencies that hampered the implementation stage could have been observed during its preparation : • The DFTRANS has its own procurement department and could not make purchases throughOut the Central Procurement Sector of the government, unless 1BRD if it has been approved. Onty in 2011 the 1BRD submitted to DFTRANS one questionnaire to validate the capacity of DFTRANS to purchase their goods Which was denied . 8 48 Programa de Moderniz.a,vio da Gestio PUblica do Oistrito Fedeul 1 SWAp • According to the IBRD Procurement Guidelines, all of the acquis~ions conducted througlh the electronic bidding system shoulcl be done through the federal electronic purchase system called ComprasNet. Since the beginning of the program until its dosure in 2012 , the Central Procurement Sector of the Federal District Government did not have technology, qualified personnel and authorization by the ComprasNet system to make its purchases electronically. • The health sector had great difficulty forwarding their acquisitions at the beginning of the execution pnase, because the IBRD nad demanded a prior revision on the major acquisitions. Besides that the way of grouping the items that would be purchased was confticiting with the way of worKing within the health sector. Only in the last year of the program, the IBRD approved a procurement sector for the Health, in order to comply wit h the IBRD procurement rules. However, there was not enough time to perform the bidding processes, so the Central Procurement Office was not sent due to the Terms of Reference (TORs) had not been approved by IBRO. • The education sector could not purcnase according to the 1BRD rules within the Eligible Expenditure Programs, because, before signing the loan agreement, the sector had already made purchases Within the Eligib'le Expenditure Programs. The peliod that preceded the agreement signature didn1 require the compliance with guidelines. Therefore, only the non-biddable, but eligible, expenditures as decentralization and transfer of resources mentioned in the Financial and Administrative Decentralization Program - PDAF were presented. The execution deadline for many acquisit ions from Component 2 became short due to the disbursement that was made in Aplil 201 1. At this time it was necessary to review the entire procurement plan for this component. In consequence of that, it became unfeasible the accomplishment of the execution in 36 months as planned, in the remaining 18 months of the Program. The lack of expelience of the public servants working With the international financial organizations increased the operational lisK . It can be observed that the experience gained by the staff of the Program is a landmark in the GDF. Despite the high turnover number, some public servants stick With the program and have had the opportunity to meet the IBRD working method. Also, they have developed their self-cliticism and the pursuit on the improvement of the processes. Undoubtedly, there could be permanent training actions for the public servants involved in this type of project. This would greatly facil~ate the operabion in projects involving international loans agreement. For future projects, it is suggested that the lisk list includes mecnanisms to ensure the continuity of the agreed actions, even in scenarios with political changes, as occurred in the OF, or in the government transitions. For instance, it is observed that the staff turnover led the IBRD and the Program Coordination to maintain all sectors updated on the peculialities of the program and its proposed challenges. However, it was not always possible to access and disseminate the information in all operational levels due to tne lack of a systematically and automatically integration mecnanisms, where participants could update the collected data and to feed the indicators in real time. The program contained many rules, acquisition projects, goals and indicators, so it would be appropliate for it to use a computelized tool to unity the project documentation or even to control an integrated timetable With all the project activities including its terms and responsible persons. 9 49 Programa de Moderniz.a,vio da Gestio PUblica do Oistrito Federal 1 SWAp For future projects emphasizing the need for directly or indirectly training for the personnel involved in the program, either through courses offered by the GDF or by 1 BRD would be important. We had requested to IBRD an extension of the numbers of GDF participants in courses of financial management and international bidding that were offered by IBRD. We thought only 2 spots per semester were not enough to meet the demand. Peli i