Document of The World Bank FOR OFFICIAL USE ONLY Report No: 22222-CO IMPLEMENTATION COMPLETION REPORT (CPL-36150) ONA LOAN IN THE AMOUNT OF US$50.0 MILLION TO THE REPUBLIC OF COLOMBIA FORA MUNICIPAL HEALTH SERVICES PROJECT October 10, 2001 Country Management Unit for Colombia, M6xico and Venezuela Human Development Sector Management Unit This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. CURRENCY EQUIVALENTS (Exchange Rate Effective ) Currency Unit = Colombian Peso $ 2,387.97 = US$ 1.00 US$ 1.00 = $ 2,387.97 FISCAL YEAR January 1 December 31 ABBREVIATIONS AND ACRONYMS ALADINO Standard Municipality Accounting System FIS Social Investment Fund IMF International Monetaxy Fund IBRD International Bark for Reconstruction and Development PSMS Municipal Health Services Project MOH Ministry of Health PAB Basic Health Plan SIAGHO Standard Facility-level Accounting System SIRES Information System for the Management of the Subsidized System Vice President David de Ferranti Country Manager/Director: Olivier Lafourcade Sector Manager/Director: Xavier Coll Task Team Leader/Task Manager: Magdalene Rosenm6ller FOR OFFICIAL USE ONLY COLOMBIA MUNICIPAL HEALTH SERVICES LN. 3615-CO CONTENTS Page No. 1. Project Data 1 2. Principal Performance Ratings I 3. Assessment of Development Objective and Design, and of Quality at Entry 2 4. Achievement of Objective and Outputs 4 5. Major Factors Affecting Implementation and Outcome 8 6. Sustainability 9 7. Bank and Borrower Performance 10 8. Lessons Learned I 1 9. Partner Comments I 1 10. Additional Information 14 Annex 1. Key Performance Indicators/Log Frame Matrix 15 Annex 2. Project Costs and Financing 18 Annex 3. Economic Costs and Benefits 20 Annex 4. Bank Inputs 21 Annex 5. Ratings for Achievement of Objectives/Outputs of Components 23 Annex 6. Ratings of Bank and Borrower Performance 24 Annex 7. List of Supporting Documents 25 Annex 8. ICR Mission Aide Memoire 26 This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. Project ID: P006854 Project Name: CO- MUNICIPAL HEALTH SERVICES Team Leader: Magdalene Rosenroller TL Unit: LCSHH ICR Type: Core ICR Report Date: October 12, 2001 1. Project Data Name: CO- MUNICIPAL HEALTH SERVICES LIC/TFNumber: CPL-36150 Country/Department: COLOMBIA Region: Latin America and Caribbean Region Sector/subsector: HC - Primary Health, Including Reproductive Health, Chi KEY DATES Original Revised/Actual PCD: 06/29/1987 Effective: 12/29/1993 01/31/1994 Appraisal: 12/02/1992 MTR: 01/14/1997 01/14/1997 Approval: 06/08/1993 Closing: 12/31/2000 12/31/2000 Borrower/lImplementing Agency: GOV OF COLOMBIA LA/MINISTRY OF HEALTH Other Partners: STAFF Current At Appraisal Vice President: David De Ferranti Country Manager: Olivier LaFourcade Sector Manager: Xavier Coll Julian Schwitzer Team Leader at ICR: Magdalene Rosenmn6ller Eleanor Schrieber ICR Primary Author: Magdalene Rosemn6ller 2. Principal Performance Ratings (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HL=Highly Likely, L=Likely, UN=Unlikely, HUN=Highly Unlikely, HU=Highly Unsatisfactory, H=High, SU=Substantial, M=Modest, N=Negligible) Outcome: S Sustainability: L Institutional Development Impact: M Bank Performance: S Borrower Performance: S QAG (if available) ICR Quality at Entry: S S Project at Risk at Any Time: No 3. Assessment of Development Objective and Design, and of Quality at Entry 3.1 Original Objective: The Municipal Health Services Project (3615-CO) was envisioned as the first stage in a 10 to 12-year program to reorient the public health system in Colombia. The project (known by its Spanish acronym, PSMS), financed with a combination of a US$50 million loan and US$33.1 million in national counterpart, sought to strengthen management and support municipal provision of a basic package of primary health services targeted to the poor. The objectives were: (i) in nine departments, to enhance municipal capacity to plan and manage provision of primary health care; (ii) to improve the quality, efficiency and coverage of primary health services in municipalities in the participating departments; and (iii) to support development of effective primary care by strengthening the design and execution of key policy measures, as well as evaluation. In short, the project sought to establish, for later replication, an effective municipal model for public sector delivery of improved basic health services, with a focus on the poor. At the time of identification and appraisal, Colombia had started a very ambitious health care reform, although key legislation has not yet been passed. The economic modernization program, set in place in the early 1990s, comprised wide-ranging trade, financial and social reforms. Despite social gains in the '80s, 40 percent of population was still classified as poor in the early '90s. Thus, upgrading and expansion of primary care services was regarded as an important part of strengthening social programs and services. Part of the overall reform was the decentralization of management and fiscal responsibilities for public sector programs, including primary health care, to the 33 departments and 1,029 municipalities. As a result, while the project design foresaw few changes in the functions of the Ministry of Health (MOH), substantial changes were anticipated at departmental level (planning and managing higher level services, and supporting municipal planning and management), and at the municipal level (planning, operating, and financing of primary services). The project originally fit well with the Government strategy of achieving improvements in health services through decentralization, and concomitant strengthening of municipal capacity to plan and provide primary health care under Law 10 - the political decentralization. 3.2 Revised Objective: At the time of appraisal, additional key policy measures were foreseen with an uncertain timing and impact, such as Law 60 (financial decentralization), and particularly Law 100 (financing and management of health care), which passed in December 1993, a few months after project appraisal. Law 100 proposed a major change in the health system, in the context of decentralization, transferring responsibilities to the municipalities not only for provision, but also for purchasing of health care. Thus, Law 100 added to the political and fiscal decentralization processes in Colombia and drastically changed the role of the MOH. The project made a major effort to adapt to the changing context. Project objectives were reinterpreted, but not changed with a formal restructuring, in response to the new reform context and consequent change in government priorities. While the original emphasis on improving provision of primary health care was diminished, the project still provided indirect support to the improvement of primary health care. -2- 3.3 Original Components: The project consisted of two sequential components, Institutional Development and Service Development, and a third component, Policy Management and Evaluation. The Institutional Development Component (expected to represent 11% of total costs) was designed to support municipal capacity to provide effective primary health care services through (a) introduction of a municipal health management information system for planning, monitoring, and evaluation of primary care services; and (b) provision of related training and technical support for municipal administrative authorities, health administrators and community representatives. The Service Development Component (expected to represent 80% of total costs) was designed to improve quality, efficiency and coverage of primary health care services through the implementation of (a) municipal subprojects centering on the introduction of a basic package of primary health care services; the provision of supplies and equipment to support the package; and the rehabilitation, expansion, construction and furnishing of selected primary care facilities; and (b) provision of training and technical support for municipal health practitioners. The Policy Development and Evaluation Component (expected to represent 9% of total costs) was intended to support the design and implementation of selected policies, including measures for the introduction of systematic pricing, cost recovery and quality control for primary health care, and for the development of a unified employment system for public health personnel; and to undertake an evaluation of project impact. 3.4 Revised Components: The project adapted to radical contextual changes motivated by the 1990s health care reform. Explicit changes were recorded in 1996 (midterm evaluation), 1998 (attempt to refocus), and 1999 (premature end of the project). The loan agreement was signed in June 1993, and effectiveness was reached in January 1994. The project suffered from a very slow start-up, due in part to the lack of support by an incoming administration. In December 1994, a study was commissioned to assess the need to restructure the project, in light of the changed context. The study concluded that technical assistance and investment as defined could support the implementation of health care reform, and no restructuring was required. The project continued to lag in disbursements, although it was a key policy instrument. By late 1996, only 10% of the loan proceeds had been disbursed due to the necessity of additional preparatory work for reform implementation. The project supported key analytical activities for the implementation of Law 100, including a new program of integrated technical assistance to support the development of municipal management. The midterm review in late 1996 documented a reorientation of project support. In addition to the change in focus, the project geographic scope was broadened to include 19 departments, instead of the original nine. -3 - In accordance with the new framework, the Institutional Development Component was changed to focus on development of a national infornation system (instead of a municipal system). Support to the planning and management of primary care services was changed to support municipal management: training of local health authorities and the elaboration of the Plan de Atenci6n Basica (Basic Health Plan, or PAB), and support for development of municipal management capacity. In the Service Development Component, the focus changed to the preparation of the municipalities for the approval of the financial criteria (certification) under Law 100. The project supported the MOH, departments and municipalities in the planning and development of human resources in management capacity, and training and workshops for social security personnel in the implementation of the new health system. Infrastructure support was provided to improve the physical quality of health services, largely contracted through the Social Investment Fund (FIS). In the Policy Development and Evaluation Component, the focus was changed to supporting studies for the implementation of the new health system. The new political administration, which took office in summer 1998, did not fully accept the program. In absence of a good monitoring and evaluation system, the project failed to fully justify its benefits. Adjustments were made, but tensions persisted. In January 1999 a large earthquake affected Colombia's economically vital coffee producing region. In light of a precarious macroeconomic situation and IMF restrictions on further indebtedness, both the Bank and Borrower decided to respond to post-earthquake rehabilitation needs by reorienting several existing loans. The remaining US$22.7 million of the PSMS project was reallocated. 3.S Quality at Entry: Quality at entry was satisfactory. At the time of design, the program fit well with both Bank and Borrower strategies. However, one significant shortcoming was the Bank's failure to identify the pending health sector reform as a risk and to foresee the development of contingency measures. Documents from the time of appraisal indicate considerable uncertainty about the timing and implications of the futures reform. An additional shortcoming at entry was the fact that monitoring and evaluation systems were mentioned in the project design, but no provisions were included to ensure that they would be fully embedded in the implementation. 4. Achievement of Objective and Outputs 4.1 Outcome/achievement of objective: The change in context and the subsequent reinterpretation of the project makes it difficult to judge the achievement of the objectives, which were quite openly defined, allowing for a flexible interpretation when this need arose. But there is little information about project impact as monitoring and evaluation systems were weak-ad hoc at best, and nonexistent at worst. Despite these limitations, it is possible to conclude that the project was satisfactory in strengthening management and planning at the municipal level for financing and coverage of the - 4 - population; contracting of health care providers; and supporting policy work at national level. While formally working only in the 270 'PSMS' municipalities, the project had a substantial spill-over effects, as the developed material (manuals, guides etc.) was made available by the MOH to all reforming municipalities. This and the support to reform implementation at national level meant that the project proved to be a key instrument in the implementation of Law 100. At the same time, it is important to note that because of the change in government's priorities, the project did not fulfill the original vision of directly improving municipal provision of primary health care, developing health information systems or monitoring and evaluation. Infrastructure upgrades surely contributed to an improvement in quality of health care provision, but was not accompanied by the originally foreseen clinical training in primary health care, as the training component was used for increasing management capacity at municipal level. In absence of a monitoring system, there is little evidence of a real rise in quality, efficiency and coverage of services provided. 4.2 Outputs by components: 4.2.1. The Institutional Development Component This component, which is judged to be satisfactory, supported municipal capacity by introducing health management information systems, and providing training and technical support to strengthen institutional capacity. The project contributed in more limited ways to the development of health management information systems by creating management accounting systems, but performed well in providing training and technical support. In training and technical support for municipal administrative authorities, the program provided a wide range of capacity-building tools, including manuals, seminars and on-the-job technical assistance. While 19 elected municipalities-demonstration centers-received a full package of technical assistance, 270 PSMS municipalities were given critical elements of the technical assistance package. In the new focus, the project strengthened management of health systems with courses and manuals for financing, affiliation and contracting: the identification and selection of beneficiaries, methodology for calculating costs of the Basic Package (PAB), and models of contracting health care providers and payment methods. The support directed to health provision included subjects such as budgeting and accounting, use of information systems and preparation of investment projects such as construction, equipment and maintenance. The manuals and courses were developed for and directed to the PSMS municipalities, but the MOH adapted them on a national level and made them available to all municipalities, thus allowing for considerable spill-over effects. With respect to the health management information system, consistent with the project's change in focus, the project centered on financial and management accounting systems, where important achievements were made: a standard facility-level accounting system (SIAGHO) was installed in 63 primary level institutions and 163 public institutions of low complexity level. A standard municipality accounting system (ALADINO) was implemented in six institutions. The information system for the management of the subsidized system (SIRES) was installed in 98 municipalities. Information technology support was provided to the departmental function of health management; - 5- and a system was designed and equipment provided to monitor the advances of 10 demonstration centers. Support was provided to the design of the Ministerial internet page. In addition, US$5 million were applied to Y2K compliance in a total of 69 hospitals. The inclusion of level 2 and 3 hospital was indicated in the situation, but not in the original scope of the project. Health management information systems were not developed, but funds earmarked for computer equipment were used for managerial capacity-building at municipality level. The lack of a health management information system was the main hindering factor for monitoring and evaluation. Although some indicators and studies were developed on an ad hoc basis, such as the 'Centinela-study' (some of the indicators can be found in Annex 1.1), no consistent system of indicators of evaluation of primary care service was elaborated. Community involvement was not consistently developed, although some good examples stand out, as in the municipality of Fonseca in the Guajira Department, where the indigenous community has been actively involved, running their own primary care center connected to the municipal services. 4.2.2. Service Development Component Overall, this component is rated as satisfactory. It included the provision of a basic package of primary health care services; the upgrade of selected facilities; and the provision of training and support to health practitioners at the municipal level. The project performed well on the investment side, but less so on the conceptual and training side. The project supported the development of the basic package ofprimary health care, which under Law 100 was to be defined at the municipal level. But its implementation was inconsistent, as municipalities seem to be preoccupied with certification and managing new responsibilities, rather than with performing the primary health care and public health functions. Overall, there is evidence that the public health care function has suffered during the reform: health data collection has been neglected, vaccination coverage has dropped, and health promotion programs have been discontinued in the country. There is some evidence that this tendency has been countered by the project in the PSMS municipalities. Vaccination coverage, for example, decreased less in the poorer PSMS communities than in other municipalities (cf. Annex 1.1). Where PSMS investments, such as construction or rehabilitation of health facilities were made, they have been successful. A total of 106 projects were carried out in 14 departments, 71 municipalities and 201 institutions (total value $20.6 million, of which $13.2 million from IBRD credit). However, the level of investment depended in part on the availability of municipal co-financing; thus, improvements of infrastructure were made where there was a high level of political commitment, which were typically not the worst off areas. An additional problem was the synchronization of funds. When municipalities finally allocated co-financing, the national co-financing often was not on hand, or there was a delay in the availability of Bank funds through the bureaucratically cumbersome FIS. -6 - Where problems of availability of funding were overcome, investments were very welcomed. A special unanticipated positive effect was noted in the zones with major security problems. Observers report that in those areas the provision of new investments gave citizens a renewed (and welcome) sense of state presence. Related to the change in focus, the provision of training and technical support for municipal health practitioners has not been done and there is no evidence of the implementation of a consistent package of clinical training and development of the basic primary care functions, as originally intended. 4.2.3. Policy Development and Evaluation Component This component, judged to be satisfactory, was intended to support the design and implementation of selected policies, including measures of systematic pricing, cost recovery and quality control of primary health care; development of a unified employment system; and evaluation of the project impact. This component's original intentions corresponded well to the eventual priorities of the government, and the project contributed to policy development with a set of studies, manuals and other materials for the launching of the reform under Law 100. In total, 18 studies were realized and 51 instruments designed. Examples of these are: a guide to reach certification, the definition of the content of the PAB, contracting manuals, and construction and equipment at the municipal level. Technical assistance under the project supported policy makers in the MOH in critical reform implementation issues, such as the definition of a communication strategy for the promotion of the reform; orientation of the education of human resources; and guidance for the management of the subsidized system. These studies were beneficiary not only to PSMS municipalities but also used by the Ministry for the national implementation of Law 100. One indisputably unsuccessful part of this component was project evaluation. In the project definition some indicators were defined, but with the change in the project's orientation, the project failed to redefine them and follow up. Interesting ad hoc studies were done, like the "Centinela" study in 2000, which originally had not made a difference between PSMS and non PSMS. The Informe de Cierre took the original data set and tried this comparison. It showed some advances in the PSMS municipalities. However, there was no consistent evaluation of the project. 4.3 Net Present Value/Economic rate of return: N/A 4.4 Financial rate of return: N/A 4.5 Institutional development impact: The project had a moderate impact on institutional development. See assessment of Institutional Development Component, above. -7 - 5. Major Factors Affecting Implementation and Outcome 5.1 Factors outside the control of government or implementing agency: Several external factors constituted challenges to project implementation. Throughout the period of project execution, Colombia suffered from a precarious security situation, which made implementation particularly difficult in affected municipalities. In addition, the 1997-98 macroeconomic crisis came about when investments were fully underway, and funds were the most needed. A plan agreed with the IMF foresaw the decrease of indebtedness and the lowering of debt ceilings. Thus, when in January 1999 the earthquake in the "Eje Cafetero" hit, no additional borrowing was possible, and thus the entire portfolio was reoriented. In the light of this strong alternative demand for funds and the lack of good project indicators, the project was not able to justify its existence. 5.2 Factors generally subject to government control: Three important factors that were subject to some degree to the government control strongly affected the project. First, decentralization and the related health reform process (Law 100), mentioned in the project appraisal document, had a profound impact on the organization and implementation of all project components. Second, project implementation suffered from multiple changes at the political level (frequent change of the Minister of Health (six ministers between 1993 and 2001), at the project level (seven project directors) and at municipal level (through four elections). The effect of these changes was particularly noted in the first two years of its execution, as the incoming administration in 1994 did not support the ongoing implementation of the reform; this put the project in a difficult, ambiguous situation, and is responsible in part for the slow start of the project. Third, access to both loan funds for civil works and counterpart was limited, resulting in the slow execution that was cited as one of the grounds for early termination of the operation. Using funds through the FIS for civil works was excessively bureaucratic and slow. Poor macroeconomic conditions, particularly in 1997-98, reduced overall availability of counterpart funding at central and municipal level. On the municipal side, there was a chronic problem with co-financing. The municipalities were chosen for their investment needs. This put poor municipalities with little co-financing abilities at a distinct disadvantage. On the other hand, where political will was high and health was recognized as a priority, funds were made available, often to the detriment of education and social protection. An adjusted co-financing scheme, based on categorization of municipalities, partially addressed this risk, but in the end the municipal co-financing arrangements significantly delayed the investment component, and directed investments to richer municipalities. 5.3 Factors generally subject to implementing agency control: The main factor subject to control of the implementing agency that affected the PSMS was the lack of use of monitoring and evaluation tools for decision making, and for better planning. At the same time, this shortcoming in part was a reflection of a quality at entry shortcoming. Only general project evaluation was noted in the project design, and monitoring was focused on advances at the level of the municipalities rather than at project level. - 8 - 5.4 Costs andfinancing: Overall, project resources were reduced by US$22.7 million (or 27 percent of the original total) when it was decided in 1999 to use remaining funds for post-earthquake reconstruction, rather than to continue activities in the health sector. For the funds spent before that period, there had been large shifts in the allocation of resources across components as a result of the evolution of project focus. In the end, the institutional development component accounted for 43 percent of the total (up from 11 percent projected originally); the development of services component represented 41 percent of actual spending (down from the predicted 80 percent); and policy development and evaluation accounted for 6 percent (down from the original 9 percent). The problem of co-financing at the municipal level added to problems of availability of counterpart funds at national level. Additionally the uncertainty of the availability of the national counterpart funds hindered implementation of the project as much as the weak administrative handling of funds at the FIS. These problems put the project in an unsatisfactory status in 1996/97. 6. Sustainability 6.1 Rationale for sustainability rating: T'he sustainability of the benefits of the program is very difficult to assess. The sudden closing of the project prevented it from making adequate provisions for sustainability. However, overall it can be said that sustainability is likely. With respect to the financial sustainability, those municipalities that decided in favor of infrastructure investments showed strong political commitment by dedicating local counterpart funds. Based on this strong commitment there is evidence that they have taken ownership of the investments made in the physical infrastructure and it is very likely that those will be adequately maintained. Regarding capacity-building, the material and manuals of the project are available and accessible in the Ministry, although the project failed to build up a corresponding inventory. The project' s training activities responded to the intensive training needs of the reform implementation at the time, and it is unlikely that training demand will continue to be as high. Personnel trained under the program continue to work in the health system at different levels: central, departmental and municipal. Some technical assistance activities that had been initiated were interrupted by the project's sudden liquidation, and could not be consolidated. With respect to institutional sustainability and borrower ownership, the project raised the importance of the municipal level in the provision of health at the central level. But it is unlikely that the municipal "advocacy" function will be carried on as the "decentralization" department in the Ministry was recently closed. 6.2 Transition arrangement to regular operations: When the project was prematurely ended in January 1999, the project assured the finalization of the implementation of infrastructure investments. As most of the activities are now part of the - 9- usual health system, no specific arrangements were made, but an inventory of all material developed under the project for future use is lacking. 7. Bank and Borrower Performance Bank 7.1 Lending: In general, the Bank's lending performance was satisfactory. However, there are several design shortcomings (mentioned earlier). Most important among these are the lack of contingency plan for identified risks at entry and the need for a more structured mechanism for systematic evaluation at regular intervals, to assess the effectiveness of the activities being supported under the PSMS. 7.2 Supervision: There was considerable continuity in task management-the same task manager was in place from 1994 until the end of the project-and this stability counterbalanced frequent personnel turnover in the country. The Bank was flexible in the face of changing conditions, ranging from reforms to natural disasters. Project progress was monitored on a regular basis and procurement issues were closely attended. Supervision missions were properly staffed and timeliness respected. Two points are worth mentioning. First, as noted earlier, a decision was made at the time of the midterm review (late 1996) to acknowledge and support a substantial reorientation in project activities. In hindsight, this decision came somewhat late in the project's development. Second, while procurement issues in the liquidation period were correctly taken care of, more attention could have been given to project content, especially in preparing the future use of the materials developed under the project. 7.3 Overall Bank performance: Satisfactory. Borrower 7.4 Preparation: The borrower was fully engaged in the preparation of the operation, participating actively in all aspects of design work. 7.5 Government implementation performance: Despite frequent changes in the MOH, the government performed in a satisfactory matter. The government complied with the loan's legal covenants, all of which dealt with routine planning and administrative issues. 7.6 ImplementingAgency: The implementation agency performed well, given the high turnover in leadership, as well as all the challenges imposed by the changing reform context. This is attributable to the dedication and commitment of the technical personnel responsible for implementing activities under this project. 7.7 Overall Borrower performance: Satisfactory. - 10 - 8. Lessons Learned 1. Flexibility yields benefits. In this case, continuous evolution in the interpretation of the project objectives permitted adaptation to radical changes in the policy context. 2. Project success depends on monitoring and evaluation systems with measurable, understandable indicators. This is particularly the case, when a project is being flexible and adapting to contextual changes. Ad hoc, highly focused evaluation activities cannot replace the existence of solid overall project indicators, which follow the project over its life. This can be an important tool for the project's survival in a changing political and economical environment. It is also essential for rigorous post-evaluation. 3. Projects supporting the health and social sector can have a reconciliatory effect in countries suffering from difficult security problems. The project investments and improvement of municipal response to health needs showed state presence at the local level and gave hope to people, especially in the areas with precarious security. 9. Partner Comments (a) Borrower/implementing agency: Translation of both ICR comments orizinally received in Spanish 1. From the Ministry of Health With regards to comments on the Implementation Completion Report of the Municipal Health Systems Program - PSMS, I would like to present the following observations: - The document presents a quite complete synthesis of the development of the Project, in relation to the objectives, achievements, difficulties and experiences in its implementation. - In general terms, the ICR picks up the results presented in the "Informe de Cierre" of the PSMS, prepared by the Program's Liquidation Team of the Ministry of Health. Nevertheless, it should be said that this report differs from the report on "Results and Impact Evaluation of the Program". This is due to the different methodologies used in the preparation, resulting in different figures. - According to the "Informe de Cierre", the total infrastructure and endowment projects implementation was 56.6%, reduced by a 76.7% in infrastructure and a 38% in endowment; therefore not all of the 106 projects were successfully finished. - With regards to the actions taken for Y2K, the Evaluation Document indicates that 143 hospitals benefited with upgraded biomedical equipment and 156 with suitable computation equipment, and not the 69 hospitals referred to in the Bank's document. - 11 - - The document in its overall analysis points as external factors affecting the program's development and results, changes in the administration at the national and territorial level, the macroeconomic crisis and the problems of public order, facts which we consider were totally predictable and subject to planning. - With regard to the changes in the administration, the "Result's Evaluation" document does not mention problems of this type for the beginning of the project, but rather related to the administrative procedures of the "Banco de la Repuiblica" and the legalization of the agreements with the fiduciaries. - In the result's evaluation no serious inconveniences related to law and order in the departments or municipalities were evidenced as difficulties for the development of the PSMS. - With regards to the flexibility and extent of the objectives of the Program, we consider that these allowed its adaptation to the changes derived from the sector reform, but at the same time, in some instances let to a deviation and lack of faithfullness to the guidelines initially defined, with approval of the Bank. - Contrary to what is expressed in the document, we consider that the sustainability of the actions undertaken was affected fundamentally by the lack of processes and instruments in the territorial entities which would have allowed the development and consolidation of the activities of the Program. - The departments as intermediaries between the Nation and the municipalities and in charge of coordinating sector policies in their jurisdiction and to provide technical assistance to the municipalities, should have received continuous, systematic and aggressive actions of institutional strengthening. In view of the sustainability of the actions of the PSMS, these territorial entities could contribute a lot. Cordially, MARGARITA JARAMILLO GIVES BOATMAN General Director of Sector Policies Analysis and Planning 2. From the National Planning Department Comments on the evaluation document of the Municipal Health Systems Project 1. In relation to the results of the project, we believe that the project can hardly be qualified as satisfactory. The comparative evaluation among municipalities that benefited from the project and those that did not, carried out by the "Municipios Centinela" project, didn't reveal significant favorable differences for the first ones. The best result was assessed in the development of policies component that allowed, while the project was under implementation, to strengthen the technical and administrative capacity of the Ministry of Health in this area. -12 - 2. The impact in the institutional development, in spite of the delivery of management tools to the municipalities and of the investment of resources for the improvement of the delivery of services, does not correspond to beneficiaries of the project. 3. The inconveniences generated by the changes in the context of the health system and mainly by the lack of continuity and consistency in the policies and sectoral strategies for the implementation of the health social security system, affected the development of the project notably, leading to a less than satisfactory borrower's performance. In addition, the capacity of support, advise and technical assistance to the decentralized administration in the Ministry of Health has weakened, affecting the consolidation and improvement of processes initiated at the start of the project. 4. For these mentioned reasons, the sustainability of the project could not be guaranteed, which led to its liquidation. 5. Finally, it's important to point out the contributions of the project in the diffusion and training of human resources and in the development of administration tools that did somehow introduce changes in the local culture at the institutional level. Sincerely, Jos6 Fernando Aryans Duarte Counsel, Secretariat of Health National Planning Department (b) Cofinanciers: N/A (c) Other partners (NGOs/private sector): N/A - 13- 10. Additional Information The Implementation Completion Report (ICR) team consisted of - Magdalene Rosemn6ller (Task Team Leader) - Cesar Granados (Team Assistant) - Patricia Bernedo (Program Assistant) Comments were received from - Eduardo Velez Bustillo - Jose Augusto Carvalho - Charles Griffin - Ruth Levine - Juan Pablo Uribe - Patricio Marquez - Maria Luisa Escobar - 14 - Annex 1. Key Performance Indicators/Log Frame Matrix Annex 1.1. Performance Indicators PSMS NON PSMS Indicators originally defined, 1993 Numbers of new patients by socio-demographic status N/A N/A % catchment population registered at health facilities N/A N/A Immunization coverage of official to subsidized regime DPT 80% /86% 72% / 88% Polio 83% /89% 74% / 92% #Women seeking prenatal care first trimester of pregnancy (Polio) N/A N/A Deliveries attended and delivery fatalities N/A N/A Child mortality (upper respiratory infections & diarrhea) N/A N/A Qualitative information in surveys N/A N/A Indicators as defined in 1997 redefinition % affiliation to the subsidized regime (1998) 56.9% 57.5% %Certification of municipalities (March 2001) 59.0% 48.7% % Transformation of public institutions in ESE 57.8% Population with knowledge on the rights and duties of the Law 100/93 Obligation 20% 24% IPS 7% 10% Family coverage 6% 8% CENTINELA Study (2000) Development DLS (Direcci6n Local de Salud) Certified 63% 50% with local health plan 100% 100% with DLS organized and functioning 63% 33% with information systems 100% 100% Development of Information Systems Insurance operating / automatic 63% / 25% 75% / 8% Service Network Operating / automatic 100% / 50% 92% /8% Invoicing operating / automatic 25% / 63% 42%18% Invoicing IPS designed / automatic 100% / 54% 100% /47% Transformed in ESE 86% 57% Financial Sustainability 39% 17% - 15 - Annex 1.2 Project Products PSMS NON PSMS Project Products Institutional Development Information Systems SIAGHO (Accounting System) 63 163 SIRES (Management Subsidized System) 98 ALADINO (Municipality Accounting System) 6 Equipments 69 Computers 177 Biomedical equipment 97 Institutional Development DLS Municipalities Departments Technical Assistance 270 19 Follow up institutional development 115 Capacity Building Accounting Plan 400 Service Development Investment Projects Approved Executed IPS - Infrastructure 12 72 IPS - Equipment 107 75 IPS - Infrastructure 82 Studies Completed studies 18 Developed methodology 51 Completed I programmed actions Institutional development 50.9% Service development 53.4% Development of policies and evaluation 49.2% Total 51.4% Source: Evaluation Study PSMS, POAS. Ministry of Health, Colombia. - 16- Annex 1.3 Change of Project Focus Products by Component SAR New Focus A. Institutional- Development 1. Development of a municipal health 1. Development of a national information information system. system. 2. Planning of administrative support and 2. Support to municipal management: Local services: training to staff and community health directories and delivery of PAB. representatives in planning and primary health care services management; technical assistance, study tours for staff; training in service; and improvement to public health training infrastructure. 3. Support to institutional management development: restructuring, quality, inventory and register. B. Services Development 1. Supply a basic package of primary health 1. Support the delivery of POSs and defining services. PAB. 2. Subprojects of investment in infrastructure, 2. Support to municipal authorities to prepare equipment and inputs to guarantee quality, projects for approval and financing by FIS. efficiency and coverage of the primary health care package. 3. Human Resources development: training 3. Support to the Ministry of Health, and supervision of doctors, nurses, departments and municipalities in planning and bacteriologist, dentists, auxiliaries, promoters; developing human resources, information to scholarships and study trips for central level IPS and EPS, about the training and staff, departmental and municipal. continuous study requirements. 4. Production of manuals about content and 4. Training of SGSSS staff; seminars with supply of primary health services, support to specific objectives oriented towards the the curricular revision of operational implementation of the health system. investigation programs in 4 universities in Bogota and Barranquilla. 5. Induction, training and supervision programs for temporary staff. C. Policy and Evaluation Development 1. Policy development: study in prices, cost 1. Support studies: single methodology to recovery, databases, organization and define POS, requirements and information flow, establishment of accreditation and quality up-to-date attention and service protocols, control entities. update UPC structure, co-payments and moderating quotes, financing, contracting models, payment mechanisms, subsidy transformation. 2. Development of a national information system and an obligatory system to guaranty the quality of health services. 3. Development of a system to follow and 3. Development of a system to follow and evaluate the proiect impact. evaluate the proiect impact. - 17 - Annex 2. Project Costs and Financing Project Cost b Component (in US$ million equivalent) Irn m~t r~ea CO,imPn _ Institutional Development 7.80 16.70 52 Service Development 54.70 10.10 40 Policy Development and Evaluation 3.10 1.80 4 Project Coordination 2.50 3.90 4 Total Baseline Cost 68.10 32.50 Physical Contingencies 0.40 1 Price Contingencies 14.70 22 Total Project Costs 83.20 32.50 Total Financing Required 83.20 32.50 Project Costs by Procurement Arrangements (Appraisal Estimate) (US$ million equivalent) 1. Clvil Works 0.00 12.80 10.00 0.00 22.80 (0.00) (3.80) (3.00) (0.00) (6.80) 2. Goods (Computers and 3.40 0.60 0.10 0.00 4.10 Audio Visual Equipment) (3.20) (0.50) (0.00) (0.00) (3.70) 3. Services 0.00 0.00 20.30 0.00 20.30 Medical Equipment (0.00) (0.00) (14.20) (0.00) (14.20) (including supplies) 4. Fumiture 0.00 0.00 4.00 0.00 4.00 (0.00) (0.00) (2.80) (0.00) (2.80) 5. Technical Assistance 0.00 0.00 21.80 0.00 21.80 and Training* (0.00) (0.00) (14.50) (0.00) (14.50) 6. Training Materials 0.00 2.00 1.00 0.00 3.00 (0.00) (1.40) (0.70) (0.00) (2.10) 0.00 0.00 3.8 0.00 3.8 7. Studies (0.00) (0.00) (3.80) (0.00) (3.8) 8. Project Administration 0.00 00.1** 3.2 0.00 3.3 (0.00) (00.1) (2.0) (0.00) (2.1) Total 3.40 15.50 64.20 0.00 83.1 (3.20) (5.8) (41.0) (0.00) (50.0) * Including training materials, provided under service contracts For vehicles - 18 - Project Financing by Component (in US$ million equivalent) Component _ SAR 1/* Implementation 21 Total % Parl-passu Total % Part-passu IBRD Counterpart IBRD Counterpart USS % USS % USS % USS % Institutional Development 35.8 43% 18.2 51 17.5 49 16.7 51% 11.3 68 5.4 32 Technical and Training Assistance 31.5 38% 14.5 46 16.9 54 14 43% 9 64 5 36 Computers and Audio-visual equipment 4.3 5% 3.7 86 0.6 14 2.7 8% 2.3 85 0.4 15 Services Development 33.7 41% 20.3 60 13.4 40 10.1 31% 5.7 56 4.4 44 Eligible Expeditures through FIS 33.7 41% 20.3 60 13.4 40 10.1 31% 5.7 56 4.4 44 Policies and Evaluation Development 3.9 5% 3.4 87 0.5 13 1.8 6% 1.7 94 0.1 6 Consulting Services Part C of the Project 3.9 5% 3.4 87 0.5 13 1.8 6% 1.7 94 0.1 6 Project Coordination 4.7 6% 3.1 66 1.6 34 3.9 12% 2.5 64 1.4 36 Unallocated 5 6% 5 100 Total 83.1 50 60 33.1 40 32.5 21.2 65 11.3 35 *After 1997 Project Restructuring - 19- Annex 3. Economic Costs and Benefits Not applicable for this project - 20 - Annex 4. Bank Inputs (a) Missions: Stage of Project Cycle No. of Persons and Specialty Performance Rating (e.g. 2 Economists, I FMS, etc.) Implementation Development Month/Year Count Specialty Progress Objective Identification/Preparation 03/24/1992 3 Health Economist, Health Spec., Consultant Appraisal/Negotiation 12/16/1992 2 Health Economist, Consultant Supervision 06/03/1994* 1 Public Health Specialist S S 12/17/1994 5 Administration/Financial Spec., S S Consultant, Public Health Specialists, Health Economist 05/19/1995 4 Adminstration/Financial Spec., S S Consultant, Public Health Specialists 04/24/1996 5 Administration/Financial Spec., S S Country Officer, Consultant, Health Specialist, Task Manager 01/23/1997 6 Administration/Financial Spec., S U Country Officer, Consultant, Health Specialist, Task Manager, Health Economist 01/27/1998 6 Task Manager, Consultants, S U Procurement Spec., Financial Spec., Operations Officer, 06/20/1998 4 Procurement Spec., Financial S S Spec., Consultant, Task Manager 09/19/1998 5 Health Spec., Procurement Spec., S S Consultants 12/16/1998** S S 05/04/1999 S S 12/17/1999 S S 06/20/2000 S S 11/30/2000 S s ICR 04/27/2001 1 Health Economist S S * Board approval June 1993, Effectiveness January 1994 ** PSR Updates, regular visits by procurement specialists to supervise infraestructure investments (b) Staff: - 21 - Stage of Project Cycle Actual/Latest Estimate No. Staff weeks US$ ('000) Identification/Preparation 5.4 58.5 Appraisal/Negotiation 23.5 216.2 Supervision 126.94 415.3 ICR 6.81 31.7 Total 162.65 721.7 -22- Annex 5. Ratings for Achievement of Objectives/Outputs of Components (H=High, SU=Substantial, M=Modest, N=Negligible, NA=Not Applicable) Rating O Macropolicies O H O SU * M O N O NA E Sector Policies O H OSUOM O N O NA l Physical O H O SU * M O N O NA EJ Financial O H OSU*M O N O NA L Institutional Development O H * SU O M O N 0 NA L Environmental O H OSUOM O N * NA Social E Poverty Reduction O H OSU*M O N O NA El Gender O H OSUOM O N * NA E Other (Please specify) O H OSUOM O N * NA El Private sector development 0 H O SU *M 0 N 0 NA El Public sector management 0 H * SU O M 0 N 0 NA E Other (Please specify) O H OSUOM O N * NA - 23 - Annex 6. Ratings of Bank and Borrower Performance (HS=Highly Satisfactory, S=Satisfactory, U=Unsatisfactory, HU=Highly Unsatisfactory) 6.1 Bank performance Rating O Lending OHS*S OU OHU O Supervision OHS *S O U O HU [O Overall OHS *S O u O HU 6.2 Borrowerperformance Rating II Preparation OHS*S O u O HU O Government implementation performance O HS O S 0 U 0 HU O Implementation agency performance O HS O S 0 U 0 HU gI Overall OHS OS OU O HU -24 - Annex 7. List of Supporting Documents 1. Mission reports: Preparation BTO/aide-memoire, March 26-April 7, 1992 (Schreiber) Pre-appraissal/BTO, June 19- July 3, 1992 (Schreiber) Appraisal/aide-memoire, December, 7-16, 1992 (Schreiber) Supervision/BTO April 21-26, 1994 (Homedes) Supervision BTO/aide-memoire, May 16-June 3, 1994 (Homedes) Supervision/aide-memoire, December 12-16, 1994 (Marquez) Supervision BTO/aide-memoire, May 8-19, 1995 (Marquez) Supervision BTO/aide-memoire, December 9-13, 1996 (Wallentin) Mid-term Review aide-memoire, January 14-23, 1997 (Marquez) Supervision BTO/aide-memoire, September 14-18, 1998 (Marquez) Closing mission/aide-memoire, April 17-26, 2001 (Rosenmoller) 2. Staff Appraisal Report 3. Informe de Cierre Final (prepared by executing agency) (May 2001) 4. Centinela Study (prepared by the executing agency) (December 2000) 5. "Evaluacion de Resultados y Efectos del Programa Sistemas Municipales de Salud PSMS "(Ministerio de Salud) - 25 - Annex 8. ICR Mission Aide Memoire AYUDA MEMORIA MISION DEL BANCO MUNDIAL Proyecto de Apoyo a los Sistemas Municipales de Salud Ministerio de Salud de Colombia Misi6n de Elaboraci6n del ICR - (Implementation Completion Report) del Banco Mundial Bogota, 17 al 26 de abril de 2001 I. INTRODUCCION Un equipo del Banco Mundial visit6 Colombia del 17 al 26 de abril de 2001 para realizar el ICR - Implementation Completion Report - del Proyecto de Apoyo a los Sistemas Municipales de Salud (PSMS). Los objetivos principales de la misi6n fueron de: (a) obtener informaci6n acerca del desempeflo, resultados e impacto del Programna de Sistemas Municipales de Salud para la elaboraci6n del ICR, que esta por complementar para Junio 30, 2001. Los temas centrales mvestigados fueron: (a) ver si la operaci6n se centro en problemas de alta prioridad en el sector salud; (b) estudiar el papel del proyecto en apoyar al gobiemo Colombiano, los entes territoriales y las instituciones aut6nomas en la mejora de la gesti6n, la financiaci6n y prestaci6n de servicios de primer nivel; (c) ver como el programa se adapt6 a cambios en la reforma Colombiana; (d) identificar los puntos fuertes y debiles de la implementaci6n, y si se adoptaron soluciones de menor coste, (e) ver si las mejoras son sostenibles desde el punto de vista financiero y de las instituciones; y (f) determinar las lecciones que se pueden aprender para mejorar el diseino y la implementaci6n de operaciones futuras en Colombia y otros sitios. El equipo estuvo compuesto por Magdalene Rosemn6ller, Economista de Salud. A lo largo de la visita se ha reunido con los integrantes del equipo liquidador del proyecto, Gloria Soraya Ulluoa (Gerente Liquidadora del programa PSMS), y los consultores del equipo: Jorge Luis Ardila, Freddy Mendoza, Patricia Delgado, Maria Cristina Cruz, Ana Edith Sanchez, Economista. Se han hecho reuniones con Luis Carlos Sandino, Exdirector General de Descentralizaci6n del Ministerio de Salud; Antonio Mendoza Pino, Exdirector de Descentralizaci6n del Ministerio de Salud; Diana Casadiego y Lyda Miranda, funcionarias del Ministerio de Salud; con Hugo Valenzuela Jefe Divisi6n de Desembolsos Credito Publico del Ministerio de Hacienda; y ejecutivos de la OEI, la Organizaci6n de Estados Iberoamericanos y de la Secretaria Ejecutiva del Convenio Andres Bello - SECAB, y con Paulo Bacci, Consultor del Fondo de Inversi6n Social - FIS. Ademas se han realizado visitas de estudio al campo. El equipo liquidador ha acompafando la misi6n en la visita de estudio del campo. Desgraciadamente no se ha podido ir a todos los sitios - 26 - por razones de seguridad del pais. Se han hecho encuentros en Santa Marta con la Secretaria del Departamento de Magdalena, y los representantes del municipio de Fundaci6n (alcaldia y hospital), con el Secretario de Salud del Departamento de la Guajira, y los directores de los hospitales de Barrancas (Guajira) y Fonseca, (Guajira). En Baranquilla se encuentro con la Secretaria de Salud del Departamento de Atlantico, con los Secretarios departamentales de Salud y Obras y los funcionarios de ambas dependencias. En Cartagena la misi6n y el equipo se encontraron con la Secretaria de Salud del Departamento de Bolivar y los representantes de los municipios de Santa Rosa y San Pablo (alcaldia y hospital). Una entrevista con la Ministra de Salud, Doctora Sara Ordofiez Noriega tuvo lugar el Jueves, 19 de abril, en compafiia del Secretario General, Francisco Arciniegas Andrade, y la Directora General de Analisis y Planeaci6n, Margarita Jaramillo de Botero. La Misi6n agradece la cooperaci6n y apoyo profesional brindados por el equipo liquidador del programa y el Ministerio de Salud y los distintos actores encontrados durante la visita. La Misi6n quiere agradecer de manera especial el apoyo de la Ministra de Salud, la Dra Sara Ordofiez, los integrantes del equipo de liquidaci6n y funcionarios del Ministerio de Salud, que han aportado su conocimiento y analisis del proyecto. Pero sobre todo el equipo liquidador ha sido decisivo en su apoyo a esta misi6n, organizando las entrevistas, el viaje de estudio de campo. Ademas, la evaluaci6n y el futuro ICR han ganado mucho en los sesiones en equipo donde las diferentes percepciones y conocimientos se han podido contrastar. F. Pr6ximos Pasos - Calendario de elaboraci6n del Informe de Cierre y del ICR Finalizaci6n del Informe de Cierre Avance del Informe de Impacto y Factores 9 de Mayo Informe Final completado 20 de Mayo Elaboraci6n del ICR Circulaci6n del borrador ICR 8 de Mayo Revisi6n informal borrador ICR (Banco y Gobierno) Videoconf. 10 de Mayo Revisi6n borrador ICR (Banco) 18-30 Mayo Revisi6n oficial del ICR (Gobiemo) 4-7de Junio Finalizaci6n ICR 15 de Junio Se recuerda que no es implicito que existe un total acuerdo sobre el contenido del ICR y la evaluaci6n del proyecto. - 27 - Futuros posibilidades de apoyo del Banco Mundial En la reuni6n con la Ministra se han evocado algunos temas de como el Banco puede apoyar la implementaci6n de la reforma de salud en Colombia en el futuro. Una posible pr6xima misi6n de identificaci6n del Banco se podria realizar en julio / agosto de 2001 Los detalles y conclusiones de la visita y la evaluaci6n se podrian consultar en el ICR - Implementacion Completion Report. SARA ORDONEZ MAGDALENE ROSENMOLLER Ministerio de Salud Banco Mundial SantaFe de Bogota, 25 de abril de 2001 -28 - IBRD 24769 78- 74- 'U A JA0AqAcAwltN C O L O M B I A es s MONt^S Cihb..e Soo Ocecn MUNICIPAL HEALTH SERVICES PROJECT s ^ LAEZU E LA It, ho *~~~~d. Bgofd n- - (ok ATLAnW KDII Carea4 Validu9, / \ a \ ~~~~~~~~~~~~BRAZIL > 5 i>. >zs';/2 ) /aA6 kRu=f)t'o'Y l R tts \ t f/ ereeWs}Pe>z A BOLIVIA t;~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ PANR' ~Y PANAj i EAMA Z_U_ELA // 4.~~~~~IN Boenonento r 6i/ / UVae j I~~~~~~~~~SI '~«X t,i,,J,a | ) E C U A D 0 C 4~~~~~~~~~~~~~~~~~~~w'- mio* 010 5 Urbrn Poovitratin oi Center Towns Elevantions (nreters): (t < If t Sp | ~5,00~50 00,00 persons 0>< o w 50 100 150 200 250 t ~~~~2,000,0003,0 >> ILMTR \ s°o°ooo° o >8 ! B R A Z I L~~?A 3 500,0 100 000~ ~ ~ ~ ~ ~ ~ ~~~~~~~~~~~~~ o 0 5 506,00 CAVCA Project Departments 6 Nationol Capital r o Cities and Towns with Population Over 40,000 *- Trop hcs on prepored Main Highways E ( clusioely fo, the co-venience I r E S RooUrsIod is for the - s_r Rivers s p E R U j j ~ iPnter-oo use of The Wvrid Book G,O, Th. deoo.n.nti-n -od Department Boundaries ,.R/nd the bo,,dor,Ts hown International Boundories Th / / no re do nt rpltB ok tGhep ~~ I of The World Book~~~th Igop.1 01k fOYtritory or ony _ X ) st~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~notusoerventV or occeptooce of 4 IL 7Ei 7,40 g o-, uch bouvdorieo._ APRIL 1993