ICRR 12951 Report Number : ICRR12951 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 09/30/2008 PROJ ID : P065954 Appraisal Actual Project Name : Health Reform Project Project Costs (US$M): US$M ): 95.7 n.a. Country : Slovak Republic Loan/ US$M ): Loan /Credit (US$M): 62.9 69.6 Sector Board : HE Cofinancing (US$M ): US$M): 0 0 Sector (s): Health (60%) Compulsory health finance (20%) Central government administration (20%) Theme (s): Health system performance (50% - P) Other social protection and risk management (25% - S) Debt management and fiscal sustainability (25% - S) L/C Number : L7195 Board Approval Date : 09/11/2003 Partners involved : Closing Date : 12/31/2006 03/31/2007 Evaluator : Panel Reviewer : Group Manager : Group : Denise A. Vaillancourt John R. Heath Monika Huppi IEGSG 2. Project Objectives and Components: a. Objectives: The overall goal of Health Sector Modernization Support (HSMS) Sectoral Adjustment Loan (SECAL) was to support comprehensive health sector reforms that promote fiscal sustainability while continuing to provide quality health care services and financial protection for the population . The overarching objectives are to : (a) improve fiscal discipline through structural changes in the health sector that will alleviate the chronic problem of arrears in the health sector, encourage greater technical and allocative efficiency, and institute a system that will promote fiscal sustainability; (b) improve the quality and cost-effectiveness of the health sector; and (c) mitigate any possible negative impacts on health care access, especially for vulnerable populations . Health sector reforms were also supported by a companion Technical Assistance Project (HSMS TAP) (US$12.38 million), designed to focus on the development of technical capacity, especially in the MoH, health insurance companies, hospitals and reform units to implement reforms supported by HSMS SECAL . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): This fast-disbursing policy-based loan was designed to be released in three tranches : a first tranche of Euro 15 million (US$17.7 million equivalent) to be released following effectiveness; a second tranche of Euro 15 million (US$17.7 million equivalent) to be released upon fulfillment of specific conditions; and a third tranche of Euro 25 million (US$27.18 million equivalent) to be released upon fulfillment of specific conditions . Conditions for the release of the three tranches were organized around a policy matrix supporting specific objectives and actions, as follows (core conditionalities are indicated in bold ). The sequencing of policy measures reflected a two -phased approach, the first phase focusing on stabilization measures aimed at controlling the growing indebtedness of the health system and the second phase focusing on measures to maintain the stability and increase the efficiency of the system . Undertake systemic reform of health care financing to curb the immediate problem of growing arrears and ensuring long -term financial sustainability . These actions were aimed at giving the five health insurance companies (HICs) the necessary autonomy and capacity to enable them to become active purchasers of health services, including contracting selectively with providers and implementing new provider payment systems to address cost containment, efficiency and quality issues . They also aimed at defining more narrowly the scope of state-guaranteed health care and the introduction of co -payments to counter excessive consumption of health services. First tranche (effectiveness): A new Health Insurance Act developed and submitted to the Cabinet that would : (i ( i) clarify Government ii) describe how the health insurance companies would be administered; (iii) contribution rates, (ii) iii ) describe the regulations governing the health insurance companies; and (iv) iv) clarify the process of selective contracting . A new Health Care Surveillance Authority (HSCA) HSCA ) Act developed and submitted to the Cabinet . A new Treatment Act developed and submitted to the Cabinet . Plans for the introduction of financial discipline, stringency and hard -budget constraints in hospital financing developed. Second tranche: The contribution of the health system to general budget deficit is reduced to 1.0 percent of GDP compared with baseline . Mechanisms and institutional arrangements developed for the evidence -based inclusion of new treatment procedures and pharmaceuticals into the essential package of health services . Selective purchasing and contract monitoring capacity developed in Health Insurance Companies . Needs assessment of Health Management Information Systems completed . 2002 . Number of hospitals incurring debts is reduced by 25 percent, compared to 2002. Third tranche: The contribution of the health care system to general budget deficit is reduced to 0.75 percent of GDP compared with baseline . Nu mber of hospitals incurring debts is reduced by 50 percent, compared to 2002. Number 2002 . Improve clinical and patient -perceived quality of health care . The main actions were to establish a quality assurance system and to introduce evidence -based medicine as well as institutional mechanisms in the MoH, HICs and hospitals for continuous quality improvement . First tranche (effectiveness): Detailed Terms of Reference (ToRs) and Implementation Plan prepared for implementation of continuous quality improvement. Second tranche: Implementation of changes acceptable to the Bank in the physician and nurse licensing system in relation to the system of Continuing Medical Education (CME). The implementation of national quality improvement plan acceptable to the Bank is initiated, specifying the roles and responsibilities of the HCSA, and providing guidance for professionals and other related institutions on their role in quality improvement. HCSA ) is operational and implementing A new institution for quality assurance, licensing and accreditation (HCSA) its business plan . Third tranche: Minimum of four clinical guidelines for primary health care and eight guidelines for secondary care adopted, tested and implemented . Implement reforms of the health delivery system . Reform measures aimed at rationalizing the number of hospitals and hospital beds starting in the large cities and to grant administrative and managerial autonomy to hospitals and implement policies that would generate market -like conditions aimed at enhancing efficiency . First tranche (effectiveness): A new Act on Status of Health Care Providers developed and submitted to the Cabinet (establishing the legal framework for hospital autonomy ). Eligibility criteria for drawing upon the resources of Hospital Restructuring Fund developed, including : autonomy; strategic business plans highlighting financial discipline and control; and zero cash losses and adherence to a balanced budget . Second tranche: An implementation plan for rationalizing hospital capacity in the districts of Bratislava, Kosice and Banska Bystrica developed on the basis of a mapping exercise . 75 percent of hospitals in the districts of Bratislava, Banska Bystrica and Kosice operating as autonomous entities with self governing boards . Third tranche: Approved plan for rationalizing hospital capacity in Bratislava, Kosice and Banska Bystrica has been implemented . 25 percent of all surgery is treated on a one -day surgery basis, average length of stay is down by 10 percent and occupancy rate is up by 20 percent in acute care hospitals implementing reforms, compared to 2002.2002 . Hospital information systems developed and functioning in 50 percent of hospitals . Implement reforms in the financing of social and long -term care . Reform measures aimed to create financing mechanisms to ensure that groups using long -term nursing and social care would not be affected as hospitals closed inefficient wards providing this care, by stimulating a supply -side response in terms of the growth or private social care services. Second tranche: Concept and implementation plan for social and long -term care is developed Strengthen stewardship function (regulation, policy -making and planning ) of MoH and other relevant institutions , including the implementation of a comprehensive Health Management Information system (HMIS) . First tranche (effectiveness): A Health Policy Unit responsible for collecting data, monitoring and evaluating health reforms is established within the MoH . A needs assessment for a public information campaign for health sector reforms completed and a public information campaign formulated. A detailed action /implementation plan outlining sequential steps in the reform program for the first year formulated . Data needs for measuring health system performance identified . Baseline data for monitoring and evaluation of health reforms collected (including indicators of allocative and technical efficiency, equity and access, and quality of care ). Second tranche: Approved public information campaign under implementation . Third tranche: An evaluation of the first two years of the reforms, with special emphasis on patient access, equity and quality of care, completed and compared with the baseline for analysis of improvements (or otherwise ) on these aspects . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The first tranche of Euro 15 million was released on January 5, 2004, within days of the expected release date of December 31, 2003. The second tranche of Euro 15 million was released on August 16, 2005, four and one half months after the expected release date of March 31, 2005, due to delays in 2004-05 related to the passage of key legislation. The third tranche of Euro 25 million was released on March 27, 2007, nine months after the expected release date of June 30, 2006, due to uncertainty about the status of the reform agenda after the change in government in mid-2006. As a consequence, the closing date was extended by three months, from December 31, 2006 to March 31, 2007. The value of actual disbursements in terms of US dollars is greater (US$69.6 million) than the initial estimate (US$62.9 million) due to the depreciation of the US dollar against the Euro over the period of the loan. The total actual cost of the reform compared with appraisal estimates is not provided . 3. Relevance of Objectives & Design: The relevance of the project's objectives is substantial. Fiscal sustainability remains high on Slovakia's agenda in view of its planned adoption of the Euro in 2009. Quality and protection of vulnerable populations' access to services also remain relevant. Indeed, despite the change in government during project implementation, it is notable that all three of the project's objectives were identified by the new administration in the Manifesto released in August 2006, shortly after it took office. The objectives of balancing efficiency, vulnerable populations' access to services and quality are also prominent in health reform in OECD countries . The project's design is also substantial overall, but with shortcomings. A strong feature of the project was the quality of the analytical underpinnings and coherence of the policy reform strategy . A comprehensive seminar on health reform conducted in February 2001 with World Bank and Slovak reform team participation and a major piece of economic and sector work in June 2002 (Slovak Republic: Review of Social Policy and Expenditures ), provided solid underpinnings of the "managed competition" model featuring insurance competition subject to government regulation, of which variants can be found in a number of OECD countries . The reform was appropriate and mindful of pitfalls related to quality and equity . Country ownership of the SECAL-supported policy reforms was strong at the time of project approval and extended beyond the MoH to include the Ministry of Finance . This domestic consensus for policy reform made a policy-based lending operation the appropriate choice of instrument . Two design shortcomings included: (a) an underestimation of the potential for and impact of a deterioration in government commitment, during an implementation period that would overlap with elections; and (b) the failure to develop a solid M&E framework, including key performance indicators, to underpin and track progress of the project's third objective (protecting and preserving access and equity among the poorest, most vulnerable ). 4. Achievement of Objectives (Efficacy): Improve fiscal discipline (sustainability ) through structural changes in the health sector : substantial achievement Overall the health sector debt situation improved substantially during the project life, although significant uncertainties remain. New annual debt in the health sector was reduced from SK 9 billion (0.83 percent of GDP) in 2002 to SK 1.17 billion (0.06 percent of GDP) in 2007. This represents notable progress, but falls short of the target of no new annual debts in the health sector . The year 2007 witnessed a slight increase in sectoral indebtedness after several years of decline. During this same period, the stock of debts in the health sector declined from SK 22.8 billion (2.1 percent of GDP) to SK 8.21 billion (0.45 percent of GDP), achieving the target of 0.5 percent. The gains have proved most significant and most durable on the balance sheets of insurance companies where the improved incentives for efficiency through reforms went furthest . Insurance arrears declined from SK 12.4 billion in 2003 to SK 72 million in 2007. There has been little improvement among municipal hospitals, which are outside the control of the MoH. The percent of state hospitals with rising arrears fell to 39 percent (15 of 38 state hospitals. This represents a significant improvement over the situation in 2003 in which all hospitals were incurring health -related debts and exceeds the target of reducing these by 50 percent. Greater health sector efficiency has been attained on some fronts, but more progress is needed . Between 2002 and 2006, the establishment of a hospital rationalization plan led to an 11 percent reduction in the number of beds: from 5,716 to 4,727 in Bratislava from 3,177 to 3,083 in Kosice and from 1,284 to 1,232 in Banska Bystrica Despite these gains, there is a widely held view that further progress in reducing bed capacity (from about 36,000 to 30,000 nationwide) is necessary. Most (ask TTL for percentage) hospitals in these three cities are operating as autonomous entities with self governing boards and are adopting the commercial code, achieving the objective of 75 percent. Hospital occupancy rates increased slightly from 65.4 in 2002 to 68.3 in 2006 Average length of stay decreased from 9.4 to 8.8 days, almost achieving the target of 8.7 days. The slow progress towards further hospital closures and the potential return of soft budget constraints, could have worrying consequences for sectoral efficiency and indebtedness . Achievements can be traced to the structural reforms intended to create the right incentives for the efficient delivery of health care and the rationalization plans referred to in the policy . However, the gains in sectoral debt levels have begun to erode and face an uncertain future as efficiency incentives weaken and hospital over-capacity lingers under the new government . Improve the quality and cost effectiveness of the health sector : substantial achievement A quality assurance system is in place, including an Integrated Quality Initiatives Model and a National Plan for Quality Assurance. Legal criteria have been established for the inclusion of new drugs and procedures in the essential package . Four clinical guidelines for primary health care and eight guidelines for secondary care were adopted . A health information system has been developed and adopted . The Health Care Surveillance Authority (HCSA) with its strong quality oversight functions has become operational. HCSA has established an increasingly active channel for citizen complaints about care provided and a pathology department to inquire into cause of death where necessary . Previously accountability mechanisms for service providers had been very weak . These input-based achievements correspond with several conditions outlined in the policy matrix as well as regularly monitored performance indicators . Outcome indicators of quality were not specified or documented . In recent years a number of new oncology drugs have been added to the state benefit package (owing in part to savings generated by a more transparent pharmaceutical procurement system ), and new cardiovascular specialty facilities have been opened with the help of the funds generated by SECAL budget support . Both of these developments are strongly suggestive of improved quality, but also capture inputs rather than outcomes . The HSCA has acknowledged that the measurement of quality in Slovakia still needs to evolve to a greater focus on outcomes. Hospital infections declined from 7.14 cases per 1000 admissions, to 4.9, exceeding the target of 6.5. Colorectal cancer screening identified 10,352 cases in 2005-06, a significant increase over the 242 cases in 2002, but falling short of the target of 15,000. However, the attribution of these improvements to the project is not fully documented. The project played a valuable role in institutional reform that has brought the issue of quality to the forefront of health reform in Slovakia, and helped lay the groundwork for significant improvements in the years to come . Mitigate any possible negative impacts on health care access, especially for vulnerable populations : substantial The introduction of very low user charges (co-payment) under the reform -- 20 SK (about US$0.50 at the time of design) per outpatient visit and 50 SK (about US$1.25) per inpatient hospital day was not expected to pose a substantial threat to service access, especially in a scenario of strong economic growth and declining unemployment . Additionally, to mitigate the negative impact of health and other cross -cutting social and economic reform, the Government provided social assistance to the 700,000 people identified as vulnerable (out of a total population of 5 million) in the amount of 50 SK per person per month. Nevertheless, this objective was set to ensure that access of the vulnerable would not be reduced as a result of co -payments. A social impact assessment (required under the third tranche ) uses only aggregate data and thus does not provide any insights into the impact of co -payments on specific socially vulnerable groups . It shows that on average utilization fell (by 11 percent for primary care and 3 percent for specialized care between 2002-04). This may be desirable due to prior over -utilization, Utilization declined from 15 outpatient visits per year per population to 9, but disaggregated data would be needed for assessing trends by income quintile . Furthermore, the social impact report states that the collection of information based on ethnicity is not allowed under Slovak law and so the monitoring of health indicators of the Roma (a priority vulnerable group in the 2001 Country Partnership Strategy) was not possible for the project to undertake . Nevertheless, according to the TTL about 500,000 of the 700,000 people receiving social assistance from the government are Romas . Despite the decline in utilization, there is some evidence (opinion polls conducted in 2004) suggesting that (a) co-payments did not have a strong adverse effect on decisions to seek care (only 1 percent of those surveyed said they did not seek health care because of co -payments) and (b) informal payments fell (World Bank, 2006). In any event, user fees were unpopular and were removed soon after the new government took power in 2006. Overall efficacy is substantial. 5. Efficiency (not applicable to DPLs): Project efficiency is not evaluated in a development policy operation . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project objectives are relevant to the current government's Manifesto of August 2006. The design was grounded in technically sound analytic work and strong government ownership . The ICR notes that the design may have underestimated political risks inherent in the election that was scheduled to occur during project implementation. Nevertheless, overall relevance was substantial, and Government fulfilled all of the conditionality linked to the three tranches . The objective of fiscal discipline was substantially achieved . The project has brought the issue of quality to the forefront of health reform in Slovakia and helped lay the groundwork for significant improvements in the years to come, but quality improvements achieved under the project were mostly captured in terms of inputs rather than outcomes . Available evidence is thin but does indicate that the vulnerable population's access to services was not decreased as a result of the introduction of co -payments. a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: The significant risk to development outcome is based in large part on the assessment of fiscal sustainability . Key market-oriented reform measures designed to stabilize the finances of the health sector have been fundamentally altered during the past 12 months. The profit motive of health insurance companies has been severely curtailed by 2007 legislation that prohibits insurers from paying out dividends and instead obliging them to re -invest all profits in the sector. An amendment was passed by Cabinet in September 2007 to halt the transformation of hospitals into joint stock companies. The minimum provider network (with which all insurance companies must contract ) as defined in October 20207 includes only state-owned institutions, and thus represents a potential disadvantage to private facilities. Co-payments (introduced to rein in high utilization rates ) were abolished in mid-2006. Quality is at risk if debt problems squeeze financing for investment, if closures of out -dated hospitals do not materialize, or if providers do not face incentives to attract patients by offering high -quality care. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: The project was well grounded in a high -quality background analysis and a coherent reform strategy . And the Bank was key in ensuring the existence of strong government ownership by both Ministry of Health and Ministry of Finance. The design of the M&E framework was sound for measuring the fiscal sustainability objective, and adequate for measuring quality, but it did not include indicators for assessing changes in service access for the poor and vulnerable (Section 10). Project supervision was sound overall, including the presence of management on some missions. While the new Government embraced the overall objectives of health reform, its vision for achieving objectives changed somewhat . The Bank's decision to continue to engage with Government at the end of this project and to support its evolved policies to achieve the goals of health reform appears to be sound. at -Entry :Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Satisfactory 9. Assessment of Borrower Performance: Only one rating is provided here due to the difficulty in distinguishing between government and implementing agency roles in this development policy lending operation . The Borrower deserves equal credit with the Bank for the strength of the background analysis and the coherence of the reforms noted above . There was also strong ownership at the outset. However, the lack of a rigorous M&E framework to track the impact of reforms on access was a notable shortcoming on the part of the government responsible for project design . a. Government Performance :Not Applicable b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: Design . The project's M&E framework varied widely across the three project objectives . M&E for fiscal sustainability was strong. Indicators for measuring progress were, for the most part, clear and amenable to measurement. Indicators for the quality assurance and cost -effectiveness objectives were more limited, focusing almost exclusively on inputs for quality, with no indicators for tracking quality outcomes or cost -effectiveness. The weakest link in the M&E framework relates to the third objective which sought to preserve service access, especially among the poor and vulnerable . While the overall reform design took access into account (e.g., fee exemptions for the poor, minimum provider network), the M&E framework did not. There was reportedly reluctance on the part of the Borrower to emphasize this element of the project . Since the issue was central to the political debates about the health reforms, this shortcoming was important . Implementation . The project regularly collected and reported data on health sector debts and thus was able to consistently monitor progress towards improved performance . Broader issues of efficiency, such as hospital closures, average length of stay and occupancy rates were also regularly monitored . The project did an adequate job of tracking inputs to better quality . In fulfillment of a third tranche condition, a report examining equity issues was prepared, but this report had insufficient data to draw clear conclusions . Data analyzed were only averages and did not allow the assessment of changes on specific vulnerable groups . No use was made of household surveys, although this was regretted by peer reviewers both prior to approval and in the context of a retrospective analytical report. The regular household survey in Slovakia is plagued with comparability issues, but subsequent studies have attempted to overcome this, and a specialized survey might have helped . Utilization . Data emanating from the M&E of the first objective was helpful in tracking progress, but it does not appear to have been used to convince the government to stay on track with these reforms or to deepen reforms and efforts (e.g., additional rationalization of hospital capacity in the three major cities ). While the inputs to quality were tracked, they are less useful to policy and program staff for decision -making and fine-tuning approaches. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): A significant by-product of the project-supported reforms is the quantum leap in the level of awareness of systemic health reform issues in advanced economies among key policy -makers and stakeholders in Slovakia . Although difficult to quantify or establish attribution for this "intangible" factor, it is nonetheless evident to long -time observers. Regardless of the future course of reform, the quality of dialogue has improved dramatically in a short period of time . In addition, the Slovak health reform experience has generated considerable interest within the EU, and has thus offered valuable lessons that are applicable elsewhere in the region . 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Satisfactory The ICR assesses the third objective to Satisfactory be non-evaluable. While evidence is thin, it does indicate that the introduction of co-payments did not in fact adversely affect access to services of vulnerable populations. Risk to Development Significant Significant Outcome : Bank Performance : Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Project development objectives, which are not underpinned with strong monitoring and evaluation frameworks (including key performance indicators ), are less likely to be achieved . The project's third objective (to mitigate any possible negative impacts on health care access, especially for vulnerable populations ) had not specific indicators, thus leaving a void in evaluating the impact of the reforms with regard to access /equity. While the focus tends to be on the analytics and design of systemic health reform, due attention should also be given to the management of reform, in particular to consensus -building. Health reform can be very political, especially when a "shock therapy" approach is taken to a system that has been in place for many years. The reform agenda supported by the project encountered several challenges because the difficulties of implementing a good design in the face of opposition were under -estimated. Policy-based lending in the health sector is an important (and possibly under-utilized) Bank instrument for promoting development objectives in cases where the political capital to undertake reforms (and not a lack of financial resources per se) is the key binding constraint to strengthening sectoral outcomes . Ministry of Finance engagement is a critical ingredient to successful health sector reform . Project experience suggests that the Bank does have the potential to be a valuable partner in the fiscal stabilization of, and systemic reforms to, the health sector of a middle -income country. Although less costly financing was available in private markets, the Client saw value in engaging the Bank in order to access its technical expertise in helping to shape reforms and its third -party role in supporting domestic consensus between the Ministry of Finance and the Ministry of Health . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is of good quality. It is candid in its assessment and systematic in reporting on fulfillment of conditionalities and on achievement of objectives . Its assessment of the political economy of the reform supported under the project (p. 9) and the lessons learned are particularly strong features of the report . A shortcoming of the ICR is that it does not fully assess performance against the second objective : it assesses quality but not cost -effectiveness of health care. a.Quality of ICR Rating : Satisfactory