Document of The World Bank FOR OFFICIAL USE ONLY Report No. 23362 PROJECT PERFORMANCE ASSESSMENT REPORT HUNGARY HEALTH SERVICES AND MANAGEMENT PROJECT (LOAN 3597) December 12, 2001 Sector and Thematic Evaluation Group Operations Evaluation Department This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. Currency Equivalents (annual averages) Currency Unit = Hungarian Foring (HUF) HUF I = US$ 0.0039 US$ 1 = HUF$ 252.46 Abbreviations and Acronyms MISS Management Information Support System MOH Ministry of Health MTR Mid-Term Review NPHMOS National Public Health and Medical Officer's Service ICR Implementation Completion Report OED Operations Evaluation Department PMU Project Management Unit PPAR Project Performance Assessment Report QAG Quality Assurance Group SAR Staff Appraisal Report Fiscal Year January 1-December 31 Director-General, Operations Evaluation : Mr. Robert Picciotto Director, Operations Evaluation Department Mr. Gregory K. Ingram Manager, Sector and Thematic Evaluation : Mr. Alain Barbu Task Manager : Mr. Timothy Johnston FOR OFFICIAL USE ONLY The World Bank Washington, D.C. 20433 U.S.A. Office of the Director-General Operations Evaluation December 12, 2001 MEMORANDUM TO THE EXECUTIVE DIRECTORS AND THE PRESIDENT SUBJECT: Project Performance Assessment Report on Hungary Health Services and Management Project (Loan 3597) Attached is the Project Performance Assessment Report prepared by the Operations Evaluation Department (OED) on the Hungary Health Services and Management Project. The project was the first and only Bank project in the health sector in Hungary. It was a US$132.6 million project at appraisal, with a Bank loan of US$91 million, and Government contribution of US$41.6 million. The project was approved on April 20, 1993, and closed on June 30, 2000. The project was restructured twice. First, at Mid-Term Review in 1996, due to unsatisfactory implementation progress, with a number of activities canceled and new ones added, but objectives remaining unchanged. Upon the new (current) Government's election in 1998, the project was restructured a second time, with US$50 million of the loan canceled, new activities identified at Mid-Term Review canceled, and lack of Government interest in a second health project indicated. At project close, an undisbursed balance of US$6 million was canceled. Project costs totaled US$53.5 million, with the final loan amount US$35 million. The two objectives of the Hungary Health Services and Management Project were to improve the health status of the Hungarian population and to support the Government's program of health sector restructuring. The project's two components were: 1) Health Services Development, which had two subcomponents, Public Health and Disease Prevention, and Institutional Care; and, 2) Policy-Making and Management, whose major subcomponents were Public Health and Management Training, Management Support Systems, and Project Management (the PMU). The project failed to achieve its development objectives. It was overly large and complex and not coherent. Its design and much of its implementation was consultant-driven, and it was never really "owned" by the Borrower. Project implementation was made more difficult by frequent changes of government, and high turnover on both the Bank and the Borrower side. Overall project outcome is rated Moderately Unsatisfactory, in contrast to the Unsatisfactory rating in the Implementation Completion Report. Rating overall project outcome was difficult because of its many separate and somewhat isolated components and activities, which had very different individual outcomes and sustainability prospects. Institutional development impact is rated as modest, project sustainability as unlikely, Bank performance as satisfactory, and Borrower performance as unsatisfactory, all of which ratings are consistent with the ICR ratings. Bank Performance is satisfactory despite unsatisfactory project design and high turnover (five Task Managers in 6 and /2 years) because supervision was proactive, including a restructuring at Mid-Term Review which QAG rated highly successful and a best practice, a judgment with which OED agrees. This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. 2 Experience with the project confirms a number of well-established OED lessons: that Government ownership is critical; that the Bank must be engaged in the political dimensions as well as in the technical dimensions of design and implementation; that projects in new countries and/or sectors should be of modest and realistic scope; and that appropriate and measurable quantitative indicators, benchmarks and targets for project performance need to be identified during design, and used for monitoring and supervision, and for demonstrating results. Attachment Contents Principal Ratings ........................................................................................................................... iii K ey Staff Responsible ................................................................................................................... m Preface.............................................................................................................................................v Sum m ary and Outcom e ......................................................................................................... 1 Project Design and Im plem entation ......................................................................................... 1 Findings and lessons ...................................................................................................................... 2 Future Directions ........................................................................................................................... 4 Relevance ........................................................................................................................................ 4 Efficacy ........................................................................................................................................... 5 Efficiency ........................................................................................................................................ 5 Institutional Developm ent Im pact ............................................................................................ 6 Sustainability.................................................................................................................................. 6 Bank Perform ance ......................................................................................................................... 7 Borrower Perform ance........................................................................................................... 7 Bibliography ................................................................................................................................... 9 Annex A: Project Outcome Summary by Component........................................................ 11 Annex B. Basic Data .................................................................................................................... 13 Annex C. Com m ents from the Borrower................................................................................... 15 This report was prepared by Roy Jacobstein, Consultant, who assessed the project in May 2001. Timothy Johnston was the Task Manager. William B. Hurlbut edited the report. Pilar Barquero provided administrative support.  111 Principal Ratings Assessment ICR Outcome Moderately unsatisfactory Unsatisfactory Sustainability Unlikely Unlikely Institutional Development Impact Modest Modest Borrower Performance Unsatisfactory Unsatisfactory Bank Performance Satisfactory Satisfactory Permitted Ratings Outcome Highly Satisfactory, Satisfactory, Moderately Satisfactory, Moderately Unsatisfactory, Unsatisfactory, Highly Unsatisfactory Sustainability Highly Likely, Likely, Unlikely, Highly Unlikely, Not Evaluable Institutional Development Impact High, Substantial, Modest, Negligible Bank Performance Highly Satisfactory, Satisfactory, Unsatisfactory, Highly Unsatisfactory Borrower Performance Highly Satisfactory, Satisfactory, Unsatisfactory, Highly Unsatisfactory Key Staff Responsible Project Task Manager/Leader Division Chief! Country Director Sector Director Appraisal Alexander S. Preker Ralph W. Harbison Andrew P. Rogerson Completion Imre Hollo Armin H. Fidler Roger W. Grawe  V Preface This is a Project Performance Assessment Report (PPAR) for the Hungary Health Services and Management Project. The project provided support to national health reform efforts, which aimed to improve the health status of the population and the efficiency and financial sustainability of the health system. The project was financed through IBRD Loan No. 3597 for US$91 million; it was approved in April 1993, and closed in July 2000. A total of $35 million was dispersed, with US$50 million canceled in 1998, and an additional US$6 million canceled at project close. The findings of this assessment report are based on the observations of an Operations Evaluation Department (OED) mission to Hungary in May 2001 to review the performance of the project. The mission also reviewed completed health projects in Estonia and Romania, which are the subject of separate reports. The mission interviewed government officials, Bank and donor field staff, academics, and representatives of nongovernmental organizations. Field visits were made to a number of the project's major subcomponents, including the large chronic disease primary prevention demonstration site at Kolosca, the Cardiovascular Referral Center in Budapest, the Training Center for Health Services Management at Semmelweis University, and the School of Public Health in Debrecen. In addition, directors of project components and subcomponents answered a common questionnaire, and met at the end of the mission to discuss its findings, which have also informed this Report. Documentary sources include the project's Implementation Completion Reports (ICRs), the Staff Appraisal Reports (SARs), and project files. The authors express appreciation to all those who made time for interviews and provided documents and information, including past and present officials of the Ministry of Health, former project subcomponent activity leaders and Project Management Unit staff, and former and current World Bank staff. Following standard OED procedures, copies of the draft PPAR were sent to the relevant government officials and agencies for their review and comments. Comments received from the Ministry of Finance of Hungary, and from the Department of International Financial Institutions, National Bank of Hungary have been reproduced in the report as Annex C  1 SUMMARY AND OUTCOME 1. The objectives of the Hungary Health Services and Management Project were to improve the health status of the Hungarian population and to support the government's program of health sector restructuring. The project had two components: Health Services Development, which had two subcomponents, Public Health and Disease Prevention, and Institutional Care; and Policy- Making and Management, whose major subcomponents were Public Health and Management Training, Management Support Systems, and Project Management (the PMU). The project was the first Bank project in the health sector in Hungary. It was a US$132.6 million project at appraisal, with a Bank loan of US$91 million and government counterpart funding of US$41.6 million. The project became effective in October 1993 and closed in June 2000. 2. The project was restructured twice. At Mid-Term Review (MTR) in 1996, a number of activities were canceled due to unsatisfactory implementation progress, and new ones were added, but none of the loan amount was canceled and the objectives remained unchanged. The project was restructured a second time in 1998, when US$50 million of the loan and the new activities that had been identified at MTR were canceled (the overall objectives remained unchanged). Also, at that time the government indicated its lack of interest in a second health project. At project close, an undisbursed balance of US$6 million was canceled. Overall, costs totaled US$53.5 million; the final loan amount was US$35 million. 3. The project outcome is rated Moderately Unsatisfactory in contrast to the Unsatisfactory rating in the Implementation Completion Report (ICR).' Rating this project is difficult because of the need to give an aggregate rating to a project with many separate and somewhat isolated components and activities, some of which have had satisfactory or highly satisfactory outcomes by objective criteria, others of which have been clearly unsatisfactory. In general it is difficult to point to documented improvement in either health sector functioning or health status (as a result of the project, if at all). Several useful project components are likely to be sustained; a number of others are not. In some technical areas, the project catalyzed "new thinking" and "new ways of doing business," but ownership and sustainability of these new approaches is uncertain at this point, and longer-term positive outcomes have yet to be demonstrated. PROJECT DESIGN AND IMPLEMENTATION 4. The Hungary Health Services and Management Project was a highly complex project, with almost 30 discrete activities under the rubrics of public health, disease prevention, institutional care, and management. Furthermore, the project was not coherent, that is, the discrete activities had little interaction or interrelationship with each other. In addition, there was marked variability in subcomponent performance. The performance of the project overall was also adversely affected by a high degree of turnover on both the borrower and the Bank side. During the project's six-and-a-half year duration there were three governments, six Ministers of Health, at least six Deputy Secretaries of State for Health, four Chief Medical Officers, three heads of the Project Management Unit (PMU), and five Bank Task Managers. 5. In effect, the project had three distinct implementation phases: * Design and Initial Implementation: November 1992-November 1996. Probably because the health sector was never a government priority, the project's design was 1. The ICR authors have a 4-point rather than a 6-point assessment scale available to them. The essence of the ICR's textual remarks suggests that the authors would also have rated the project Moderately Unsatisfactory had such a rating been possible. 2 largely the product of outside consultants (whose priorities were not necessarily the government's). The complex project was proceeded with inadequate sector analysis, had little integration with the government's own program, and generally lacked performance indicators and monitoring plans. In addition, there was a long learning curve as the borrower adjusted to Bank procedures and practices, and the Bank adjusted to working in the health sector in Hungary. Implementation was very slow and unsatisfactory as confirmed in the 1996 Mid-Term Review (MTR). * Restructuring, Consolidation, Better Focus: November 1996-June 1998. At MTR, a number of activities were cancelled, most notably the Close the Gap Committee, whose function was to review proposed new initiatives in health promotion and disease prevention and to recommend which should be funded. A Health Services Delivery Modernization component was added, and plans were approved to expand the Management Information Support System (MISS) subcomponent nationwide. There were also changes in personnel and structure at the PMU and the Ministry of Health (MOH), which translated into a period of better management and implementation, and a greater degree of government ownership. None of the loan amount was canceled and there was no change in project objectives. A World Bank Quality Assurance Group (QAG) review rated the restructuring process as highly satisfactory and a best practice. * Restructuring, Downsizing, Project Windup: June 1998-June 2000. After a review of the project by the newly elected government, both the new Health Services Delivery Modernization component and the nationwide expansion of the MIS Support component were canceled. The government requested cancellation of $50 million of the original $91 million loan and indicated no interest in an extension or a second health project.2 At project close, an additional $6 million was canceled. FINDINGS AND LESSONS 6. The project was too large and complex, and its components failed to be coordinated and/or to interrelate meaningfully. Attempts to strengthen the National Public Health and Medical Officer's Service (NPHMOS) and broaden its approach to public health largely failed. Similarly, the health promotion and primary disease prevention activities largely failed to achieve results (of either a qualitative or quantitative nature). Secondary prevention activities have yet to be shown to have decreased the disease burden of the Hungarian population. Beyond the timely purchase of expensive diagnostic equipment, the institutional care component was never tracked, so the extent of the results of purchase and use of such new equipment on health outcomes or health status is unknown. Perhaps most tellingly and fundamentally, there was a general absence of high-level government support and ownership of the project. 7. Nonetheless, the project had a number of successes. An integrated, management- supporting MIS designed to control costs and improve organizational decision-making was introduced into one-sixth of all Hungarian hospitals.3 Two key academic institutions were significantly expanded and their technical capabilities strengthened. The work of these two institutions, the Training Center for Health Services Management at Semmelweis University, 2. The fact that Bank loan funds were now no longer additive to the Ministry of Health budget had a large role in this decision, as did the sense that the project "belonged" to the previous government. 3. A second phase, with nationwide rollout to the rest of the hospitals was a key part of the 1996 redesign. Unfortunately, the government's 1998 cancellation prevented this necessary step from taking place. Nonetheless, the majority of the remainder of the hospitals participated in the first phase, so a foundation may have been laid for meaningful health sector rationalization and restructuring, should the current government elect to follow this course. 3 Budapest, and the School of Public Health at Debrecen University, is likely to be relevant to future government restructuring/reform efforts. 8. There are other less certain, or farther off, influences or contributions of the project as well. Project emphases, experience and approaches have been incorporated into the new National Public Health Programme for 2001-2010.4 Some project activities are continuing, to varying degrees, including a national cancer registration and countrywide extension of secondary prevention efforts, dissemination of information on tobacco control to policymakers, professionals, and the general public, and the introduction of a more modem school health curriculum. A number of observers advanced the argument to the OED mission that the project catalyzed new thinking and approaches in many segments of the Hungarian health system, which will prove beneficial in the long run. That may well prove to be the case, although evidence of the persistence-and value in terms of outcomes-of such "changed thinking" is not yet available. 9. Experience with the project confirms a number of well-established OED lessons. Government ownership of projects is a necessary (if not sufficient) factor in project success. This implies first that the sector needs to be a government priority, second, that the government needs a long-term strategic vision and plan for that sector, and third, that senior government policymakers, decisionmakers, and leaders need to have substantial initial and ongoing involvement in design and implementation. Finally, the Bank must be as engaged in the political dimensions of project implementation as in the technical.5 10. Additional lessons from Hungary that confirm well-established OED lessons include: * In new countries and/or sectors, initial projects should be of modest and realistic scope, commensurate with borrower experience and capacity. * Project components are more likely to succeed when they have pre-existing support from "champions." Thus if these technical and programmatic areas are consistent with Bank priorities-often not a small caveat-they should be supported, since prospects for sustainable success will be higher.' * An external expert advisory group can often be of value to a project, but it cannot function as a substitute for direct government engagement in the project, or for a formal and active Project Coordination Committee. * Population-level indicators are not appropriate for health projects.8 Quantitative (or operationally-defined qualitative) benchmarks, targets, and performance indicators of 4. At the time of the OED Review, the National Public Health Programme, 2001-2010 did not have an implementation plan or budget attached, so the extent to which the project may have a lasting or continuing influence on this program remains uncertain. 5. This includes wide, iterative, and transparent engagement with all interested parties, for example, a wide range of politicians (including opposition political parties), key health sector stakeholders, and public opinion leaders. 6. Design and early implementation phases are even more critical for such projects. This implies the need for frequent, well-staffed missions, provision of robust technical assistance and training to PMU staff, project activity leaders, and key government officials, and an optimal balance in use of foreign and local experts (from within as well as outside the government). 7. In the Hungarian context, this phenomenon is exemplified by the successful and marked expansion and strengthening of the School of Public Health and the Center for Health Services Management. The lack of such support also explains the failure of NPHMOS to embrace the broader approach to public health envisioned for it in the project design. 8. Even though it is likely that the unsatisfactory level of those indicators-for example, life expectancy or age-specific or gender-specific mortality-is an overall motivating factor for a project, such indicators are inappropriate. If they 4 achievement are preferable, and are needed from the start. They need to be used for supervision and monitoring, and to be realistic-and measurable-in terms of the causal linkage between project-supported activities and anticipated results. FUTURE DIRECTIONS 11. The Government is still faced with the challenge of controlling health sector costs and fostering health sector reform. Major aspects of this challenge are to rein in secondary and tertiary care costs while simultaneously promoting public health and primary health care. The Bank- financed work in modern, cost-conscious, hospital management with the implementation of a comprehensive MIS in 1/6 of all the Hungarian hospitals, and (canceled) plans for scaling up to the remainder of the hospital sector may yet offer a sound basis upon which future Government efforts can be structured. 12. Similarly, the infusion into the sector of professionals trained in the latest principles and practices of both public health and of health management should provide Hungary with a base of needed human capital to engage in the reform effort. One key area where these new professionals would be of great use is in the implementation of the ambitious new ten-year National Public Health Programme of the National Public Health and Medical Officer's Service. The translation of this Programme into effective action will depend upon a number of approaches and emphases which the project introduced, including the need for indicator- and results-based programming, modern public health and preventive medicine, and cost-effective management. RELEVANCE 13. Project relevance is rated Modest.' Notwithstanding the project's design and performance flaws, the project's overall objectives, which focused on health sector reform, health system restructuring, health promotion and primary disease prevention, were substantially relevant to the imperatives the government is facing in the health sector. Offsetting this, however, was a clear divergence of the project activities from the activities and priorities of the Ministry of Health. 14. The institutional care component, which was meant to support a "balanced short-term remedial approach to addressing primary and secondary effects of cardio-cerebrovascular diseases," was less relevant to the project's overall reform/restructuring and public health goals. This component, projected at appraisal at 24 percent of overall project costs ($35 million), actually absorbed 29 percent of overall project costs ($15.35 million), of which 79 percent came from the loan. In practice, this component entailed only the procurement of expensive equipment. However, any leveraging of this investment, in terms of policy change or greater overall political and financial support for the project, was not apparent. Data for the health impact of this new equipment are lacking, because the collection of such data was not a part of project design. From the standpoint of relevance, it could be argued that such equipment-cardiovascular imaging equipment that cost $1- 2 million per unit-does relate directly to the project objective of reducing cardiovascular mortality and morbidity. At the National Institute for Cardiology, which was visited as part of the assessment improve over the life of the project, causality will be difficult to demonstrate; conversely, a project can be successful, and yet these indicators-which typically have complex and multifactorial causation and need years to show significant change-may not. 9. OED rates relevance, efficacy, efficiency, and sustainability on a four-point scale: negligible, modest, substantial, and high. 5 of this component, the equipment was in heavy and appropriate use. Its availability also allowed studies of children, studies which previously were not available in Hungary. EFFICACY 15. Project efficacy, the extent to which the project's objectives were achieved, or are expected to be achieved, is rated Modest. The determination of efficacy, like other determinations and judgments about the performance of this project, is difficult for several reasons. First, the project's objectives were couched in lofty and general terms (i.e., to improve the health status of the Hungarian population and to support the government's program of health sector restructuring). Second, there were no indicators for operationally defining these objectives and tracking their achievement. Third, the activities within the project components and subcomponents were unrelated to each other in practice (if not also in theory), thus synergies could not be developed. Fourth, performance varied widely across components. 16. To take polar examples, despite the critical importance of public health, the activities of the Public Health and Disease Prevention Subcomponent were largely unsatisfactory, marginalized, or too small to make a difference. For various reasons, ranging from flawed design to lack of government interest and support, any meaningful and lasting impacts in that subcomponent are difficult to identify at this point. It may indeed prove to be the case that "thinking [about public health and disease prevention] has changed," because of project activities, and that such changed thinking will translate into meaningful action in the Hungarian public and private sector. However, that view, which was often made to OED, remains speculative. On the other hand, the Policy Making and Management Component supported the substantial growth and development of the School of Public Health, the Health Services Management Training Center, and the hospital-based MIS work. These important project achievements are likely to contribute in a sustained way to efforts to reform the Hungarian health system and improve health status. EFFICIENCY 17. Project efficiencylo is rated Modest. With respect to allocative efficiency-whether the project financed the most cost-effective types of interventions-the record is mixed. Some of the project-financed activities, such as cancer screening, were among those considered to be highly cost-effective. But as discussed above, project design and implementation did not give sufficient emphasis to preventive interventions that are among the most cost-effective means to address Hungary's disease burden. In terms of technical efficiency, some components do achieve their objectives through least-cost approaches and interventions (for example, the MISS subcomponent). But overall, a number of components did not appear to make efficient use of project funds-with excessive spending on expensive equipment, high-priced consultants, or study tours without clearly defined objectives. Thus, for example, the Institutional Care Component absorbed 29 percent of overall costs and its benefits have not been documented; similarly, the Public Health Component, 28 percent of overall costs, has little tangible evidence of impact. Finally, the actual cost of the PMU was 218.6 percent of the appraisal estimate, 8.6 percent of overall project costs, as opposed to the estimated share of costs at appraisal of 1.6 percent. 10. The extent to which the project achieved, or is expected to achieve, a return higher than the opportunity cost of capital and benefits at least cost compared to alternatives. Quantitative rate of return analysis is not required for health projects. 6 INSTITUTIONAL DEVELOPMENT IMPACT 18. Institutional impact is rated Modest, consistent with the ICR rating. A key institutional development objective at the time of appraisal was the transformation of NPHMOS into a "modern, action-oriented" organization, with a broader, proactive approach to public health. This did not occur." Similarly, relatively little institutional development impact can be seen at this point within most of the other subcomponents of the Health Services Development Component. 19. On the other hand, the project's most important and long-lasting contribution may have been to institutional development. Two important schools, the Training Center for Health Services Management at Semmelweis University and the School of Public Health at Debrecen University, had their faculties enlarged and their technical capabilities strengthened. They are now internationally recognized, are awarding degreeS12 and are conducting policy-relevant and program-relevant studies bearing on government (and project) health priorities. Additional institutional development impact can be seen in the MISS subcomponent where 21 hospitals13 - one-sixth of all hospitals in Hungary-are continuing to follow the MISS approach. In fact, 75 percent of the hospitals that were not selected to participate also indicate that they are still using and benefiting from the plans they submitted for the competition. Thus a foundation may have been laid for a holistic and effective approach to meaningful health sector rationalization/restructuring, should the current government elect to follow this course. SUSTAINABILITY 20. Overall project sustainability is rated Unlikely, consistent with the ICR rating. Several project subcomponents seem sustainable while many others that were more fragmented and/or lacked wide support are unlikely to be sustained. Many of the primary prevention activities are not sustainable, and the secondary prevention activities, while ongoing in some respects, have yet to be integrated into widespread practice. On the other hand, both the Training Center for Health Services Management at Semmelweis University4 and the Public Health School at Debrecen University are functioning well and are likely to continue into the future. They are being funded not only by the government but also from international sources and from student fees. The two schools are granting degrees and their faculties and students are producing studies relevant to policies and programs of health sector restructuring/reform. Also, the MISS subcomponent's management support interventions and the new approaches to cost containment and other aspects of data-driven management are likely to be sustained. Finally, it is possible that the new thinking in NPHMOS, as reflected in its Public Health Programme, 2001-20 10, may be translated into an implementation strategy that is followed by the government. Should that occur then some project- catalyzed "new thinking," such as modem approaches to non-communicable disease prevention, may also be sustained. 11. The project's largely unsuccessful attempt to foster reorientation of the government's National Public Health and Medical Officer's Service (NPHMOS) toward greater emphasis on prevention of non-communicable disease may be beginning to bear fruit. Some 15-20 NPHMOS staff have been trained at the School of Public Health, and prevention of major noncommunicable causes of premature morbidity and mortality has been incorporated into the new National Public Health Programme for 2001-2010. 12. In the past five years there have been over 300 graduated or current students for advanced professional degrees at the two schools. 13. These hospitals were selected in an open and transparent competition, apparently a new approach in Hungary. They had their management information systems upgraded and received technical assistance in order to begin basing their management decisions on cost and utilization data. 14. The Training Center reports that over 50 percent of its costs are being covered by nongovernment sources and it is serving as a satellite venue and partner for the World Bank Institute's flagship course in health reform. 7 BANK PERFORMANCE 21. Bank performance is rated Satisfactory, consistent with the ICR rating. This assessment required weighing a number of factors that would tend to lead to opposite judgments. On the one hand, Bank performance was clearly unsatisfactory at initial project design. The project was too complex, consultant-driven, and tangential to government strategies, priorities, programs, and interests. Also, at least some share of the first four years' generally acknowledged unsatisfactory implementation must be attributed to the Bank. The fact that during overall project implementation there were five Task Managers, while perhaps largely unavoidable, was a function of Bank decision-making. And throughout that period of project operation, the Bank's performance in the political arena-generally acknowledged by all informants in Hungary to be a difficult and complex environment-was sub-optimal and ineffective. Lastly, the lack of indicators, targets and benchmarks for achievement during most of the project's implementation period was not good practice. 22. On the other hand, the Bank consistently worked hard and not without success in supervision, in order to improve project performance and to engage government interest. The Bank gave careful oversight and control to procurement, and what procurements were made were done without cost overruns or other untoward events. In addition, the Bank successfully deflected requests f6r costly (anesthesia) equipment that was tangential to project goals and objectives. Also, notwithstanding the flaws in the project design process and outcome, the project was quite relevant at appraisal, that is, it was largely working in the right technical areas. The restructuring at the 1996 Mid-Term Review was done well, as QAG also determined, labeling that effort a "best practice" and judging it to be Highly Successful." The Bank's ways of doing business, in project design and management as well as in procurement, were valuable in introducing greater rigor and fairness into the Hungarian health system. Similarly, many Bank-supported interventions, from increasing the knowledge and skills of many individual Hungarian health professionals to the establishment of two important schools related to health management and to public health, were important and likely to have lasting benefit. BORROWER PERFORMANCE 23. Borrower performance is rated Unsatisfactory, consistent with the ICR rating. At no time during the project was there real government ownership. The project was almost always largely tangential if not irrelevant to the MOH's activities and priorities. Although the project placed a strong emphasis on prevention of noncommunicable disease, reorientation of the government's NPHMOS has been largely unsuccessful to date. The lack of government's interest in the project is reflected in its perhaps overly hasty, but nonetheless final, decision to cancel the majority of the loan and not to seek additional Bank support. On the other hand, many project component leaders were-and still are-highly committed to their technical areas. Several important components and approaches were well-implemented, as discussed above; they have relatively strong government support, are making meaningful and ongoing contributions, and could well be sustained. In its comments on the ICR, the government concurred with the unsatisfactory rating for borrower performance, while rating Bank performance as satisfactory. 15. It is most unfortunate that two of the key project revisions-a new Health Services Delivery Modernization component, and the nationwide roll-out of the deservedly-praised MISS component-were cancelled by the new government in 1998. It is likely that these would have had a substantial positive impact upon restructuring and reform.  9 Bibliography Gaal, P., B. Rekassy, and J. Healy. 1999. Health Care Systems in Transition. Hungary. European Observatory on Health Care Systems. Godinho, Joana. 1998. "Health Promotion in World Bank Financed Projects in Europe and Central Asia" (ECA). World Bank. Washington, D.C. Health Promotion Research Institute (EKFI). 2001. "National Health Survey 2000, Preliminary Report." Hungary. Ministry of Health. 2000 "Detailed Implementation Completion Report, Health Services and Management Project," Project Management Unit, Budapest, Hungary. National Public Health Service (NPHMOS). 2001 "For a Healthy Nation: Public Health Programme, 2001-2010," Office of the Chief Medical Officer, Budapest. Draft. April 20. Orosz, Eva and Andrew Burns. 2000. The Healthcare System in Hungary. Economics Department, Organization for Economic Cooperation and Development. Staines, Vernon S. 1999. A Health Sector Strategy for the Europe and Central Asia Region. World Bank. Washington, D.C. WHO Regional Office for Europe. 1997. Investment for Health in Hungary. World Bank. 1993. Hungary Health Services and Management Project, Staff Appraisal Report No. 11292. Washington, D.C. World Bank. 1997. Hungary-Country Assistance Strategy, Report No. 17257. Washington, D.C. World Bank. 2001. Health Services and Management Project. Implementation Completion Report No. 21430. Washington, D.C.  11 Annex A Annex A: Project Outcome Summary by Component This Annex provides a brief summary of project outcomes by component. For additional detail, see the Implementation Completion Report (ICR). COMPONENT A: HEALTH SERVICES DEVELOPMENT Sub-Component A.1: Public Health and Disease Prevention i. Strengthening of the National Public Health and Medical Officer's Service (ANTSZ): Little progress was made in reorienting the Service to a more modem approach, including intervention into non-infectious and chronic disease. All funding was discontinued at mid-term review (MTR). ii. Primary Prevention Activity Cluster: This area received substantial investment, particularly a large chronic disease primary prevention (community) demonstration project. This component was not closely supervised, and evidence of health impact was minimal, as verified by an OED site visit. iii. Secondary Prevention Cluster: This activity worked toward the establishment of a national cancer registry, and the work is ongoing. Pilot screening programs for cervical, breast and colorectal cancer did not expand, and overall contribution of these activities to health outcome is unclear. Sub-Component A.2: Institutional Care: Sophisticated diagnostic and therapeutic equipment was provided for 5 regional/national cardiovascular centers. Use of the equipment is reportedly high, which the Team was able to verify on a site visit to the National Center. Sub-ComponentA-3: Health Services Delivery Modernization: Added after Mid-Term Review, this activity entailed a competitive award for a pilot, comprehensive regional restructuring of health services, but it was not implemented. COMPONENT B: POLICY MAKING AND MANAGEMENT Sub-Component B-1: Public Health and Management. This included support for: i. School ofPublic Health, Debrecen University, Debrecen ii. Health Services Management Training Center (HSMIT), Semmelweis University, Budapest. The faculties of both schools were enlarged, and their technical capacities strengthened. They have over 300 graduated or current students and are conducting policy-relevant research. They are being sustained by government, international sources and student fees. HSMT has become the regional center for the World Bank Institute. 12 Annex A SUB-COMPONENT B-2: MANAGEMENT SUPPORT SYSTEMS i. MOW Computing Center ii. Management Information Support Systems After a slow start, 1/6b of all Hungarian hospitals established modem information support systems, which are providing true costs and informing decisions. The planned nationwide rollout was cancelled at MTR, though a base has been laid for national expansion, representing a potential resource in the government's further reform efforts. SUB-COMPONENT B3: PROJECT MANAGEMENT UNIT The PMU had a mixed record, largely failing in the first three years of the project (1993-1996), and showing better performance after MTR (with new personnel). Costs were 218 percent of planned costs, largely because of high consultant costs in the early years. SUB-COMPONENT B4: PREINVESTMENT STUDIES Meant to inform a revised reform agenda after MTR, these studies were not done, as the newly- elected (and still current) Government canceled all new activities in 1998. 13 Annex B Annex B. Basic Data HUNGARY-HEALTH SERVICES AND MANAGEMENT PROJECT (LOAN 3597) Key Project Data Appraisal Actual or Actual as % of estimate current estimate appraisal estimate Total project costs (US$) 132.6 53.5 40.3 Loan amount (US$) 91.0 35 38.5 Cancellation (US$) 56.0 Date physical components completed: June 30, 2000 Project Dates Steps in project cycle Actual Approval 04/20/1993 Signing/Agreement 04/27/1993 Effectiveness 10/22/1993 Closing 06/30/2000 Staff Inputs (staff weeks) Stage of project cycle ActualVLatest Estimate Weeks US$ Identification/Preparation 77.9 158.4 Appraisal/Negotiation 39.2 89.6 Supervision 250.32 385.6 ICR 9.62 20.4 Total 377.04 654.0 14 Annex B Mission Data Stage of project cycle Date Number of Performance Ratings (month/year) staff in field Specializations represented Implementation Development Status Objectives Identification/Preparation 05/91 5 Health Economist, Sr. Legal Counsel, Implementation Specialist, IT Spec., Operations Officer Appraisal/Negotiation 07/92 6 Health Economist, Sr. Implementation Specialist, IT Spec., Operations Officer, Health Financing and Cash Benefit Spec., Counsel. Supervision 05/93 This is the initial summary S S 08/93 7 Mission Leader, Info Systems Spec., 2 S S Public Health Specs., Health Management Specs., Clinical Computing, Operations Officer 06/94 5 Mission Leader, Health Management Spec., S S Proj. Management Specialist; Info Systems Spec., Implementation Spec., Public Health Spec. 12/94 6 Mission Leader, Health Management Spec., S S Proj. Management Spec., Info Systems Spec., Implementation Spec., Public Health Spec. 02/96 3 Sr. Health Specialist, Procurement Spec., U U Research Assistant 06/96 2 Sr. Health Economist, Procurement U U Specialist 12/96 7 Task Manager, 2 Public Health, Proj. U U Management Spec., Sr. Info. Management Spec., Health Management Spec., Procurement Spec. 07/97 3 Task Manager, Public Health Specialist, S S Research Assistant 12/97 3 2 Task Managers, Economist S S 05/98 5 Task Manager, Health Specialist, Health S S Economist, Procurement Specialist, MIS Specialist 10/98 5 2 Health Specialists, Health Economist, MIS S U Specialist, Procurement Specialist 03/99 7 PTL, Public Health Spec., Procurement S U Spec., Economist, MIS Spec., Operations Officer, Team Assistant 06/99 7 PTL, Task Manager, Public Health Spec., S S Procurement Spec., MIS Spec., Operations Officer, Team Assistant 12/99 4 Task Manager, MIS Spec., Procurement S S Spec., Team Assistant 06/00 4 . 2 Health Specialists, Economist, S S Procurement Specialist ICR 07/00 4 2 Health Specialists, Economist, Procurement Specialist 15 Annex C Annex C. Comments from the Borrower NATl0NA BANK OF HUNGARY DEPJARXANT FOR EUROPEAN IEGRATION * AND INTERNATIONAL ORGAN1qSAIONS Alain Barbu Manager Sector and Thematic Group Operations Evaluation Department Dear Mr. Barbu, Re: Hungary: Health Services and Management Project (Loan 3597) Draft Project Performance Assessment Report (PPAR) Thank you for the draft PPAR of the Health Services and Management Project. We agree with the main findings of the report and have no comments on it. Waiting for the final report Budapest, November 14,2001 Best regards, Akos Cserds General Manager H-850 Budapasr V. Szab,,sa4 16, V9. Pbon- (36-1)302-3000 Telefai- e36-1)332-3973  17 Annex C PANZIJGYMINIMSUUM TARSADALMI KOZKIADASOK F6OSZTALYA 1051 BUDAPEST, J6ZSEF NADOR TAR 2-4 Postacim: 1369 Budapest, Postaffik 481 Telefon. 318-2066 Telefax 318-2570 Nytm.s. .S \20", To Mr Alain Barbu 29.11.2001 Manager Sector and Thermatic Group Operations Evaluation Department -Washington D.D 20433 1818 H Street N.W. Dear Mr. Barbu, Re: HUNGARY: Health Services and Management Project (Loan 3597) Draft Project Performance Assessment Report According to the above performance assessment report and the project oucome summary, I do not have additional recommendations. I also aggree with the detailed evaluation and I can accept that the project in spite of many difficuolties, has some unambiguous benefits. Yours s'ncerely Anna rces, Szikszai