ICRR 12680 Report Number : ICRR12680 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 09/04/2007 PROJ ID : P051174 Appraisal Actual Project Name : Health Investment US$M ): Project Costs (US$M): 20.00 20.50 Fund Project Country : Moldova Loan/ US$M ): Loan /Credit (US$M): 10.0 10.5 Sector Board : HE Cofinancing (US$M ): US$M): 8.4 10.0 Sector (s): Health (95%) Central government administration (5%) Theme (s): Health system performance (29% - P) Rural services and infrastructure (29% - P) Other communicable diseases (14% - S) Population and reproductive health (14% - S) Access to urban services and housing (14% - S) L/C Number : C3408 Board Approval Date : 08/22/2000 Partners involved : Govt. of Netherlands Closing Date : 11/30/2005 12/30/2006 Evaluator : Panel Reviewer : Group Manager : Group : Judyth L. Twigg Martha Ainsworth Alain A. Barbu IEGSG 2. Project Objectives and Components: a. Objectives: The Development Credit Agreement (DCA) states that the objectives of the project were to improve the health status of the Moldovan population and to increase the quality and efficiency of the public health sector by: (a) guaranteeing universal access to a minimum package of health services; (b) modernizing emergency services and primary health care; (c) reducing excess capacity in the health sector; (d) strengthening health sector institutional capacity; and (e) supporting the development of tuberculosis and HIV/AIDS strategies. The Project Appraisal Document (PAD) and ICR add that these objectives would be achieved by improving access to essential services for the poor . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): I. Policy Development and Institutional Strengthening (US$1.67 million estimated, US$2.50 million actual) A. Health Policy Development . Development of a health reform strategy, key policies, and pertinent legal framework for health sector reform, including arrangements for : (i) health care financing (sources and resource allocation); (ii) a basic package of health care services; (iii) provider-payer split; (iv) user charges; (v) human resources; and (vi) tuberculosis and HIV/AIDS strategies. B. Communication of Health Sector Reform . Development of a public information campaign and other communication activities to inform shareholders about the sector reform . C. Management and Technical Training Program. Management and technical training programs, including (a) General Practice Program, involving : (i) training of general practitioners and retraining of specialists on primary health care; and (ii) training of nurses at the primary and secondary health care level; and (b) Management Program, including training of health services managers and administrators at central, judet, and health care facility levels to provide them with the necessary business skills to manage health services. II. II. Health Investment Fund (US$17.72 million estimated, US$17.40 million actual) This component would establish the HIF, with the primary aim of sponsoring change through demonstrating improvements in access, quality, and efficiency of health care services at all levels of the system. It would create a competitive mechanism to allocate resources among the Municipality of Chisinau and Judet Health Authorities and, to a lesser extent, hospitals, mainly financing regional restructuring proposals that include development of emergency care services and primary health care based on general practitioners. There were two sub-components: A. Emergency and Primary Health Care . Funds would be made available on a competitive basis for these services, replenished in three phases, depending on compliance with a restructuring plan . B. Hospital Refurbishment. Limited investments by the HIF in key hospital refurbishments up to $ 300,000 per facility, funded conditional on the consolidation or closing of unnecessary services . III . Project Management and Evaluation (US$0.61 million estimated, US$0.6 million actual) III. A. Project Management, to finance a small unit to provide project management, coordination and monitoring of the health sector reform program, including local and foreign technical assistance for project management, financial management, and procurement. B. Project Evaluation . To finance epidemiological, social, institutional, and economic assessments of the project at the beginning of implementation, mid-term, and project end. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The closing date was extended twice, once (for 9 months) due to elections, and the second time (for 6 months) to ensure completion of activities that were financed with additional funding that became available because of currency fluctuations between the SDR and the US$ . The difference between estimated and actual costs is because of currency fluctuations between the SDR and the US$ . 3. Relevance of Objectives & Design: The relevance of the project’s objectives is substantial. The Bank’s most recent Country Assistance Strategy (2004) stresses the need to continue with reform of the health care system, focusing on removing excessive infrastructure, strengthening primary health care, and increasing access to essential health services. The project is also directly supportive of the Government ’s 2004-2006 Economic Growth and Poverty Reduction Strategy Paper, which includes a program to render health care services through the mandatory health insurance system. The relevance of the project’s design is substantial. The project’s design contains a clear results chain connecting objectives to components to indicators, although some output and outcome indicators are not differentiated. The project maintained a focus on equity and access to health care during the restructuring process. Major risks were taken into account, although the project failed to account for some political economy risks surrounding the differential costs and benefits of hospital restructuring to various stakeholders. The inclusion of an explicit component for communicating the benefits of health sector reform indicates the strong degree to which risks were identified and mitigated . The overall relevance of the project is substantial. 4. Achievement of Objectives (Efficacy): Based on substantial achievement of increasing the quality and efficiency of the health sector and modest achievement of improving the health status of the Moldovan population, efficacy is rated as substantial. Increasing the quality and efficiency of the public health sector : Substantial . Outputs : Health care providers now receive public funds on a contractual basis to provide services included in the basic package. A population-based resource allocation mechanism has been implemented, with refinements still under development . The project financed refurbishment of 100 Primary Health Centers (PHC), or 25% of all PHC facilities, with variations in the extent of the refurbishment and equipping with medical devices . 300 health managers, 749 physicians (30% of the total need), and 1,474 nurses received training. Numerous evidence-based clinical guidelines and treatment protocols were developed. Fifteen students obtained public health and health care management training abroad. Equipment was provided and civil works were undertaken to improve the quality of health facilities, including the purchase of portable medical equipment kits for 100% of GPs, lab equipment for 85% of PHCs, medical equipment for emergency rooms and operating theaters for all raion hospitals and some other hospitals, ambulances for raion hospitals and Chisinau municipality, and training equipment for the Medical College and the Emergency Training Center . TB and AIDS strategies were approved early in the project period. Outcomes : The number of hospitals (276 to 116), hospital beds (48,261, or 112.3/10,000 population, to 22,961, or 63.9/10,000 population), and acute hospital beds (62.8/10,000 to 56.8/10,000) were reduced from 1998 to 2005. Most downsized or closed hospitals are in rural areas, so that further downsizing is necessary in Chisinau municipality . The number of family doctors increased from 1285 in 1998 to 2096 in 2004, decreasing the population-to-family-doctor ratio from 2951:1 to 1713:1 (this did not meet the original target of 1500:1). According to pre- and post-test training on practical skills and theoretical knowledge, the training course for family doctors produced a 225% increase in theoretical knowledge and a 66% improvement in practical skills; nurses showed a 107% improvement in theoretical knowledge and a 53% improvement in practical skills . The percentage of respondents to a beneficiary survey who had called for emergency assistance decreased from 52% in 2002 to 45% in 2003. Informal payments offered to doctors remained at the same level from 2002 to 2003 (52% of respondents), but the number of people who said they would prefer to pay only official charges increased from 41% to 51%. The project did not take advantage of the variations in extent of refurbishment and equipping of PHCs to provide evidence on the degree to which these changes made a difference in access and utilization of health services . The ICR also does not include information on changes in the quality of care . In addition, the Borrower's ICR points to some serious remaining concerns regarding the limitations of a short, 4-week retraining program in family medicine for existing doctors, the human resources challenges facing the entire medical profession (particularly family medicine in rural areas ) and the high continuing level of discomfort among some doctors for some of the responsibilities of family medicine . Improving the health status of the Moldovan population : Modest. Modest Outputs : The percentage of all public health expenditures allocated to primary care increased from 21% in 2000 to 27.6% in 2005 (not meeting the target of 35%). A national health insurance scheme, guaranteeing access for the non-employed and other vulnerable groups to a minimum package of health services, was established. Patients do not pay user fees for the services covered in the package . Outcomes : The ICR provides very limited evidence about improved health outcomes on average and even less on changes in health outcomes among the poor. The fatality rate in emergency cases decreased from 10.2 per 100,000 residents to 8.9 per 100,000 inhabitants, although some confusion persists over the definition of “emergency cases�; this did not meet the original project target of 7.65/100,000. The infant mortality rate was reduced 33%, from 18.54 per 1000 live births in 1999 to 12.42 in 2005. As the ICR states, however, reduced infant mortality is a result of many factors, in addition to investments in primary health care. The summary of the Borrower’s ICR cites improvements in life expectancy and overall mortality rates, but it does not quantify those improvements . A representative population survey found that in the second year of the health reform, 30% of rural residents and 18.4% of urban residents considered that access to health services had increased, with increased access mentioned most frequently by people over 55 years of age. However, the percent reporting no change or a reduction in access was not reported. The Borrower's ICR states that access to essential services among the young, mothers, and the elderly has improved significantly . There were no baseline or end of project surveys to compare access at two points in time, on average or among the poor . 5. Efficiency (not applicable to DPLs): Efficiency is rated as substantial . During project preparation, a thorough cost-effectiveness analysis was conducted showing a present value of net benefits, after investment and recurrent costs, of almost $10 million and an internal rate of return of 39%. The ICR does not explain whether the activities forseen by that cost-benefit analysis were actually implemented; no overall internal rate of return or net present value calculations were carried out at the end of the project . The Bank team effectively mobilized the project’s financial resources to engage and leverage the government to achieve the project development objectives by: (i) quickly mobilizing a PHRD grant to help the implementation of the national insurance scheme; (ii) including disbursement triggers on both the credit and the grant that were linked to the overall health care restructuring and reform objectives; and (iii) leveraging funding at local levels to maximize results at individual project sites, particularly with regard to leveraging additional financing for infrastructure. The Borrower’s ICR states that a significant benefit of the project has been the demonstration that managers will be willing and able to mobilize and leverage additional funding if they see a benefit from the activity. According to the Borrower’s ICR, some funds were spent to procure ambulances that could have been more effectively spent on upgrading additional family medicine facilities. ERR)/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 39% 100% ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Based on substantial relevance, substantial efficacy, and substantial efficiency, the overall outcome rating is Satisfactory . a. Outcome Rating : Satisfactory 7. Rationale for Risk to Development Outcome Rating: The Government has demonstrated its sustained commitment to the project ’s goals despite significant reshuffles, including four different ministers of health and two reorganizations of the Ministry of Health (MOH). The Bank is currently working with the MOH on the design of a new project, “Health Services and Social Assistance,� that was scheduled to be presented to the Board in June 2007; that project contains components to continue hospital restructuring and strengthening of primary health care . The ICR states that the capacity of the MOH still needs to be enhanced, particularly given the closing of the project ’s Project Coordination Unit (PCU). Relatively low salaries in the public sector make it difficult to attract and retain competent staff. The average age of family doctors is around 50, so that there is a risk of a coming shortage of family doctors to staff PHC facilities, particularly given their relatively low salaries and poor working conditions in rural areas . Primary care centers still have insufficient financial autonomy from the raion health authorities, which may limit their ability and /or incentives for further growth and development. The Borrower’s ICR states that “reform exhaustion� is evident at all levels, but that all levels agree that reforms have gone too far for there to be a turning back or deceleration at this stage . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Ensuring Quality -at- Entry : Satisfactory. Project preparation consulted stakeholders and at-Entry: beneficiaries, including discussions with patients, health care professionals, health organizations, and other donors. Alternative designs were considered and rejected for clearly stated reasons . Thorough institutional and economic analyses were conducted; the institutional analysis was key to good project design. The inclusion of competitive mechanisms and triggers for disbursement linked to the national health policy reform was creative and innovative . Project design carefully links objectives to components and target indicators. However, the design lacked adequate outcome indicators on access to health care among the poor and on the efficacy of the newly-trained family medicine practitioners. More attention could have been paid to the specification of benefits in the minimum guaranteed package of health services. Quality of Supervision : Satisfactory. There were three different TTLs on the project, but the ICR states that the transitions were smooth. Supervision missions effectively engaged stakeholders at all levels. Communication activities, media relations, and public opinion surveys throughout the project were very effective in ensuring and measuring public awareness and support for the reforms, mitigating a risk common to health restructuring projects . The Bank successfully mobilized a PHRD grant to help with the implementation of the national health insurance policy, and worked well with WHO experts on the implementation of health financing reforms . The Bank team worked effectively with Government to resolve several potential problem points during implementation, including lines of budgetary authority of the PHCs and the raion hospitals. at -Entry :Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Satisfactory 9. Assessment of Borrower Performance: Government Performance: Satisfactory. The Government remained committed to strengthening primary health care and reducing excessive hospital capacity through numerous political changes . The Government identified key financial, administrative, and political risks regarding the establishment of the health insurance scheme, and with the consultation of technical experts, it established clear rules regarding the Health Insurance Company's (HIC) legal status, fiduciary management, reporting requirements, oversight, and auditing. As a result, budget allocations for those who could not contribute to health insurance through payroll taxes have been sustained through government contributions. Implementing Agency Performance: Satisfactory. The PCU effectively coordinated and implemented the project, with substantial capacity developed in the areas of procurement and financial management. During implementation, the PCU effectively consulted health care providers to get their inputs on needed capital improvements, with funding to fulfill those requests then granted based on competitive mechanisms. The PCU is now the core implementation agency for the Bank-financed HIV/AIDS project, a Global Fund grant, and a Council of Europe Bank-financed Blood Security Project. a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: M&E Design was Substantial. The PAD contains a results chain from objectives to components to indicators, with a plan for data collection . The ICR states that the M&E framework had too many indicators, although the number does not seem to be excessive. Some of the indicators were repeated as both outcome and output indicators. There were not enough measurable outcome indicators, particularly regarding access to health services for the poor and the efficacy of the new family medicine practitioners . M&E Implementation was Substantial. The project supported and enhanced the work of the Center for Public Health and Management to collect data on the key performance indicators at the raion level . The Center then sent the data to the PCU, which sent monitoring and progress reports to the Bank on a quarterly basis. Data on a small number of indicators were not collected properly . M&E Utilization was Modest. The MOH involved the Center in the preparation activities for the follow -on Health Services and Social Assistance Project, indicating that the Center ’s data and analysis are in part driving the structure of the new project. The ICR states that most of the project’s gains in hospital closure and restructuring, and in the support of PHC facilities, were made early in the project . The ICR presents no evidence that the Center’s data and analysis were used during the project period to assess the project ’ s progress to data and make corrections in design or activities . a. M&E Quality Rating : Substantial 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): The National Health Insurance scheme, while providing full state -financed coverage for people who do not work, leaves approximately 25% of the population (the underemployed and informally employed) not covered. Out-of-pocket expenditures therefore remain a large percentage, perhaps the majority, of health expenditures, according to the Borrower’s ICR. As hospitals are closed and downsized, the ICR appropriately recognizes the importance of upgrading the quality of the remaining hospitals in order to provide acceptable quality of care . The enhancement of the PCU’s capacity under this project, particularly in terms of procurement and financial management, clearly had spillover benefits to other projects financed by the Bank and other donors. 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Based on substantial relevance, efficacy, and efficiency. Although some targets were not met, there was substantial progress toward the achievement of the development objectives. Risk to Development Moderate Moderate The Government is committed to Outcome : reform and institutional capacity has been established, but the reforms are in jeopardy of imploding if careful attention is not paid to building and sustaining the cadre of family doctors. Bank Performance : Satisfactory Satisfactory Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Tension can emerge between the goals of health care restructuring with an emphasis on the provision of primary care, on the one hand, and the emergence of a national health insurance scheme with a guaranteed package of basic benefits, on the other. As patients come to expect free health care under the latter, it is important to make sure that: (i) there is ample capacity developed in the PHC system to accommodate this demand; and (ii) there are financial and other incentives in place to support the development of the primary care sector. In the Moldovan case, physicians have sometimes been subject to demands for free care in an environment where they used to receive out -of-pocket payments for services, without systemically compensating them with higher salaries or other incentives . Because of the many stakeholders whose core interests can be threatened by restructuring, a clear and high-visibility communication and information dissemination strategy is essential for success . Creating an entity outside the MOH can facilitate the launching of reforms . The MOH can contain too many players with a vested interest in old strategies, procedures, and habits; a new vehicle is often a prerequisite for reform. The training, practice and financing of family doctors as part of a health reform program, is a lengthy process requiring sustained attention following initial design and implementation . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The quality of the ICR is satisfactory, with caveats . IEG appreciates the careful attention to presentation of data and evidence directly relevant to achievement of the project development objectives . However: The ICR does not explain some of the major institutions and processes involved in the project . The health insurance scheme, for example, is alluded to but not explained; the PAD (p. 20) indicated that the country was going to delay implementation of health insurance, and it remains unclear exactly what is involved in the health insurance system . The ICR is also not sufficiently detailed about the mechanisms that provided increased access to health services among the poor; in particular, it does not explain how the activities that the project financed led to this increased access. There is very little detail about the activities of the Health Investment Fund, its relationship (if any) to the health insurance system, and the functioning of its demand-driven, incentive-creation mechanisms. Annex 1 does not contain information on counterpart contributions and the ICR does not include the comments of the co-financers. a.Quality of ICR Rating : Satisfactory