ICRR 11119 Report Number : ICRR11119 ICR Review Operations Evaluation Department 1. Project Data: Date Posted : 05/22/2002 PROJ ID : P003965 Appraisal Actual Project Name : Health IV:Improving Equity Project Costs 134.3 66.44 & Quality of Care US$M ) (US$M) Country : Indonesia Loan/ US$M ) 88.0 Loan /Credit (US$M) 48.69 Sector (s): Board: HE - Health (100%) Cofinancing US$M ) (US$M) L/C Number : L3905 Board Approval 96 FY ) (FY) Partners involved : Closing Date 03/31/2001 03/31/2001 Prepared by : Reviewed by : Group Manager : Group : Martha Ainsworth Andres Liebenthal Alain A. Barbu OEDST 2. Project Objectives and Components a. Objectives The overarching objective of the project was to improve health outcomes . While the major beneficiaries were expected to be women and children from poor and non -poor households, the poor were expected to benefit disproportionately because of the project activities that enhanced equity (SAR, p. ii). Specific relevant objectives were to improve: (1) the quality of basic health services (BHS) that benefit the poor, through decentralized health management, improved efficiency in BHS delivery, and adoption of quality assurance standards; (2) access and utilization of health services by the poor; (3) resource allocation and revenue generation, through financing of cost -effective services, increased cost recovery, and retention of the revenues by the health sector and facilities where collected . (This objective was implicit in the SAR and explicit in the components; it was formally added in May 1997). b. Components The project was implemented in five provinces (West and East Kalimantan, East Java, Nusa Tenggara Barat (NTB), West Sumatra), with the following components in each province (SAR, Annex 5): (1) Improvements in health facilities ($74.2 million): (a) upgrade and construct health facilities to improve delivery of basic health services; and (b) provide discretionary Special Assistance Funds (SAF) and other resources for operational and recurrent costs to support quality assurance (QA) and other BHS programs initiated by provincial, district, and frontline providers . (2) Institutional development ($39.2 million): (a) establish quality assurance mechanisms; (b) support training/curriculum development activities; (c) develop effective performance -linked incentives; (d) replicate innovations from the Third Health project and launching other pilot programs; and (e) strengthen the capacity of Central MOH units to give technical guidance, disseminate quality assurance and other innovations, and carry out oversight functions. (3) Resource allocation and revenue generation ($1.4 million): technical assistance, data collection and analysis, materials and workshops to design and promote facility -level fee increases and revenue retention, and more effective targeting of health subsidies to appropriate recipients and activities . c. Comments on Project Cost, Financing and Dates The devaluation of the rupiah due to the East Asian economic crisis in 1997, the realization by the government that they qualified for IDA resources, and the results of the mid -term review resulted in two partial loan cancellations totalling $39 million. The TM notes that the second cancellation was due in part to the complexity of the project and resulted in downsizing the civil works component, which was having problems . The project objectives were not revised; after the cancellations the project focused on quality assurance and the pilot projects, and the government declined to use the loan to finance additional planned technical assistance . 3. Achievement of Relevant Objectives: The three supporting objectives were partially achieved, although no evidence is presented that they improved health outcomes in the aggregate or among the poor, who were expected to benefit disproportionately from the project . BHS ): (1) Quality of Basic Health Services (BHS) The target of coverage of 886 health centers for BHS and 323 for QA (Annex 2, SAR) was substantially exceeded, with coverage of both BHS & QA training in 1,559 health centers (Source #1, Table 7). The number of facilities rehabilitated also exceeded the target (2,685 health centers and sub-centers targeted (Task Manager) vs. 3,700 actually rehabilitated (ICR)). The extent and quality of repairs achieved for this higher number of facilities and given the reduction in the loan is not documented; clearly the level of rehabilitation and repair per facility had to be more modest than planned . The target of adopting quality assurance (QA) mechanisms for all 10 BHS in 80% of health centers was only partly achieved. As of February 2000, QA for exactly 4 BHS (immunization, acute respiratory tract infections, antenatal care, diarrhea) were adopted by 85% of health centers in W. Sumatra, 62% in W. Kalimantan, 97% in E. Kalimantan, 25% in NTB, and 69% in E. Java, with the remaining health centers reporting more than 4 (except for 16% of health centers in E. Java reporting QA for none of the 10) (Source: #1). Among these four BHS the compliance rate (the % of facilities for which at least 80% of procedures are followed correctly ) averaged from 60-70% (Source #1, Table 5). The SAR set as an objective a 10% annual increase in the visit rate (the number of monthly visits/catchment population), presumably to measure the demand response to higher quality health services (SAR, Annex 3). However, trends for 1995-1999 reveal a decline in the visit rate throughout the period in E . Java and NTB, an increase in E. Kalimantan, and in W. Kalimantan and W. Sumatra a decline before the crisis (1997), followed by an increase (Source #2). This was not an appropriate indicator for measuring the impact of the project (see below). (2) Access and use of health care by the poor : No evidence is presented in the ICR concerning achievement of this objective. (3) Resource allocation and revenue generation . All provinces except W. Kalimantan substantially adjusted user fees upward by 1999, although they did not reach as high as proposed and did not cover unit costs (Source #1, Table 17). No objective for fee retention by the health facilities is stated in the SAR; Source # 1 (p. 22) maintains that the objective was 100% fee retention. Pressures from resource-starved local governments following the East Asian economic crisis made this difficult to achieve : only 25%-50% of revenues were retained at facilities in most districts (Source #1, p. 22). In many cases most revenues were used outside the health sector . The ICR maintains that allocation of resources to the health sector improved significantly during the project, both from the government budget, the Social Safety Net of the ADB (not a part of the project), and SAF resources, but no evidence on the amount of the increase in health spending is presented to support this claim . Source #1 (p. 23) shows evidence of increased per capita health budget in nominal terms in the five provinces, but given the high rate of inflation during the project period, it is not possible to conclude what happened in real terms. The Task Manager indicates that the recurrent budget for primary health care declined in real terms (1993 constant prices), from 1,988.9 to 1,785.0 billion IDR over the period 1996/97 - 2000. Sources: #1: MOH, Republic of Indonesia, "Annual Report, Health Project IV, June 1999-May 2000", August 2000; #2: National Institute of Health Research and Development, Health Services Research and Development Center, MOH. "Report of Performance Indicator, Health Project IV ", May 2000. 4. Significant Outcomes/Impacts: User fees were raised at health facilities, based on a study of the ability and willingness to pay done during project preparation, and a system to exempt the poor was developed . At least a portion of the fee revenue was retained in most health centers . Quality assurance improvements were widely introduced for four basic health interventions that are believed to disproportionately benefit the poor . Four small pilot projects were successfully implemented and provided a testing ground for innovative approaches to improving the health care system : (a) the "Family Doctors experiment" (training 25 family doctors to go into private practice, financed on a capitation basis by the insurance scheme for government employees ); (b) the "SKM experiment" that used non-medical administrators to manage health centers; (c) a risk-pooling experiment; and (d) the "rational drug use" (RDU) experiment These projects were not formally evaluated, but the RDU was eventually implemented in all of the health centers of the five provinces, reducing misuse of antibiotics from a range of 49-73% misuse in 1996 to 20-42% in 2000. Misuse of injection was reduced from a range of 17-57% in the 5 provinces in 1996 to 2-18% in 2000. (Source: Task Manager) The monitoring and evaluation component mustered significant statistics to monitor the project's outputs . However, very little of this information was used in the ICR and it provides no information on the completeness and accuracy of reporting or the usefulness of these data for real -time decisions. 5. Significant Shortcomings (including non-compliance with safeguard policies): No evidence was provided that access by the poor was improved or that health outcomes improved in general or among the poor. Many of the proceeds from increased cost recovery are not being retained by the health facilities or the health sector, reducing the incentives for health staff to improve quality . The civil works component had several problems : (a) inadequate preparation; (b) insufficient capacity by implementing agencies in the provinces and districts to implement civil works activities; and (c) irregularities in the bidding process ("characterized by widespread (and unregulated) corrupt practices") due to the poor qualifications and lack of experience of the members of the District Bidding Committees . The SAF sub-component was implemented late in the project and was often not effectively linked to the preparation of development plans and budgets; further, the introduction of the Social Safety Net program by the ADB offered competing sources of funding with fewer requirements, weakening the incentives to base resource requests on careful planning . Performance-linked incentives were not successfully implemented . 6. Ratings : ICR OED Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately Satisfactory [OED's moderately sat. rating has no equivalent in the ICR's 4-point rating scale]. The three relevant objectives were to improve the quality, equity, and finance of health care; achievement of the financial objective appears to be moderately satisfactory, achievement of the quality objectives to be moderately satisfactory; and evidence that the project had an impact on equity could not be established. Institutional Dev .: Substantial Modest Results were highly uneven across provinces and levels of administration . Decentralization and cost recovery were implemented without adequate training and preparation. The training of staff in QA was less than needed for full commitment to quality improvement. Sustainability : Highly Likely Highly Likely Expansion of the RDU to all health centers in the 5 provinces, of the QA program to 8 additional provinces in the follow-up loan, and of block grants to all 314 Indonesian provinces is evidence of continued government commitment to the activities and reforms supported by the fourth project. Bank Performance : Satisfactory Satisfactory The ICR is very critical of preparation, but OED finds it satisfactory, based on ifnformation provided in the SAR. Borrower Perf .: Satisfactory Satisfactory Quality of ICR : Unsatisfactory NOTE: NOTE ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness. 7. Lessons of Broad Applicability: Lessons for this project (not necessarily broadly applicable outside of Indonesia ): The ICR maintains that this project had too many components and was overly complex for the capacity of the implementing agencies, particularly implementation of the civil works activities . Future projects should focus on fewer interventions. The project is a good example of the successful use of pilot projects to test new approaches for broader application. However, it also provides a good example of the difficulties of coordination and implementation in "scaling up" a successful pilot project to improve service quality . The visit rate is not a good indicator for measuring the impact of quality improvements on the demand for services, since service quality is only one factor that can affect demand . During the course of this project, the visit rate was likely reduced by the introduction of user charges in public health facilities and raised by the East Asian economic crisis as patients shifted from more expensive private to less expensive public sources of care . If improvements in the quality of services truly improved health outcomes, then holding other factors constant one would expect a decline in the visit rate for these conditions (not an increase, which was the target in this project). 8. Assessment Recommended? Yes No Why? (1) These are important reforms in health finance and quality improvements by a key borrower, but the ICR provides very little in the way of specifics to document exactly what happened or why . It would be of interest to understand the key factors that led to the implementation of cost recovery /exemption reforms, on the one hand, and whether it was indeed the complexity of the initial project design (as hypothesized in the ICR) or other factors that were responsible for shortfalls in implementation . (2) This project demonstrates the potential role of pilot projects in pushing the health reform envelope . An in-depth investigation of their outcomes, impacts, and evaluability would yield lessons for future projects, as would evidence of the difficulty of 'scaling up' a previously evaluated pilot project. 9. Comments on Quality of ICR: While the ICR describes qualitatively the results of the project and offers insights for success and failure, it provides inadequate evidence to substantiate the evaluative ratings . Few indicators are mentioned, and those that appear are not broken down by province . There's no evidence presented on changes in resources allocated to the health sector . The visit rate in Annex 1 is shown for the end of the project, but no numbers are offered for the beginning of the project to asses the change . Many of the indicators in the annex are left blank . The lack of specificity and lack of care used with the evidence makes it extremely difficult to rate the components and will limit the amount of learning and the institutional memory from this project . The ICR claims that the monitoring and evaluation component was not implemented, yet the Task Manager provided several documents to OED from the monitoring and evaluation units with indicators for all five years of the project, by province. Very little discussion is offered on the impact of the East Asian crisis on project implementation or of the specific issues that led to the second cancellation . The information provided in this ES was gleaned from a discussion with the task manager. The drop in real wages of health workers could have affected the quality of services and their time allocation. There is no mention at all of the impact of the dramatic political changes (the fall of the Suharto regime) on project performance.