ICRR 12221 Report Number : ICRR12221 ICR Review Operations Evaluation Department 1. Project Data: Date Posted : 09/02/2005 PROJ ID :P003967 Appraisal Actual Project Name :Id-fifth Health Project Project Costs 56.2 39.9 US$M ) (US$M) Country :Indonesia Loan/ US$M ) Loan /Credit (US$M) 44.7 32.7 Sector (s):Central ): government US$M ) Cofinancing (US$M) administration; Sub-national government administration; Health L/C Number :L4374 FY ) Board Approval (FY) 99 Partners involved : Closing Date 03/31/2004 07/31/2004 Evaluator : Panel Reviewer : Group Manager : Group : Denise A. VaillancourtSoniya Carvalho Alain A. Barbu OEDSG 2. Project Objectives and Components a. Objectives The project objectives were to : (a) achieve greater efficiency of utilization and equity of distribution of health personnel; (b) increase the skills of health professionals; and (c) improve the quality of health professional practices . In support of these PDOs, the project was designed to support activities to : (a) strengthen the decentralization of planning, deployment and management of personnel and the role of the private sector; (b) strengthen the licensing of health personnel and support professional associations and quality assurance; and (c) improve the quality of pre-service and in-service training and strengthen the accreditation of institutions . b. Components (or Key Conditions in the case of Adjustment Loans ): I. Province Components (focusing on three poor provinces with inadequate access to quality services ): A. Improve the efficiency of utilization and equity of distribution of health professionals (cost at appraisal: $17.0 million; actual cost $6.0 million or 35 percent of estimate), including investments to improve personnel planning and management (technical assistance, training, fellowships and equipment ) and grants for innovations in staff deployment and productivity (pilots to test incentives and performance ); B. Improve the quality of health professional practices (cost at appraisal: $5.5 million; actual cost $3.7 million or 67 percent of appraisal estimate), including investments to strengthen the regulatory system, strengthen professional associations and promote quality assurance for health services; C. Improve the quality of health professional training (pre-service and in-service for nurses and midwives and in-service only for doctors) (cost at appraisal: $22.8 million; actual cost: $10.6 million or 46 percent of appraisal estimate), including improvements to training methods and programs (competency-based and case-based methodologies linked to quality assurance ); and strengthening of training institutions (quality, institutions, teachers, management, materials); and D. Project management in each of the three provinces (cost at appraisal: $3.9 million; actual cost: $4.0 million or 102 percent of appraisal estimate). II. II . Central Components E. Build central support for health professionals (cost at appraisal: $5.1 million; actual cost: $4.0 million or 78 percent of appraisal estimate), including investments to strengthen planning capacity and technical support and to improve the professional environment (regulation, licensing, accreditation, etc .). At the MTR this component was slightly shifted to place more emphasis on capacity for human resources policy development . F. Project management (cost at appraisal: $1.9 million; actual cost $0, as this was scaled back and incorporated into component E at mid-term). Fellowships (not a standalone component at appraisal, but accounted for as a separate component /cost in the ICR, with actual costs estimated at $ 11.6 million). c. Comments on Project Cost, Financing, Borrower Contribution, and Dates Total project cost was $39.9 million or 71 percent of the appraisal estimate . Of the total actual cost, $32.7 million was financed by IBRD or 73 percent of the original loan amount, with the remainder financed by government in the amount of $7.7 million (vs. $11.5 million planned). Due to slow implementation several project amendments canceled some loan funds and the closing date was extended by four months . 3. Relevance of Objectives & Design : The relevance of the project objectives is substantial. They were consistent with the 1997 CAS, which emphasized (a) greater decentralization through increased autonomy and accountability; (b) increased efficiency and productivity in public and private sectors, including raising the quality of health services; and (c) enhancing equitable development. They were also consistent with Government's 7th 5-year plan, which called for greater decentralization of management and planning, renewed focus on quality, and an emphasis on equity and private sector development . Objectives are also reflective of Bank's analytic work on Indonesia's health workforce, health planning and budgeting and public expenditures and the poor . However, it is not clear how this project fits into general health issues in Indonesia and the Bank's overall strategy /objectives for Indonesia's health sector (addressed through a very large portfolio of projects and analytic work ). The relevance of the design is modest. The design reflected past lessons, especially the importance of addressing quality as well as quantity, and demand as well as supply issues; and it incorporated the importance of incentives, which are key to human resources performance . It was also appropriately limited in scope from a number of perspectives. It did not support health service components, and limited health financing only to that related to HR issues. It excluded pre-service training of doctors and limited the geographic scope to 3 provinces. However, the premise was somewhat naive that HR investments by themselves would translate into quality improvements . While the intention was to target project benefits to the poorest beneficiaries, the design did not fully assess needs and challenges of these populations and alternative ways of lifting constraints to services access . The design was also lacking in that it did not benefit from adequate baseline data or an M&E framework and plan (see Section 6 below). 4. Achievement of Objectives (Efficacy) : The project was not successful in achieving its DOs . Outcome is difficult to measure because of the absence of clear indicators, targets and relevant data for DO -level indicators, and a dearth of project -related information prior to 2002. 1. Greater efficiency of utilization and greater equity in the distribution of health personnel was not achieved . Data are not available to show trends in availability of health personnel in project districts . While most districts were using project-financed software for HR planning and management, plans were of varying quality, and it is unclear how many were implemented. Difficulties in obtaining reliable and timely information from districts supported by the project suggest that systems were not functioning well even during the final two years of the project . Project inputs (software, training) were late in coming and not fully assessed, and have not affected more equitable staff allocations. Staff incentive schemes were still at an embryonic stage at project completion, revealing no clear links to improved performance, workloads or (for the most part) more equitable deployment to under -served areas. The project supported incentives for improved deployment and performance, but they have had little impact on policies to date. Interventions to strengthen the role of private providers were not implemented at all . Central-level capacity-building interventions were initially not responsive to district needs and even after they were reoriented around policy development and technical assistance, they did not culminate in meaningful changes in workforce policy. 2. Increased skills of health professionals in the project provinces was partially achieved . All training institutions in the three provinces were improved and accredited . The accreditation instrument was developed under the project . Sister School Programs were successfully established, with some delays . Short-course nurse-training schools were successfully converted to academies except for two private ones in Central Java . Final evaluation was unable to determine whether upgraded status reflects improved teacher qualifications, facilities or equipment . Most of the training provided was competency -based. Learning resource centers were established in all three provinces although their utilization rates were low at project completion . Progress in establishing distance learning initiatives was limited. Project supported improvements to provincial training institutions included : (1) updated curricula; (2) plans for future upgrading of facilities; (3) equipment, books, journals and other learning materials; (4) workshops and studies; and (5) signed memoranda of understanding between training institutions and clinical training sites . Some 3640 public health staff received fellowships from the project to support long -term training, mostly to improve clinical skills, with some preference given to staff from poor, remote locations . No assessment of this training was made on its success in meeting actual HR needs . Most recipients returned to their jobs, but many changed jobs and transferred to central positions . 3. Improved quality of health professionals' practice was not achieved . Most districts claim to have licensing systems, but these were in place prior to the project and the contribution of such systems to the quality of health professionals' practice has not been documented . The project did support an improved system piloted in one province, but its future is uncertain because of an emerging legal framework . Project support strengthened professional associations for doctors, nurses and midwives and increased membership in the project provinces, but there is not evidence that this has contributed to improving service quality provided by members . A substantial amount of quality assurance (QA) training was provided by the project and many decentralized entities reported establishment of QA teams and programs . However, there is no evidence whether this has had any effect on actual quality or utilization rates. Some feedback on the training noted it to be too theoretical at the time of the MTR; and the number of hospitals operating QA programs declined towards the project's end . 5. Efficiency : Project efficiency is negligible. The 1997/98 economic crisis limited the availability of GOI resources (financial, human, managerial) to support project implementation. Decentralization was largely politically driven; and reluctance to address civil service reform left ambiguities in roles and responsibilities of districts . Late release of the Government budget also impeded some activities . Project management was a major impediment to project implementation. Project management was undermined by : failure to monitor and report on project performance, inadequate staffing, procurement delays, lack of needed technical assistance, inadequate monitoring of provincial and district activities, absence of annual work plans, failure to independently assess outputs, failure to follow up on audit recommendations, inadequate procurement capacity, slow follow -up on MTR recommendations. There appears to be a lack of efficiency in the project's design . The ICR suggests that there may have been more effective ways to ensure that the project would benefit the poorest and most remote populations and to ensure the quality of professional health service delivery . 6. M&E Design, Implementation, & Utilization: Monitoring and Evaluation was weak both in its design and implementation . Design : The PDOs presented in the project logframe in the PAD are different than those stated in Part A of the PAD and DCA. Key performance indicators are ill -defined, and lacked both baseline data and targets . The design did not provide for an M&E framework or plan. There were no appropriate indicators for measuring achievement of objectives such as efficiency of utilization, quality of staff and services, and equity . Project conditions on M&E were vague and boilerplate (calling for annual reviews and a MTR ). The Project Implementation Plan covering the first 18 months of implementation included nothing on setting up and implementing project M&E . Evaluation studies were mentioned in the detailed description of selected components, but were not well -defined. Implementation : It was only after the MTR that the list of indicators were pared down to 7 performance indicators, but (still) without targets. Regular project monitoring and reporting began only in mid -2002, thus limited to the two final years of implementation. Utilization : There was insufficient data generated to be particularly useful for finetuning project design and implementation and for decision-making. 7. Other (Safeguards, Fiduciary, Unintended Impacts--Positive & Negative): Government did not fully honor its counterpart financing obligations . 8. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Unsatisfactory Unsatisfactory Institutional Dev .: Modest Modest Sustainability : Unlikely Unlikely Bank Performance : Unsatisfactory Unsatisfactory Borrower Perf .: Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness . 9. Lessons: Improvements to quantity and quality of inputs without making necessary systems changes to use those inputs effectively will not culminate in improved performance . Also important are performance monitoring and incentives to motivate personnel and effective leadership for policy reform and implementation . The absence of a carefully developed monitoring and evaluation framework and plan can seriously compromise project implementation and ongoing improvement; and it can undermine the ability to document the achievement of objectives. It may not be feasible to make substantial progress in the human resources domain in some settings unless efforts are preceded (or at least accompanied by) needed civil service reform. 10. Assessment Recommended? Yes No 11. Comments on Quality of ICR: The ICR provides an adequate overview with the very limited information and data available . The difficulty in assessing the achievement of objectives is a factor of an ill -conceived logframe, lack of baseline data, inappropriate indicators and collection of data only during the last two years of the project .