Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004290 IMPLEMENTATION COMPLETION AND RESULTS REPORT IDA-4229 IDA-5160 ON A CREDIT IN THE AMOUNT OF SDR 143 MILLION (US$ 211.80 MILLION EQUIVALENT) TO THE REPUBLIC OF INDIA FOR A KARNATAKA HEALTH SYSTEM DEVELOPMENT AND REFORM PROJECT ( P071160 ) January 19, 2018 Health, Nutrition & Population Global Practice South Asia Region CURRENCY EQUIVALENTS (Exchange Rate Effective Jan 18, 2018) Currency Unit = Indian Rupee (INR) INR 63.86 = US$1 US$ 1.42 = SDR 1 FISCAL YEAR April 1 – March 31 Regional Vice President: Annette Dixon Country Director: Junaid Kamal Ahmad Senior Global Practice Director: Timothy Grant Evans Practice Manager: Rekha Menon Task Team Leader(s): Jorge A. Coarasa Bustamante ICR Main Contributor: Gandham N.V. Ramana ABBREVIATIONS AND ACRONYMS AF Additional Financing ANM Auxiliary Nurse-Midwife BPL Below Poverty Line CPS Country Partnership Strategy DCA Development Credit Agreement DHO District Health Officers DOHFW Department of Health and Family Welfare DPMU District Project Management Unit GOK Government of Karnataka HIMS Health Management Information System HIV/AIDS Human Immunodeficiency Virus / Acquired Immune Deficiency Syndrome IDA International Development Association IT Information Technology M&E Monitoring & Evaluation MDG Millennium Development Goals NCDs Non-Communicable Diseases NFHS National Family Health Survey NITI National Institution for Transforming India NRHM National Rural Health Mission OD Organizational Development PDO Project Development Objective PforR Program for Results PHC Primary Health Center PHCF Public Health Competitive Fund PHRD Japanese Policy and Human Resource Development Rs. Indian Rupees SPMU State Project Management Unit SAST Suvarna Arogya Suraksha Trust SDR Special Drawing Rights TB Tuberculosis PAD Project Appraisal Document US$ US Dollar VAS Vajpayee Arogyashree Scheme VHSC Village Health and Sanitation Committee TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 7 A. CONTEXT AT APPRAISAL .........................................................................................................7 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) ..................................... 10 II. OUTCOME .................................................................................................................... 17 A. RELEVANCE OF PDOs ............................................................................................................ 17 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 18 C. EFFICIENCY ........................................................................................................................... 20 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 22 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 23 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 25 A. KEY FACTORS DURING PREPARATION ................................................................................... 25 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 26 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 27 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 27 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 30 C. BANK PERFORMANCE ........................................................................................................... 31 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 32 V. LESSONS AND RECOMMENDATIONS ............................................................................. 33 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 35 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 44 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 46 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 47 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name KARNATAKA HEALTH SYSTEM DEVELOPMENT AND P071160 REFORM PROJECT ( P071160 ) Country Financing Instrument India Specific Investment Loan Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Related Projects Relationship Project Approval Product Line Additional Financing P130395-India: 27-Sep-2012 IBRD/IDA Karnataka Health Systems Additional Financing Organizations Borrower Implementing Agency Government of Karnataka Department of Health and Family Welfare Project Development Objective (PDO) Original PDO To improve health service delivery, public-private collaboration, and financing, particularly for the benefit of underserved andvulnerable groups in Karnataka. Page 1 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) PDO as stated in the legal agreement The original PDO (as stated in the DCA): To support the Program and to assist Karnataka in improving the utilization of essential curative and public health services particularly in the underserved areas and amongst vulnerable groups. Revised PDO (as stated in Project Paper for Additional Financing): To improve health service delivery, public-private collaboration, and financing, particularly for the benefit of underserved and vulnerable groups in Karnataka. The Revised PDO (as stated in the FA for AF): To improve health service delivery, public-private collaboration and financing, particularly for the benefit of underserved and vulnerable groups in Karnataka. FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 141,830,000 141,636,631 148,290,739 IDA-42290 400,000 219,126 219,126 TF-99435 70,000,000 69,736,129 65,237,657 IDA-51610 Total 212,230,000 211,591,886 213,747,522 Non-World Bank Financing Borrower 64,650,000 0 100,301,553 Total 64,650,000 0 100,301,553 Total Project Cost 276,880,000 211,591,885 314,049,075 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 22-Aug-2006 11-Jan-2007 16-Jul-2010 31-Mar-2012 31-Mar-2017 Page 2 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 10-Dec-2007 25.00 Other Change(s) 02-Dec-2010 95.76 Change in Results Framework Change in Components and Cost Reallocation between Disbursement Categories 25-Feb-2012 118.81 Change in Loan Closing Date(s) 31-Jul-2012 118.81 Change in Loan Closing Date(s) 26-Dec-2012 122.37 Additional Financing Change in Project Development Objectives Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Change in Disbursements Arrangements 24-Mar-2015 162.37 Change in Results Framework Change in Components and Cost Reallocation between Disbursement Categories 29-May-2015 168.83 Other Change(s) 17-Mar-2016 184.32 Change in Results Framework Change in Loan Closing Date(s) KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 13-Feb-2007 Satisfactory Satisfactory .58 02 10-Aug-2007 Satisfactory Moderately Satisfactory .58 03 21-Feb-2008 Moderately Satisfactory Moderately Unsatisfactory 25.58 04 18-Aug-2008 Moderately Satisfactory Moderately Satisfactory 37.74 05 24-Feb-2009 Moderately Satisfactory Moderately Satisfactory 56.70 Page 3 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) 06 30-Aug-2009 Moderately Satisfactory Moderately Satisfactory 65.41 07 12-Mar-2010 Moderately Satisfactory Moderately Satisfactory 77.42 08 17-Oct-2010 Moderately Satisfactory Satisfactory 96.35 09 12-Feb-2011 Satisfactory Satisfactory 98.15 10 21-Sep-2011 Satisfactory Satisfactory 119.39 11 05-Jun-2012 Satisfactory Satisfactory 119.39 12 14-Dec-2012 Satisfactory Satisfactory 122.37 13 16-May-2013 Satisfactory Satisfactory 129.57 14 20-Oct-2013 Moderately Satisfactory Moderately Satisfactory 132.76 15 17-May-2014 Moderately Satisfactory Moderately Satisfactory 145.63 16 02-Aug-2014 Moderately Satisfactory Moderately Unsatisfactory 150.24 17 03-Feb-2015 Moderately Satisfactory Moderately Satisfactory 157.64 18 31-Jul-2015 Moderately Satisfactory Moderately Satisfactory 168.83 19 03-Feb-2016 Moderately Satisfactory Moderately Satisfactory 180.03 20 15-May-2016 Satisfactory Satisfactory 189.92 21 17-Oct-2016 Satisfactory Satisfactory 196.79 22 24-Mar-2017 Satisfactory Satisfactory 213.53 SECTORS AND THEMES Sectors Major Sector/Sector (%) Public Administration 100 Sub-National Government 11 Financial Sector 100 Insurance and Pension 7 Page 4 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) Health 100 Health 72 Social Protection 100 Social Protection 10 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Private Sector Development 0 Public Private Partnerships 10 Social Development and Protection 0 Social Protection 16 Social Insurance and Pensions 16 Human Development and Gender 0 Disease Control 17 Tuberculosis 17 Health Systems and Policies 67 Health System Strengthening 33 Reproductive and Maternal Health 17 Child Health 17 ADM STAFF Role At Approval At ICR Regional Vice President: Praful C. Patel Annette Dixon Country Director: Michael F. Carter Junaid Kamal Ahmad Senior Global Practice Director: Julian F. Schweitzer Timothy Grant Evans Practice Manager: Anabela Abreu Rekha Menon Task Team Leader(s): Paolo Belli Jorge A. Coarasa Bustamante ICR Contributing Author: Gandham N.V. Ramana Page 5 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) Page 6 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context Original project 1. In the 1990s, Karnataka was one of the fastest growing states in India. Between 1993-94 and 2000-01, per capita state domestic product grew by 6.41% compared to a national average of 4.39 %1. This increase in per capita income had also led to better standards of living and improvements in health outcomes contributing to progress towards the Millennium Development Goals (MDGs). Infant and child mortality as well as stunting was lower than the national average, health system performance was above average and the state was ranked second best performing state in India. Despite these achievements, in the early 2000s, critical gaps remained. One in 14 children died before their 5th birthday, only about half of deliveries took place in institutions and this share was much lower in rural areas (38 percent), in the poorest districts (31 percent), among illiterate (32 percent) and women belonging to scheduled castes and tribes (34 percent). Disparities in health outcomes across different socio-economic groups and geographical areas had widened. There were also concerns regarding the performance of the extensive network of public health delivery systems with respect to effectiveness, access and targeting of the most vulnerable. These concerns were further compounded by insufficient and misallocated public financing. In 2001, to address these concerns, the State Government of Karnataka (GOK) created an independent task force and prepared a policy document Karnataka State Health Integrated Policy (KSHIP) 2004 that articulated its long-term vision for the health sector. In addition, the GOK committed to significantly increase resources allocated for the health sector under its medium-term fiscal plan 2006-2010. 2. Within this context, the World Bank’s support for the health sector in Karnataka through the Health System Development and Reform Project was well justified. The project was consistent with the strategic principles and priorities identified by the Country Assistance Strategy (CAS), September 2004: (a) focusing on outcomes as means to improve governance and service delivery; (b) being selective in support of activities that have greatest impact on the poor; and (c) envisaging a strengthened role of the Bank as a knowledge provider and generator. 3. By supporting innovations and providing sound technical assistance, the project was also consistent with the Government’s KSHIP 2004 which was aimed at further developing and sustaining its reform program. The Bank leveraged its extensive global experience and lessons learned from over two decades of engagement in the health sector of Karnataka through both national and state operations. Further, the project was designed to help the Government to better coordinate and maximize the impact of externally assisted projects and vertical national programs sponsored by Government of India (GOI). The operation built on concrete plans prepared by three core teams supported through a Japanese Policy and Human Resource Development (PHRD) grant to improve the performance of the public sector, enhance the involvement of 1 Dev and Ravi 2003; Hand Book of South Asian Economics edited by Anis Chowdhury, 2003 ISBN-1848441290 Page 7 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) private sector in the delivery of essential services and develop a health financing strategy respectively. More importantly, the project aimed to contribute to improving essential maternal and child health outcomes and reducing the burden of communicable diseases using a “results-based” approach as a key design feature. Project Development Objectives (PDOs) 4. The PDO for the original operation, defined in the Development Credit Agreement (DCA, pg. 22) is “to support the Program and assist Karnataka in improving the utilization of essential curative and public health services particularly in the underserved areas and amongst vulnerable groups”, where ‘program’ means the overall health sector program of Karnataka consistent with KSIHP. The PDO in the PAD, pg. 8 is worded differently as “to increase utilization of essential health services (curative, preventive and public health), particularly in underserved areas and among vulnerable groups, to accelerate achievement of the health-related MDGs”. While the PDO as stated in the DCA is framed in the context of the state government’s policy and vision 2020 for achieving health related MDGs, the PAD omits reference to these, explicitly referring to the achievement of health-related MDGs as the higher goal of the operation. Key Expected Outcomes and Outcome Indicators 5. Essential curative and public health services were broadly defined as those that would produce improvements in maternal and child health outcomes and reductions in communicable diseases and were prioritized by the health-related Millennium Development Goals (MDGs). Underserved areas were defined as locations where essential health services were largely not available such as rural areas in the poorer districts. The vulnerable groups were defined as those at higher risk of morbidity and mortality such as pregnant mothers and young children based on NFHS II (1998) and RCH survey (2002). Key indicators were: • PDO1: Percentage of safe deliveries (%) • PDO2: Percentage of institutional deliveries (%) in rural areas; SC/ST mothers • PDO3: Percentage of mothers and newborn children visited within 2 weeks of delivery by a trained community-level health provider (%) • PDO4: Percentage of children immunized (%) in poor districts/illiterate mothers • PDO5: Percentage of outpatient attendances (%) • PDO6: Percentage inpatient attendances • PDO7: Annual prevalence rate for malaria (%) • PDO8: Percentage number of women receiving information on HIV / AIDS during pre-natal/ post- natal or family planning visits (%) • PDO9: Percentage TB cure rates (%) 6. For a sub-set of key output indicators, described in (Table 11 of the PAD), quantitative targets were set by Government of Karnataka for the next 15 years in line with their Vision 2020 document. It was expected that KHSDRP would monitor the performance of these indicators disaggregated by geography and socio- economic groups, the data being generated from the planned NFHS and RCH surveys. Page 8 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) Components Component 1: Strengthening existing Government health programs towards the achievement of more effective and equitable delivery of services (total cost US$59.5 million). 7. Subcomponent 1A: Organizational Development. This subcomponent aimed to strengthen the stewardship role of the Government in the health sector and move towards a results-based approach. Specifically, this subcomponent was to support acquisition of new skills and knowledge through technical assistance and capacity building, and the adoption of ‘new ways of doing things’ and learning by doing. Achieving a change of culture (of doing things differently) was intended through a long period of sustained effort throughout the project and beyond. The District Health Officers and their staff in consultation with the local governments were to review their performance, identify bottlenecks that prevented better service delivery, and develop Service Improvement Plans to be supported through a competitive Service Improvement Challenge Fund described under Component 2. In addition, a series of capacity-building activities was envisaged to enhance the ability of the Government and the private sector to design and enforce accreditation processes for health facilities, and to work together under service agreements. 8. Subcomponent 1B: Improving Primary and Secondary Care Services’ Effectiveness. This subcomponent was to support a broad set of Government programs to deliver essential public health, primary and secondary care curative services using a “programmatic approach” with clearly defined milestones. The subcomponent was intended to: (a) contribute to increased Health and Family Welfare expenditure described in the Medium Term Fiscal Plan 2004-05 to 2007-08, focusing specifically on the peripheral health services; (b) help to strengthen its planning, execution and monitoring processes for public health and primary healthcare activities; and (c) contribute to development and implementation of a Government-led program to achieve the health-related MDGs. The programmatic funding was to be done in the following sequence: (a) identification of activities eligible for financing and of the specific budget lines (heads of account) in GOK’s budget that finance those activities; (b) agree on a financing mechanism or formula, linked to commitment of the government to increase expenditures on the eligible activities; (c) reporting of expenditure; (d) Bank reimbursement of the expenditure according to the achievement of milestones achieved; and (e) ex-post adjustment of the initial disbursements according to actual expenditures resulting from audited accounts. Component 2: Innovations in Service Delivery and Health Financing (total cost US$60.7 million) 9. Subcomponent 2A: Innovations in Service Delivery Linked to Need and Performance. Under this subcomponent, the project was to invest in primary care and in public health activities using an innovative approach. Two different funds were to be created: Service Improvement Challenge Fund, and Public Health Competitive Fund. The Service Improvement Challenge Fund was to finance selected initiatives to scale-up quantity and quality of curative, primary and secondary care services, including construction, upgrading and renovation of infrastructure, and innovative schemes of service delivery while the Public Health Competitive Fund (PHCF) was to support community proposals for public health activities aimed at producing Page 9 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) community-wide reductions in disease incidence. This was based on the premise that efforts at tackling community-based health issues are more likely to be effective if they are driven by a community initiative. 10. Subcomponent 2B: Innovations in Health Financing .Innovations in health financing was to contribute to reducing the financial barriers to accessing health services, thus leading in the long-run to an increase in the utilization of essential hospital services for poor families. One major experiment was planned under this subcomponent: the design and implementation of a health insurance pilot. This subcomponent was to support two major activities: (a) technical assistance for the design, implementation, monitoring and evaluation of an insurance pilot, and (b) financial support for providing additional premium subsidies (together with GOI and GOK) and additional benefits to Below Poverty Line (BPL) families who decide to enroll in the scheme. Component 3: Project Management, Monitoring and Evaluation (total cost US$13.6 million) 11. This component was to support the establishment and operations of the State Project Management Unit (SPMU), District Project Management Units (DPMUs), and other project- related Committees. The activities were to include: (a) consulting services to cover technical issues, as well as procurement and financial management; (b) hiring and training of project management staff; (c) provision of necessary office equipment; and (d) incremental operating costs. This component was also to finance data collection through baseline, mid-term, and end-of-project household and facility surveys and all other evaluation activities. Finally, as part of monitoring activities, this component was to support the establishment of a State Health Informatics Center and a new integrated Health Management Information Systems (HMIS). B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets 12. As per the Development Credit Agreement (DCA), the original PDO was “to support the Program and to assist Karnataka in improving the utilization of essential curative and public health services particularly in the underserved areas and amongst vulnerable groups”. “Program” was defined as "the overall health sector program of Karnataka consistent with KSIHP”. 13. The PDO for the Additional Financing (AF), “to improve health service delivery, public-private collaboration and financing, particularly for the benefit of underserved and vulnerable groups in Karnataka”, was the same in both the Financing Agreement (FA) and the Project Paper except for the comma after public-private collaboration in the Project Paper. The lack of separation of public-private collaboration and financing with a comma is not seen as substantive as the AF aspired for improvements in both as separate outcomes. Revised PDO Indicators 14. There were three revisions in Key Outcome Indicators during project implementation as a result of the restructurings of December 2010, December 2012 and March 2015 respectively. Each revision in outcome indicators responded to the changed environment in which the project operated and brought specificity in the results chain for achieving the stated development objective. Page 10 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) 15. The original project had a set of 9 PDO indicators that tracked utilization of essential preventive, curative and public health services. In December 2010, three indicators pertaining to control of communicable diseases (malaria, HIV and TB) were dropped from the results framework considering the significant outcomes achieved by the state because of its well-performing dedicated vertical disease control programs. The remaining six indicators were sharpened to specifically track utilization amongst the vulnerable and poor populations from underserved areas. 16. At AF, five of the six revised PDO indicators (percentage of births/deliveries occurring in a health facility; percentage of births/deliveries occurring in a facility among the poor; percentage of population receiving outpatient care in government facilities in seven less-developed districts; and number of claims paid by the health insurance pilot program benefitting the Below Poverty Line households) were retained. An additional two PDO indicators were introduced to capture outputs from the proposed Non Communicable Diseases (NCD) pilots (number of women screened for cervical cancer) and strengthening of the emergency transportation service in the state (percentage change in the number of road traffic accident patients transported by 108- ambulance system who receive emergency care at government health facilities in pilot districts). 17. In March 2015, targets for two PDO indicators were modified—the end-line target for number of claims paid by health insurance pilot program was enhanced by over 250% from 45,000 claims to 120,000; while the throughput for the cervical cancer screening program in pilot districts was drastically reduced from 60,000 to 1,000. The latter was due to rationalized expectations from the severely delayed operationalization of the NCD screening and treatment pilot and its truncated geographic scope. Additionally, the scope of the indicator tracking uptake of emergency transportation in the state and access to emergency care in government health facilities in pilot districts was enhanced to also capture access to emergency care in private sector health facilities. Revised Components 18. Each of the restructuring of December 2010, December 2012 and March 2015 resulted in changes in the scope of the project components, adding and dropping activities to respond to the dynamic environment in which the project operated. The project components in their revised scope continued to support the revised development objective of improving health service delivery, public private collaboration and financing for benefit of underserved and vulnerable groups in Karnataka. The component-wise costs were revised with the restructuring of December 2010 and the AF of 2012 (Table 1). Table 1 Project restructuring and component-wise reallocations IDA Financing in US$ million Components Original Phase Restructuring Additional Financing Restructuring December 2010 December 2012 March 2015 Component 1 59.6 69.7 75.3 75.30 Component 2 60.7 57.0 115.9 115.80 Component 3 13.6 15.1 20.6 20.70 Unallocated 7.9 0.0 0.0 0.0 Total 141.8 141.8 211.8 211.8 Page 11 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) Component 1: Strengthening Existing Government Health Programs (revised cost US$ 75.3 million) 19. Subcomponent 1A (Organizational Development) was revised during the restructuring of December 2010 to drop (i) training and technical assistance for stakeholders in the private sector to allow for a stronger focus on strengthening project capacity for management of PPPs, and (ii) development of an accreditation system for private sector health service providers since a national accreditation structure had been established. The scale of two ongoing activities, namely (a) management training of local level health administrators and managers of Primary Health Centers (PHCs); and (b) implementation of strategy to improve quality assurance of PHCs was enhanced and evaluations built in. Additionally, the component now also supported accreditation of select district hospitals in the state. With Additional Financing, the component emphasized both administrative and clinical capacity strengthening of PHC officers. The quality assurance program at the PHC level was scaled-up to the state with use of tools like checklists and institutionalized assessments. Additional district hospitals were identified for accreditation and quality standards were developed for community health centers and taluk hospitals. In line with the enhanced scope of the project, additional administrative and technical staff, especially for the NCD and road safety program were supported. 20. Subcomponent 1B (Improving Primary and Secondary Care Services’ Effectiveness) was dropped during the additional financing phase of the project. Component 2. Innovations in Service Delivery and Health Financing (revised cost US$ 115.9 million) 21. Subcomponent 2A (Innovations in service delivery) was revised during the restructuring of 2010 to upgrade approximately 50 PHCs based on a needs assessment. These numbers were further enhanced based on need to service underserved ‘backward’ blocks, especially with refence to primary and maternal health care services during the AF. Additional capital investments for district planning offices, training facilities, pharmaceutical warehouses, liquid waste management systems in district hospitals, pharmaceutical quality control systems and food testing and regulation systems were proposed. The financing of PPPs for operationalization of PHCs was dropped since the investments were being made by NRHM. The PHCF interventions were restricted to 40 NGO contracts and results based financing was incorporated in the NGO contracts at the AF. Capacity building of VHSCs for enhanced community level public health knowledge and interventions was also included. In March 2015, due to ongoing and unresolved legal issues beyond the control of the project, procurement processes for NGO contracts for citizen help desks and mobile clinics were dropped. 22. Two important additions under the component in the AF were (i) implementation of two pilot models-- National Program Plus and Project Model in two districts each for community sensitization (including at worksites and schools), mobilization and screening at the Sub-Center and PHC levels respectively of selected NCDs and their risk factors, with treatment of cervical cancer provided upstream from the CHC level. Evaluation of these two models and the model adopted by the Government of India’s National Program for Prevention and Control of Cancer, Diabetes, Cardiovascular Diseases and Stroke were also planned; and (ii) a situation and needs analysis of two safety demonstration corridors that were proposed under the World Bank financed Karnataka State Highways Improvement Project 2 and appropriate investments in both hardware and software in support of ambulance services and emergency health care facilities in the proposed corridors. Investments in an IEC strategy towards road safety was also proposed. Additionally, investments in blood Page 12 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) storage and management system of the state in support of trauma care at block and district hospitals were also planned. 23. In March 2015, the strategy being supported by the project for prevention, screening and treatment of select NCDs and their risk factors was modified to pursue a single model in two districts. 24. Subcomponent 2B (Innovations in health financing): With the restructuring of 2010, the analysis of national insurance schemes to inform interventions under this component was dropped as the analysis was being financed by other Bank operations. The AF supported institutional development of the state government’s initiative to scale up-the Vajpayee Arogyashree Scheme (VAS) that provided insurance cover for the Below Poverty Line (BPL) households for accessing tertiary healthcare. Specifically, the AF financed claims for hospital services based on achievement of institutional development milestones such as strengthening the verification and follow-up systems, undertaking a cost analysis of benefit packages and prices, and establishment of a grievance redressal system, amongst others. In March 2015, over US$ 15 million were reallocated from sub- component 2A to this sub-component with the objective of enhancing the scope of VAS to finance hospital care for road accident victims, with disbursements being mapped to milestones aimed at institutional strengthening of the VAS. 25. The AF also envisaged results-based (RBF) co-financing through the Health Results Innovation Trust Fund, to PHCs and First Referral Units for improvements in quality and coverage for maternal and child health, disease control, NCD and other specified services. In March 2015, due to lack of demonstrated progress, the deployment of RBF strategies for improved primary and preventive health services was dropped. Component 3. Project Management, Monitoring and Evaluation (revised cost US$ 20.6 million) 26. The restructuring of December 2010 proposed financing a health facility survey to assess impact of investments in health sector under the component. The component would also support impact evaluations of innovations including the health insurance pilot program and the conditional transfer scheme for maternal health under NRHM. Planned support to the State Health Informatics Center was dropped as the state received financing from other sources. At the AF stage, use of next generation Information Communication Technology (ICT) in the health sector including a pilot beneficiary tracking system supported by the Bank’s Governance Partnership Facility Trust Fund and its evaluation were proposed. However, in March 2015, the activity was dropped as its implementation was entrusted to another department within the Government of Karnataka. Other significant changes (in design, scope and scale, implementation arrangements and schedule, and funding allocations) 27. The project was restructured seven times between December 2010 and March 2016. Specifically, a. December 2, 2010: Level II restructuring for (i) revision of project key indicators and targets in order to incorporate the International Development Association (IDA) core indicators as well as better reflect the Project Development Objective (PDO); (ii) reallocation of the credit to reflect decisions taken at the Mid-Term Review to scale-up certain activities and reduce the planned scale of others; and (iii) specification of implementation and disbursement modalities of the project’s support to a pilot health insurance program. Page 13 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) b. February 25, 2012: Level II restructuring for extension of the project closing date from March 31, 2012 to September 30, 2012. This was done to allow time for the appraisal Government’s request for Additional Financing with an extension of project closing date by three years. This was the first extension of the project. c. July 31, 2012: Level II restructuring for extension of the project closing date from September 30, 2012 to March 31, 2013 to prepare the Additional Financing and proposed three-year extension of the project. This was the second extension of the project by a cumulative of 12 months. d. September 27, 2012: Level 1 restructuring (i) effecting the Additional Financing of US$ 70 million for the project; (ii) revision in project scope (as discussed in Section IB) and consequently the project development objective; (iii) changes to the results framework (as discussed in Section IB); and (iv) extension of the closing date of the original Credit to co-terminate with the closing date of the new Credit, i.e. March 31, 2016. e. March 19, 2015: Level II restructuring for (i) expanding scope of sub-components towards achievement of the PDO; (ii) reallocation of US$ 15 million from Category 1 to Category 2 of the project to support continued implementation and expansion of the hospital care financing program for the poor under Subcomponent 2B (discussed in Section 1B). The results framework was revised to reflect the changes. f. May 21, 2015: Level II restructuring for (i) extension of timeline for submission of annual audit report from 6 to 9 months, based on request from Government of India for several projects in the portfolio. g. March 17, 2016: Level II restructuring for (i) extension of the closing date of project to March 2017; and (ii) changes to the results indicators. Page 14 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) Other Changes Rationale for Changes and Their Implication on the Original Theory of Change 28. Since the project was originally prepared, the overall context had changed dramatically: a. Utilization of essential health services had significantly improved as evidenced by the performance of the original project’s outcome indicators. The proportion of births delivered in a health facility has risen from 65% in 2005-06 to 86% in 2009 (with an end-project target of 90%); the proportion of children fully immunized has increased from 55% in 2005-06 to 78% in 2009 (against a target of 80%). At the same time, despite some narrowing, the socio-economic and geographic disparities remained between (i) the better-off southern and poorer northern partners of the state and (ii) urban and rural areas. The available additional resources from GOI’s NRHM and GOK were earmarked for activities that didn’t directly address this disparity. b. Government health expenditure had significantly increased. Because of both increased GoK allocations and the rapid growth of NRHM, health expenditure in the state more than doubled between 2004-05 (US$ 232.4 million to US$ 587.9 million in 2010-11) and per capita public expenditure on health grew from US$ 4.20 to US$ 9.67. c. The burden of NCDs and road traffic injuries was growing rapidly. According to a household survey (DLHS 3), 12.2% of Karnataka’s urban and 8.5% of its rural population had a high blood sugar of >140 mg/dl indicating pre-diabetes; and 23.5 % of the population above 18 years of age suffered from hypertension (systolic >140mmHg and diastolic >90mmHg) and although specific incidence data for Karnataka was not available, in 2008, cervical cancer was the most common cancer among Indian women above 15 years of age (Krishnan et al. 2013). In fact, by 2012, Cardiovascular diseases had replaced diarrhea and lower respiratory infections as the number one cause of disease burden in the state (see Figure 1, Annex 4). Karnataka also bore a disproportionate burden of road accidents, over 10,000 fatalities in 2010 with the exponential rise in number of vehicles from 4 million in 2000 to 10.5 million in 2011. The injuries were several-fold higher than this number. Over 75% of the fatalities and morbidities were suffered by the 15-44 age group, causing immense human and economic loss. d. Catastrophic health expenditures contributed to households remaining or falling into poverty. About a fifth of Karnataka’s citizens were estimated to incur catastrophic health expenditure and the share of out of pocket expenditure out of total household expenditure was about 14.4%. The GoK had piloted a poverty-targeted health insurance scheme in two districts with support from the original project and was ready to scale it up across the state. e. There was a growing realization that leveraging the private sector through different forms of collaboration could allow the GoK to further improve health outcomes in the state. Page 15 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) 29. In addition to scaling-up of specific organizational development and institutional capacity building activities (Component 1) and investments in primary care maternal health services with clear focus on underserved blocks (component 2), the US$ 70 million AF specifically focused on more ambitious initiatives that included supporting (i) activities that could not be easily done through other mechanisms such as quality assurance and accreditation, (ii) activities that would benefit from the technical engagement with the Bank by setting specific milestones for institutional development, verification, transparency and grievance redress in the health insurance scheme, and (iii) introduced pilot programs for NCDs and Road Safety, which if successful could be scaled-up using Government funds. The AF of was thus designed to build on the success of the original project while adjusting to the changing landscape in the health sector and burden of disease pattern to allow the project to further contribute to improved delivery and utilization of health services by vulnerable and underserved groups. The original theory of change remained relevant as the overall program of the state health department and NRHM was still focused on increasing coverage and utilization of essential services. The AF subsumed and enhanced it by providing a stronger focus on the effectiveness of services utilized by vulnerable and underserved groups, improving health financing towards better financial protection to the poor as well as piloting prevention and treatment of NCDs and road traffic injuries, all supported by improved public-private collaboration. THEORY OF CHANGE CONSEQUENT TO ADDITIONAL FINANCING Original Project Additional Financing Improved financial protection Page 16 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) II. OUTCOME 30. Both scope and ambition of the project expanded significantly after the AF and, moreover, the original PDO of the project was very broad which was better clarified during the AF and restructuring. The PDO changed with the AF and the scope and results framework were adjusted multiple times. However, the original theory of change was retained and the AF enhanced it by graduating Bank’s support from a contribution to improved “utilization of services” to strengthening the health system to more effectively “deliver” essential health services. At the end of the original project phase, utilization of essential public health services increased significantly (i.e. 86% of deliveries in a health facility and notable increases in use of communicable disease preventive and control services), therefore Bank support was re-focused to ensure that those services already being utilized by the poor were of sufficient quality to improve their health status which resulted in adding improved service delivery to the PDO. Further, financing and public-private collaboration were added to respond to the changing environment and rapid private sector growth. 31. Despite several restructurings, the project clearly retained its focus on “underserved and vulnerable groups” and notwithstanding the change from utilization to delivery, most of the PDO indicators still focused on the use and equity. As changes PDO, components and results clearly reflect a better path and more focused approach to achieve the same expected outcomes a split rating was not done and the project was assessed on revised outcomes and revised targets as per the Bank’s guidance2. A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating High 32. The relevance of the PDOs is high it was well aligned with the Country Assistance Strategy (CAS) 2004-2008 and both the original and revised PDOs remained relevant to the subsequent Country Assistance and Partnership Strategies. The original PDO was highly relevant to the strategic principles and priorities identified by the CAS 2004-2008: (a) focusing on outcomes as means to improve governance and service delivery; (b) being selective in support of activities that have greatest impact on the poor; and (c) envisaging a strengthened role of the Bank as a knowledge provider and generator. The PDOs remained relevant throughout implementation. The revised PDO contributed to the Bank’s 2009-12 Country Strategy by increasing the effectiveness of service delivery in the health sector, notably through strengthening institutional mechanisms, capacity, and information systems to further improve the focus on results, as well as enhance engagement with the private sector. It also contributed to the CPS 2013-2017 focused on “inclusion” which aimed to promote human development and strengthen social programs, so that economic integration and spatial transformation generated inclusive growth. In relation to health, the CPS specifically envisaged better accountability arrangements in service delivery, adequate regulation and oversight of private healthcare providers, and expanded coverage of health insurance among disadvantaged groups 2 Bank Guidance: Implementation Completion and Results Reporting (ICR) for Investment Project Financing (IPF) operations – July 5, 2017 Page 17 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) 33. Furthermore, the PDOs were directly relevant to the Karnataka State Integrated Health Policy 2004 that articulated state’s long-term vision for the health sector to provide quality healthcare with equity, which is responsive to the needs to the people, and guided by principles of transparency, accountability and community participation. The Medium Term Fiscal Plan of the state also committed to significantly increase resources allocated for the health sector, which was used as a milestone for project disbursements for eligible expenditure categories. 34. With a clear focus on improving service delivery and financial protection for underserved and vulnerable groups in the State, the PDO is well aligned with Bank’s twin goals of elimination of extreme poverty and promotion of shared prosperity. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 35. While some of the project financing, especially under component 1, supported the broader program of the Government, specific areas of support provided by the project played a catalytic role in shifting state government’s policy towards improving delivery of essential health services in the lagging districts; strengthening institutional mechanisms, efficiency and transparency of health insurance agency; and piloting innovations in prevention and control of NCDs and promote road safety as well as promoting health financing with a clear focus on reducing the financial burden on poor and promoting partnership with the private sector. The Bank also supported Impact Evaluation of the health insurance program through its technical assistance that informed state as well as national policies. 36. Improving health service delivery for the underserved and vulnerable groups (includes the original focus of increased utilization): Page 18 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) a. 52% of the direct beneficiaries were women, one of the vulnerable groups targeted in the PDO. b. The percentage of deliveries taking place in a health facility went from 65% to 94% among the general population, against a target of 90%, and from 37% to 77% amongst the poor against a target of 55%. The poor were the main underserved group targeted in the PDO. c. 392 facilities were constructed, renovated and/or equipped against a target of 375. Most of these facilities were in the 7 underserved districts of northern Karnataka targeted in the PDO. d. A total of 91,415, health personnel received training (against a target of 25,500 with specific focus on organizational development, quality assurance and healthcare waste management. e. The project helped strengthening of essential public health services such as promotion of food safety and successfully concluded pilots on NCDs and road safety, which informed government’s strategies to address their increasing burden. f. About 27,156 women were screened for cervical cancer in the two pilot districts against a target of 1,000. The target was 60,000 at the time of the AF but was later revised based on a better understanding of prevalence of cervical lesions and since the pilots only were ran for a year instead of four (see implementation issues for more detail). g. 60% of the eligible population or 319,012 people were screened for diabetes in the two pilot districts and 91% of those diagnosed were put on treatment (IMS Health evaluation report), no target was set for this. h. 77% of the eligible population or 409,055 people were screened for hypertension in the two pilot districts and 95% of those diagnosed were put on treatment (IMS Health), no target was set for this. i. The number of road traffic accident victims transported by the state ambulance system ran by a private operator that was supported by the project increased by 183% against a target of 10% in the road transport accident pilot districts. j. Training materials for NCD prevention and management, and guidelines and standards for trauma care services developed under the project will continue to help the state to more effectively address the emerging burden of NCD and road traffic accidents. After the project closure, the transport department created a new cell for road safety coopting the Department of Health staff, who worked on this subcomponent. 37. Public Private Collaboration and financing for the benefit for underserved and vulnerable groups a. Health financing. k. A total of 153,237 claims were paid by the health insurance pilot program benefitting Below Poverty Line households. The achievement was three-fold higher than the original target of 45,000. l. By linking disbursements to milestones focusing on institutional development, the project facilitated stronger systems for monitoring and verification of medical claims, development and implementation of a communication strategy to sensitize the poor, undertaking cost analysis of insurance packages and making IT and grievance redress systems more effective. b. Public-private collaboration. Page 19 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) m. The number of claims paid to private sector hospitals by the government health insurance program supported by the project was 123, 462 against a target of 120,000. n. The number of private hospitals empaneled by SAST that submitted at least 100 claims was 256 against a target of 50. o. The number of public and private facilities empaneled to provide care to road accident victims under the government programs was 766 against a target of 150. Justification of Overall Efficacy Rating 38. The rating is substantial because the project achieved both its original and revised Development Objectives. All end-of-project outcome targets were achieved or substantially surpassed. C. EFFICIENCY Assessment of Efficiency and Rating Substantial 39. The efficiency analysis is presented in annex 4 and assesses the project’s technical and allocative efficiency. Furthermore, this section reviews the efficiency implications of project extensions as well as some of the issues presented in the Key Implementation Issues section of this document. 40. Technical Efficiency: A large portion of the project funded investments in the renovation and construction of primary health care facilities. A total of 364 primary healthcare facilities were renovated or constructed under the project. This allowed facilities to expand their patient volume. While less than half (48%) of primary health centers conducted at least 10 deliveries per month in 2007-08, the share rose to 62% by 2012-13. The focus on facilities that had at least 10 deliveries per month is notable, as volume has been found to be positively associated with quality of maternal and obstetric services (Kruk et al., 2016). Furthermore, outpatient utilization increased substantially at government facilities in the seven less developed districts, rising to 78% in 2016 as compared to 46% in 2006. The project also supported training of 91,415 health personnel the development of a quality assurance function in the state government as well as accreditation of public hospitals. The quality improvements resulting from higher patient volumes, better trained health-workers and systematic quality assurance are likely to have resulted in cost savings by reducing complications and eliminating waste. 41. The Lives Saved Tool was used to estimate the cost-effectiveness of Component 2A Innovations in service delivery ($85 million). Conservatively, we only accounted for changes in institutional delivery, attributing those directly to the project. Gains were calculated for each year of project implementation as compared to the baseline situation of 2006. Compared to the initial rate of 65%, increases in institutional delivery during the project resulted in averting 2763 maternal deaths. Taking the average age at death and female life expectancy, this can be converted into 121,572 life years. Since the total cost of Component 2 was $115.9, this resulted in a cost-effectiveness ratio of $699/ life year saved. Using the threshold of Karnataka’s GDP per capita of approximately $3200, this component was highly cost-effective even under the conservative estimates. Furthermore, the project supported cervical cancer screening which can significantly reduce mortality as early Page 20 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) detection and treatment has been identified as a highly cost-effective intervention in India, with a cost- effectiveness of $10 per life saved (Goldie et al., 2005). During the project, 27, 156 women were screened for cervical cancer. 42. Allocative Efficiency: An explicit goal of the project was to increase utilization of health services for the poor and vulnerable populations. As noted in the PAD, at the outset of the project, significant differences in utilization existed between urban and rural areas and the better-off southern and poorer northern districts in the State. The project reduced inequities by focusing on the less developed districts: Bagalkot, Bidar, Bijapur, Chamarajanagar, Gulbarga, Koppal, and Raichur. The construction of primary health care centers in districts where there was a shortfall reduced regional disparities in access to and utilization of health services. As evidenced from household surveys, in 2007-08 the institutional delivery rate ranged from 25% in Koppal to 95% in Udipi district (Figure 2). By 2012-13, this gap narrowed to 71% and 99%, respectively. Despite free care at public facilities, a significant share of the poor and vulnerable populations sought care from private facilities before the implementation of the project due to poor quality and lack of access to public facilities resulting in large out-of-pocket expenditures. During the project period, the share of deliveries at public facilities nearly doubled from 35% to 61%, with over two thirds of deliveries in rural areas occurring at public facilities supported by the project indicating improved efficiency of public service delivery. Figure 1: Institutional delivery (% of pregnant women), 2007 and 2013 by districts in Karnataka 43. Furthermore, expansion of insurance for the poor significantly reduced out-of-pocket payments and resulted in better health outcomes. The project supported the government insurance program (Vajpayee Arogyashree scheme) that provided free tertiary care to households below the poverty line (BPL) in Karnataka. Results from an independent evaluation suggest lower mortality rates among households below the poverty line covered under the scheme, as well as substantial reductions in out-of-pocket payments with OOP at tertiary level facilities (64% lower) for BPL households covered under the scheme than those just above the threshold (Sood et al., 2014). According National Family Health Survey (NFHS) data, coverage of government sponsored health Page 21 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) insurance schemes at household level nearly tripled (from 10% to 28%) by 2015-16 compared to 2005-06, and this trend was more predominant in rural areas - the main target areas for the project – as compared to urban areas (32% and 23%, respectively). 44. By introducing systematic screening and treatment of NCDs at primary and secondary levels, the project helped the government of Karnataka to start redirecting human and financial resources to some of the biggest causes of morbidity and mortality in the state. For example, by 2012, cardiovascular diseases replaced diarrhea and respiratory infections as the number one cause of loss in DALY. In the same year, India had the highest age-standardized rate of cervical cancer in South Asia (22 per 100,000) (Error! Reference source not found.). Almost 1.2% of deaths among women in Karnataka were due to cervical cancer in 2012 (IHME, 2017). 45. By strengthening core public health functions such as promoting food safety across the state and introducing road safety in selected corridors, the project also helped the state to address private market failures and develop institutional capacities to promote public health. 46. Implementation Efficiency: The Key Implementation Issues section describes the challenges faced throughout the project with high turnover of officials, delays in procurement and others. These issues caused delays that affected the efficiency with which the project was implemented. The main reason for the last extension of the Closing Date, from March 2016 to March 2017 was due to the delays in technical work and procurement necessary to pilot systematic NCD screening and treatment at the primary and secondary levels in two districts. However, these delays were offset by the results delivered once activities were implemented. For example, the last extension allowed for the screening of more than 27,000 women for cervical cancer and over 70% of the population eligible for diabetes and hypertension in the two pilot districts. With the exemption of women screened for cervical cancer, all outcome targets were achieved within time and budget ad while the cervical screening target required a year extension, it was achieved without incurring cost overruns. D. JUSTIFICATION OF OVERALL OUTCOME RATING Satisfactory 47. The PDOs of the project remained highly relevant throughout the 11-year implementation period. All outcome targets were achieved and in some cases substantially surpassed and the project funded cost effective interventions and contributed to redirecting resources to the right things and the neediest people. There is evidence that the project contributed to the impacts identified in the theory of changed, including a 25% reduction in the Maternal Mortality Rate and a 35% reduction in the Infant Mortality Rate as well as improved financial protection for the poor. The project also helped Karnataka to be identified as a pioneer in the areas of health insurance, NCDs and road traffic emergency services. As evidence of this, the World Bank remains engaged in the state as the SAST has been identified as a national and global knowledge hub for health insurance as part of the Lighthouse India project started by the India CMU in 2017. Page 22 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 48. The project had a strong focus on women. Four PDO indicators in the PAD measured access and utilization of preventive and essential public health services by women. All four indicators recorded an improvement in performance over baseline. Specifically, (i) % of safe deliveries (deliveries attended by skilled personnel or taking place in a health facility) increased from 66% in 2002-04 to 73% in 2007-08; (ii) institutional deliveries increased from 58% in 2002-04 to 65% in 2007-08, almost reaching the end line target of 66%; (iii) % of mothers/newborns visited by trained provider within 2 weeks of delivery increased from 58% in 2005-06 to 69% in 2007-08; and (iv) % of pregnant women receiving HIV/AIDS information increased from 12% in 2006- 07 to 59% in 2009-10. The maternal and child health indicators were impacted by launch of benefit schemes like Janani Suraksha Yojana, launched by NRHM in 2005 and state specific schemes like Prasuti Arakey and Madilu launched in 2008. The latter two schemes targeted pregnant women from BPL households and those from vulnerable groups, and provided conditional cash through bearers cheque for second trimester antenatal check-up and institutional delivery and mother and child kits for them. In year 2009-10, about 2.5 lakh Madilu Kits were distributed and 2.4 lakh women benefited from the Araike scheme out of 8.81 lakh deliveries that took place in the State. These numbers suggest that all women eligible to receive these benefits were covered. 49. The AF introduced two indicators namely, percent of births/deliveries occurring in a health facility among the poor and number of women screened for cervical cancer in pilot districts to measure women’s access to health services and progress of the NCD prevention and control pilot. The percentage of birth/deliveries in a health facility among the poor increased from 37% in 2004 to 77% in 2016, 22 percentage points over the end line target. The number of women screened for cancer cervix was 27,156, almost half of the original target of 60,000 but significantly more than the downwardly revised target of 1000. The percentage of births/deliveries in a health facility increased from 65% in 2006 to 94% in 2016, again 4 percentage points over the end line target. Finally, of all beneficiaries of public health service delivery through the project in the state, 52% were female. The data, from household, national surveys and project HMIS clearly demonstrating that women were disproportionately larger of the beneficiaries of the project investments. Institutional Strengthening 50. The milestone-based approach which has broken down key institutional reforms into monitorable actions/steps and linked disbursements to their achievement helped to strengthen the existing institution, SAST, to more effectively deliver Universal Health Coverage in the state focusing on improved verification of claims, better information systems and grievance redress mechanism. 51. The project also helped to build better organizational capacity at the district and state levels and strengthened quality assurance through the implementation of quality checklists, training and periodic assessments. Two district hospitals received prestigious accreditation by the National Board of Hospitals and Healthcare Providers (NABH), while other district hospitals and community health centers included under the quality assurance program achieved the Kayakalp award and reached progressive level of national quality assurance standards (NQAS) developed by National Health System Resource Centre (NHSRC). The additional ICT interventions planned under the project were subsequently dropped as the state was developing a Page 23 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) comprehensive ICT program covering all departments. However, the project supported successful networking of all blood banks (100%) and 50% of blood storage centers. 52. One noteworthy contribution was the technical assistance provided to the State in strengthening the Karnataka Drugs Logistics and Warehousing Society (KDLWS). The project initially supported the functioning of KDLWS by providing HR support and through involving them in all the procurement activities of the project, the necessary capacity building and cross learning was ensured. By the end of the project, KDLWS was recognized as a reliable institution for procurement and supply chain management, with semi-autonomous status and financial sustainability. 53. The project also used the government system for implementation of civil works. The engineering division of DOHFW has executed all the works, and introduced the concept of IT based project management system. By end of the project, the DOHFW brought all its sub-divisional offices and civil works under this system which facilitated more effective monitoring. 54. Special cells for PPPs, NCDs and road safety created under the project helped to build institutional capacity in the DOHFW in preparing relevant tools and standardized protocols. At the end of the project the road safety cell was moved to transport department, and the project staff working on PPPs joined the DOHFW. 55. The project has institutionalized the training of medical staff on Basic Trauma Life Support (BTLS) and Advance Trauma Life Support (ATLS) by developing the local capacity of medical colleges. The system developed by the project is being used beyond the project and expected to be sustained by the state. The high-quality training modules developed by the project for BTLS and ATLS have been replicated and used by other training institutions in the State. 56. Contracting support for Village Health and Sanitation Committees (VHSCs), mobile health clinics, citizen’s help desks, laboratory services and specialist staff helped to build capacity of DOHFW in more effectively handling public-private partnerships. The performance monitoring system of contracting-in of specialists have been replicated in the DOHFW to monitor all the contracts funded under NHM. 57. The project also supported institutional development for promoting food safety. The project supported development of three Food Laboratories with state-of-art equipment to test food and beverages, the detailed organizational structure and operational manual. By the end of the project, registration and licensing of about 60% of eligible food business operators was completed, and an online registration and complaints system are both up and running. All Food Safety activities have been mainstreamed into the DOHFW, under the authority of the Food Safety Commissioner, and sufficient funds have been allocated to manage and sustain the functioning. 58. The project also supported the State in setting high standards for environment Safeguards. The project was instrumental in achieving the status of “mercury-free” medical equipment in public health facilities, the state- wide implementation of low cost bio-medical liquid disinfection and management initiative, and for sustainability of these activities. A State policy was developed to give priority for environment safeguards in handling bio-medical waste management and specification for medical equipment. Page 24 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) Mobilizing Private Sector Financing 59. The project did not have private sector mobilization as an objective. However, it provided both technical and financial support for setting-up and subsequently expanding the state health insurance schemes. There is evidence from the United States (Finkelstein, 2007) that expansions in government sponsored health insurance crowd-in investments in private hospitals as newly covered patients represent an additional revenue stream. 60. The project also supported public-private partnerships for mobile clinics and ambulance services to improve access to essential health services for the underserved communities and to enhance access to emergency services. The ambulance service now provides statewide coverage effectively using call centers and reduced inefficiencies in operating ambulance services by the public sector. It is however difficult to quantify the magnitude of specific private sector financing. More importantly, the project helped the Government to create enabling policies for private sector engagement and introduce results based contracting and transparent provider payment mechanisms. Poverty Reduction and Shared Prosperity. 61. The project investments in addressing critical gaps in health infrastructure and quality of service delivery helped in poverty reduction and promotion of shared prosperity. A steep increase in skilled care at child birth (from 65% to 94.3%) during 2005-06 to 2015-16 was achieved mainly through reduction of differences between urban and rural areas as well as increased use in underserved districts where the project supported addressing critical gaps including use of mobile clinics contracted through public-private partnerships. 62. The project supported institutional development for providing catastrophic health insurance coverage for the poor using a milestone-based approach. Over 150,000 claims paid by the health insurance program (VAS) specifically benefitted poor families by providing a package of tertiary healthcare services. The impact evaluation of VAS has shown that the catastrophic insurance cover contributed to reduction of out-of-pocket expenditure for the ailments covered under tertiary healthcare (as detailed in the efficiency section above). Other Unintended Outcomes and Impacts 63. There were no unintended outcomes and impacts, either positive or negative, evidenced during project implementation. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 64. Project Design: The overall project design was innovative and relevant at the stage of preparation. However, the preparation period was very long, almost 2 years partly due the India Detailed Implementation Review (DIR) which affected the Bank’s engagement in the health sector and required re-negotiation of the project including introduction of some changes in programmatic component on using country systems for procurement. However, institutional development of state drugs and warehouse society was retained. By the time the project became effective, several developments related to National Rural Health Mission (NRHM) had significant bearing on the initial approach. As several activities planned under the original project could be financed by NRHM, the Page 25 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) focus of the project gradually shifted from financing a package of services to targeting key systems-level strengthening activities and reforms. These changes in context resulted in several restructurings during the first three-four years as detailed in the “other significant changes” section. 65. The project introduced a concept of results-based financing for the disbursement of some components such as Service Improvement Challenge Fund (SICF) and Public Health Competitive Fund (PHCF). However, these activities had been superseded by the district planning process, community level initiative and other activities of the NRHM that supported service delivery up to the last mile and hence were dropped during the project restructuring in 2010. The activities under the Innovative health financing sub-component were not funded by NRHM and thus were retained and demonstrated the results-based financing approach throughout the original project and AF. Overall, SDR 21.9 million was disbursed against achievement of milestones. 66. Risk assessment: Critical risks and possible controversial aspects including mitigation measures were well- presented. However, the project covered a wide range of reforms from -improvements in performance of public sector to re-direction of government resources towards the poor- and the concepts of results-based financing and public private partnership in health were innovative. Therefore, while the objective was relevant, the overall risk rating could have been “Substantial” rather than “Moderate”. 67. Implementation arrangements: The project was prepared under the assumption that implementation would rely on the existing institutional structures of the Department of Health and Family Welfare as well as the district administration. However, the activities supported by the project, especially after the restructuring of 2010 were innovative and departed from the conventional implementation of the government program. Moreover, 90% of the time of the state and district level staff were already committed to deliver the regular government program which left them with very limited time to focus on implementation of the project. B. KEY FACTORS DURING IMPLEMENTATION 68. Frequent management changes: The DoHFW leadership took an active role in the implementation. However, there were frequent changes in the project administration. During the first 3 years, the leadership at State and project level had major shifts that has slowed down implementation. Similarly, during 2012- 2014, the project again had high turnover of Project Administrators with average tenure of less than 12 months. This resulted in delayed decision making affecting some of the new activities introduced by the AF and ultimately requiring an extension of the Closing Date to complete them. The frequent changes in leadership had a direct correlation with the implementation rating. During the period of frequent changes, the project was downgraded (2008 and 2014) and once there was stable leadership, the ratings were upgraded reaching Satisfactory again in 2010 and 2016 respectively. The project also suffered implementation delays due to the state election periods especially the new initiatives under the AF, such as pilots on road safety and non-communicable disease (NCD) prevention and control. 69. Counterpart Funding: The delays caused by unstable leadership also affected the availability of counterpart funding. The slow implementation and consequent low disbursement was interpreted by the State Treasury as low absorption capacity leading to insufficient budget allocations at the start of the Fiscal Year. Eventually the Page 26 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) allocations were increased mid-year through supplementary budget but by then it was hard for the project to catch up and make up for the lost time. 70. Integrated Implementation: Despite the challenge of state and district officials not having enough time for the project specific activities as mentioned above, enhanced coordination and integration of project activities into the regular state programs is expected to ensure better sustainability. Therefore, the implementation arrangements presented a trade-off between efficiency and sustainability and it is not clear whether an alternative approach relying on a parallel project implementation structure would have been better in the long term. 71. Getting incentives right: During the project implementation, there were some delays in reform initiatives as some managers and staff considered the project supported innovations as an additional burden. The exception was SAST, the agency implementing the health insurance scheme for the poor (VAS) where funding from the project was directed to its core program resulting in full ownership and commitment from its management and staff. 72. Enhanced public funding for health: Recurrent and investment financing for the health sector by both the GOK and Central Government were steeply enhanced during the project period. The annual per capita government spending more than doubled from Rs.190 (US$ 4.20) in 2004-05 to around Rs.440 (US$ 9.67) in 2010-11. This led to changes in project design to selectively focus on complementary activities where Bank’s technical engagement would be more beneficial and on pilots that could be scaled-up with Government funds if found to be successful. 73. Parallel financing: There were several project activities for which it was clear that funding would be provided by NRHM and therefore were dropped. There was also a clear set of activities that would remain funded by the project (like health insurance). However, there were activities were this was not clear-cut and there was significant back and forth on whether they would be funded by the project or NRHM. One example of this was procurement of ambulances for the 108 service and medical equipment that were originally in the procurement plan, then taken out and proposed by the state to be funded by NRHM PIP but this was not approved by the central MOHFW and therefore they were again included in the project and the procurement process had to start over again resulting in delays of up to 18 months in implementation. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 74. The M&E design proposed in the PAD was complex and convoluted. Several indicators in different formats were documented in the PAD. Table 9 (p. 31) listed PDO indicators but no targets. Table 11 (p. 35) listed numerous ‘priority performance indicators’ and targets, of which only two were PDO indicators. Table 12 (p.36) listed Page 27 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) health outcome indicators and targets. The ‘Implementation plan for intermediate results monitoring’ (p. 37-42) provided numerous process indicators for each project component. 75. Nine PDO and 15 intermediate results indicators were identified to provide a proxy metric for assessing progress towards the four health-related MDGs. Data for these indicators were meant to be obtained from MIS of ongoing vertical programs such as RCH, immunization program, TB program, vector borne disease control program and the HIV/AIDS control program. These indicators were meant to be disaggregated by geography and vulnerability, though this was not specifically articulated in the construct of the indicators, making it difficult to assess achievement of the PDO. The project also monitored performance against a set of output indicators (Table 11 of PAD) from NFHS and RCH surveys, prioritized by Government of Karnataka and for which quantitative targets were set in their Vision 2020 document for achievement over a period of 15 years. In support of attribution, the M&E design relied heavily on household level data to infer changes in health status of proposed beneficiaries, the preference and satisfaction with government and private health care providers, OOP, perceived and real financial and non-financial barriers to accessing care; and sources of financing for health consumption. 76. The restructuring of 2010 was noteworthy in that it rationalized, revised and clarified the PDO and intermediate results indicators and targets to better reflect causality, also adding relevant IDA core sector indicators. Consistent with the Credit’s support to the overall sector program, PDO indicators were measured state wide. Indicators measuring utilization by the poor and vulnerable were added and most appropriate sources of data for the indicators to minimize risk of unreliable or lack of data, were specified. The project now tracked sharply defined 6 PDO indicators and 11 intermediate results indicators with data from household surveys, administrative records, HMIS and facility surveys. 77. The results framework was further amended during the AF of 2012 to align focus on revised project priorities, namely improvement in health service delivery especially for poor and in underserved areas, public-private collaboration through public-private contracting of mobile health clinics and financing of tertiary care for poor beneficiaries; and financing of pilot activities for NCD prevention and control and road safety. Appropriate standardized and process indicators were included to measure progress towards improved service delivery and project supported pilots. The M&E framework was aligned with the state governments performance management and independent evaluation initiatives. 78. In March 2015, the PDO indicators of the results framework for the project was further revised through a Level II restructuring to (i) make more ambitious targets for the indicator tracking beneficiaries of the health insurance pilot, commensurate with the increased financing to the activity; (ii) significantly reduce the end line target for the program screening eligible women for cancer cervix given severe delays in the roll out of the NCD program and availability of only one year for implementation of the pilot; and (iii) track provision of emergency care in both public and private hospitals for patients of road traffic accidents transported by ambulance system in pilot districts, to capture progress towards integration of road accident injuries in the health insurance pilot supported by project. Three additional intermediate results were added to track public-private collaboration and emphasis on hospital care insurance/purchasing program for road accident victims. Two indicators pertaining to public-private collaboration for operationalization of mobile health clinics and ICT pilot were Page 28 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) dropped due to procurement progress (legal stalemate) and transfer of ICT in health portfolio to another department of the Government. M&E Implementation 79. The project was supported by staff dedicated for monitoring and evaluation of its interventions. Right at the outset, the M&E framework for the project relied largely on data from vertical disease control and surveillance programs 9RCH, IDSP, NVBDCP, NACP, RNTCP etc.), national surveys (NFHS), household and facility surveys to assess improvements in service utilization. The restructuring of December 2010, rationalized the sources of data to complement strengthening of existing administrative and health data systems (Health Management Information System) and reliance on national and state wide household surveys to minimize risk of reliability/quality of data sets, timeliness and cost and effort of securing the data. The project also relied on and implemented rigorous evaluations—health insurance pilot, implementation of health care waste management, quality of works financed by the project. 80. The Task Team must be commended for systematically and proactively leveraging the facility of project restructuring to rationalize, clarify and revise the results framework to provide better evidence for achievement of its development objectives. M&E Utilization 81. Data available from non-project sources, such as other vertical disease control programs and facility and household surveys were used extensively to track progress of the project. Independent evaluations for pilot interventions were conducted and findings used to inform policy decisions regarding scale. The project effectively used milestone-based approach to disburse funds linked to key institutional and health financing reforms both in the original project as well as the AF. The changing landscape and government/project priorities were effectively factored into the project M&E design and framework, through three critical restructurings (Dec 2010; Dec. 2012 and Mar. 2015) over the life of the project to support tracking of implementation progress and achievement of development objectives. Justification of Overall Rating of Quality of M&E Substantial 82. The M&E design as it evolved with the project, the context it operated in and the priorities it addressed, and its implementation was sufficient to assess the achievement of objectives. The initial M&E framework was convoluted and had weaknesses such as absence of targets for 6 of the 9 PDO indicators, numerous priority performance indicators and targets which reflected only two PDO indicators; health outcome indicators and targets and implementation plan for intermediate results monitoring, making progress towards achievement of DOs difficult to assess. The Aide Memoire from the MTR (August 2010) does an excellent analysis of data available from various independent sources [NFHS 2 (1998-99) and 3 (2005-06), District level health and facility survey-2 (2002-03); DLHS 2 (2007-08), Health utilization and expenditure study in Karnataka (2004); and Human resource study (2008)]; to assess trends in PDO indicators and their estimates disaggregated by geography and Page 29 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) socio-economic status and how these compare with program data, HMIS and administrative data. The M&E framework was proactively fixed with the restructuring of December 2010 to allow for unambiguous tracking of the implementation progress of the project and its contribution to the achievement of stated DO. The causal chain articulated in the results framework for the Additional Financing was clear and well-defined. Additionally, the Task Team effectively identified and calibrated the results most reflective of the project design and priorities and could track progress efficiently. Finally, the project judiciously used data from HMIS, national programs, independent surveys and studies and rigorous third party evaluations to validate progress, trends and outcomes. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 83. Review of project documentation indicate compliance with both environment and social safeguards policies of the Bank. The Tribal Action Plan was developed and implemented by the project to ensure improved access and uptake of preventive, curative and essential public health services. Social safeguards were assessed to be Satisfactory in all 17 ISRs filed for the project. Aside of complying with both national guidelines for management of bio-medical waste, the project also met Bank’s policies for environment assessment (EA), which were assessed to be Satisfactory in 10 of 17 missions. The seven times EA was assessed to be Moderately Satisfactory was on account of delayed and slow implementation of the environment action plan and due to a break in the contracts of district consultants for health care waste management. The project undertook interesting innovations for handling bio-medical waste management, especially liquid waste from health facilities and making them mercury free. At the end of the project, capacities of over 9,000 government health facilities were enhanced to segregate, properly disinfect and dispose solid and liquid biomedical wastes. Disposal of solid wastes in line with the National/State Biomedical Waste Management Rules/Pollution Control regulation was conducted through contracted private agencies—Common Treatment Facilities. 84. Procurement towards the traditionally financed components of the project were in accordance with specified World Bank guidelines. Procurement risks were adequately identified and risk mitigation measures specified in the PAD to ensure efficient and effective outcomes. Procurement under the programmatic component of the original credit followed the Karnataka Transparency in Public Procurement (KTTP) Act and Rules, 2000, in pursuance of the implementation of the procurement reform action for the state consequent to the World Bank’s state procurement assessment report of 2001. The project appropriately developed and adapted procurement manual and standard bidding documents of Bank for procurement of works and goods through National Competitive Bidding. The Detailed Implementation Review of the India Health Sector released in 2006- 07 had consequences for all World Bank operations in the health sector. A health sector procurement reform action plan was introduced in the project to address critical operational concerns relating to procurement of health sector goods and services. Its implementation was reviewed every six months with a detailed progress report being provided by the project. The state did make progress on procurement reform—systematically building capacity; strengthening institutions such as the KDLWS; clarifying and streamlining procurement procedures; and operational management. The e-procurement platform proposed by the state for goods and works was assessed acceptable to the Bank and deployed. Overall, procurement progress was consistently rated Moderately Satisfactory (15 of 17 ISR assessments) through the life of the project, due to slow pace of decision making and procurement processes. Delayed release of payments and lax monitoring and control of contracts were identified as internal controls issues, limited capacity for procurement and contract management in project Page 30 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) and delayed decision making were identified as issues during review missions and specifically during the procurement post reviews of 2008-09, 2009-10 and 2010-11. 85. Financial Management during the life of the project was largely rated Satisfactory (11 of 17 review mission). Early in the project implementation cycle (2008), FM performance was downgraded to Moderately Unsatisfactory on account of delayed finalization of the project FM manual and internal audit arrangements. The issue was resolved by September 2008, causing the FM rating to be upgraded to Moderately Satisfactory. From FY 2010, the project submitted Financial Management Reports and audit reports in time and followed all due diligence process. An FM action plan was agreed during the MTR to mitigate risks around weak capacity of FM personnel in health department and poor effectiveness of internal audit resulting in consistent delays of appointment and reporting. The consequent action taken on the plan was timely and acceptable to the Bank. For effective financial management, the project has hired the services of financial management consultant that ensured better internal control and timely response to all the financial management related actions. This was identified as one of the good practice, and the national health mission have adapted the same for implementation of central program, and further strengthen the district level reporting on financial management for the entire state. There were no major issues concerning financial management. The project disbursed 100% of both original and AF Credit demonstrating Satisfactory financial performance. C. BANK PERFORMANCE Quality at Entry 86. The design of the project relied on the experiences and lessons learned from Bank-supported national and state level projects, especially the Karnataka Health Systems Development Project implemented during 1996-2004. These included the creation of incentives to improve identified performance indicators; strengthening of stewardship role of the Government and establishing stronger partnerships with private and NGO sectors. Using a PHRD grant, the Bank supported 3 core teams to undertake an extensive consultative process and assisted the government to come with concrete plans to improve health service delivery covering aspects of (a) improvement of public sector performance; (b) Involvement of private sector in service delivery; and, (c) Development of new health financing strategies. In terms of safeguards, the client undertook an environmental assessment and prepared an indigenous people’s plan. A Quality Enhancement Review was undertaken for the original project, which raised concerns about complicated and ambitious project scope, the need to ensure better linkages with the past and ongoing project, and strengthen economic and financial analysis sections. The project team accordingly simplified the project scope, expanded the PAD Annex on ongoing projects and redid the economic analysis. The quality at entry therefore is considered Satisfactory. Quality of Supervision 87. Despite changes in the task team, the core focus on performance-based financing to achieve the PDOs and broader impact on the poor and achievement of the MDGs was retained. In addition to regular six monthly implementation support missions, the Task Team members, who were based in the Country Office, frequently visited the state, undertook field visits and corresponded almost daily with project officials to keep the implementation momentum going. Over the last nine years, over 30 visits were undertaken by the Task Team Leaders and Operations Officers on the team to the project. Project related documentation was of acceptable Page 31 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) quality, clearly articulating issues for Borrower and Management attention and agreed action points. The Task Team is commended for its candor witnessed in downgrading the implementation ratings during the slow-down in February 2008 and August 2014. There were 22 documented implementation support missions and the task team rated the DO as Moderately Satisfactory in 12 instances while IP was rated Moderately Unsatisfactory and Satisfactory on 2 and 10 missions respectively. 88. The Task Team was opportunistic in responding with alacrity to the changing environment and emerging priorities of the state by swiftly preparing and negotiating an AF to further strengthen ongoing systems strengthening interventions, and shepherd in new reforms. Prevention, screening and treatment of NCDs and its risk factors using a pilot approach; focus on road safety and its emergency and clinical response; systematic strengthening and deployment of the state health insurance scheme—VAS were areas that the Task Team correctly identified, prioritized for implementation and evaluation for state-wide scale-up using government resources. Global experts on management of NCDs and their risk factors; road safety, health insurance, results based financing and leveraging ICT for improved service accountability were brought in as required to assist the design of interventions and their implementation. 89. Additionally, the Team was bold enough to recognize risks posed by the unresolved legal stalemate in procurement of NGO contracts in 2015 and restructure the project to re-allocate unspent funds from identified NGO contracts to the health insurance component. The disbursement of this reallocated amount was against milestones to coopt road safety casualties and injuries under the ambit of the health insurance scheme. 90. The overall quality supervision is rated Satisfactory. Justification of Overall Rating of Bank Performance Satisfactory 91. The project design went beyond the traditional investment financing operation in that it supported the reform program of the Government and introduced a results-focused milestone based financing modality, which was more closely aligned with the new PforR instrument that was introduced several years later. The innovative design helped the state to build institutional capacity and advance reforms, despite limited capacity, especially at the district level. During supervision, steeply increased resources from the GOI’s NRHM as well as the GOK required the project to make mid-course corrections and focus on areas aligned with PDOs and not supported by new national and state initiatives. The task team however sustained focus on equity and institutional capacity development through restructuring of the project several times and using the milestone-based approach with a clear focus on the improving services for the poor and addressing market failures. D. RISK TO DEVELOPMENT OUTCOME 92. The risk that the development outcome is not sustained is negligible. The project never represented more than 7% of the total state funding and the Government of Karnataka has consistently integrated activities started by the project into the state’s health sector program through its own or the National Health Mission funding. There is always a risk that with a change of government, priorities may change but in the current context of Karnataka this is unlikely as health has acquired significant prominence (thanks in part to the results of the project) at both state and national level. Karnataka was selected as one of three states supported by the NITI Aayog (former National Planning Commission) to be a model for the other states as part of the central government’s push for Page 32 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) competitive federalism in the social sectors, signaling that even with a change of government activities initiated by the project are more likely to be scaled-up and further expanded than discontinued. 93. The success of SAST has also brought state and national-level prominence as a successful health insurance agency. Thus, it now operates all state and centrally funded insurance schemes which presents a risk of overburdening its institutional and implementation capacity. This risk is mitigated by the announcement on November 1, 2017 that the multiple schemes operated by SAST will be consolidated and merged into a single Universal Health Coverage Scheme. This integration both makes sense from a technical point of view and will allow SAST to achieve significant efficiencies and reduce the operational burden as duplication is eliminated and both human and the use of human and financial resources is rationalized. Moreover, the Bank continues to provide technical assistance to in its role as a knowledge hub through the Lighthouse India initiative. V. LESSONS AND RECOMMENDATIONS 94. KHSDRP has provided many lessons over the years and these have already been used to shape the Bank’s strategy for the health sector in India. The lessons learned from the original project and incorporated into the AF were also used to inform the design of a new wave of state-level health systems strengthening projects in India. 95. Aligning Bank investment lending project to government’s reform program can help to steer the overall focus of public financing in the sector towards equity and efficiency. With the advent of the National Health Mission, the strategy of the Bank evolved from funding the essential package of services to targeting specific system-wide improvements such as financing, systematic quality assurance and public-private collaboration that would result in improved effectiveness of the much larger health sector program funded by a combination of resources from state and central governments. 96. A longer-term engagement by the Bank clearly focusing on building critical institutions and health financing reforms including finding innovative approaches address often neglected areas such as healthcare waste management contributed to the success in Karnataka in terms of reducing long standing inequities and improving health system performance. 97. Results based financing catalyzes effective policy dialogue and institutional development. KHSDRP opened the path to shift from input-based financing to results-based financing. Most health projects in India are now hybrids that combine input finance with results based financing through Disbursement-Linked Indicators and it is likely that future projects will be purely result-based. Using a milestone-based approach for disbursement will help in bringing about key policy and institutional reforms as shown in case of the SAST. This clearly demonstrates that more effective and transparent public institutions can be built to achieve State Government’s objective of providing financial protection for the poor and make progress towards achieving Universal Health Coverage. 98. Well-designed pilots for responding to emerging situations such as increased burden of NCDs and road traffic accidents helped the government to assess operational feasibility and cost- effectiveness for scaling-up of new initiatives using domestic finances. The NCD pilot project has done a systematic diagnosis of the NCDs in the state and demonstrated the means of implementing a more effective intervention. The implementation lessons have been disseminated internally within the state government and the results are also being used to re-design the national NCD prevention and treatment program. The pilots also used innovations such as outsourcing of Page 33 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) confirmatory diagnostics to private Medical Colleges which was successful and incorporated by other disease control programs in the state. 99. Bank operations can effectively be used to spearheaded policy dialogue on new areas that need priority attention such as road safety and setting standards for service delivery. This was well recognized by the State and Central governments and as result the point person from the project working on road safety was seconded to the Roads department for taking the initiative forward beyond the project mandate. 100. Realistic implementation ratings helps renewed focus on PDOs. Realism in implementation rating, especially when there was a slow-down and timely restructuring of the project responsive to changes in the operating environment (such as NRHM and enhanced GOK resources for the health sector) helped in course corrections and renewed focus on PDO achievement. . Page 34 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Improve health service delivery, public-private collaboration, and financing, particularly for the benefit of underserved and vulnerable groups in Karnataka Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Direct project beneficiaries Number 0.00 306000000.00 306000000.00 446000000.00 31-Mar-2006 31-Mar-2016 31-Mar-2017 31-Mar-2017 Female beneficiaries Percentage 0.00 50.00 50.00 52.00 31-Mar-2006 31-Mar-2016 31-Mar-2017 31-Mar-2017 Comments (achievements against targets): The project mainly benefited the underserved populations from seven less developed districts and also played an important role in enhancing health insurance cover for Below Poverty Line Households across the stage. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percent of births/deliveries Percentage 65.00 90.00 90.00 94.00 Page 35 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) occuring in a health facility 31-Mar-2006 31-Mar-2016 31-Mar-2017 31-Aug-2016 Comments (achievements against targets): The National Family Health Survey (NFHS) round 4 (2015-16) by the International Institute for Population Sciences confirms that Karnataka state has achieved institutional delivery rate of 94.3% and deliveries at public facilities contributed to 61.4%. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percent of birth/deliveries Percentage 37.00 55.00 55.00 77.00 occuring in a health facility among the poor 31-Mar-2004 31-Mar-2016 31-Mar-2017 31-Mar-2017 Comments (achievements against targets): The National Family Health Survey (NFHS) round 4 (2015-16) by the International Institute for Population Sciences confirms that Karnataka state has achieved institutional delivery rate of 94.3% and disaggregated data of backward districts indicates achievement of 77 percent. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Percent of population Percentage 46.00 78.00 78.00 78.00 receiving outpatient care in government facilities in 7 31-Mar-2006 31-Mar-2016 31-Mar-2016 31-Mar-2017 less-developed districts Comments (achievements against targets): This data is based on reported administrative data by the Health Management Information System and the as reported by the end-line evaluation commissioned by the borrower. Page 36 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of health claims paid Number 0.00 45000.00 150000.00 152337.00 by health insurance pilot program benefitting Below 31-Mar-2006 31-Mar-2016 31-Mar-2017 31-Mar-2017 Poverty Line (BPL) households Comments (achievements against targets): The project successfully achieved the revised target which is three fold higher than the original. An Impact Evaluation undertaken by external agency has shown that eligible poor households had significantly reduced out-of-pocket health expenditures with potential increases in use. More importantly, the study reported a reduction in mortality rates from conditions covered by the scheme. Unlinked Indicators Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of women screened Number 0.00 60000.00 1000.00 27156.00 for cervical cancer in pilot districts 31-Mar-2006 31-Mar-2016 31-Mar-2017 31-Aug-2016 Comments (achievements against targets): The project exceeded the revised target for screening of cervical cancer. The original target however was revised based on implementation experiences. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion percent change in number of Percentage 0.00 10.00 10.00 183.00 road traffic accident patients transported by government 31-Mar-2006 31-Mar-2017 31-Mar-2016 31-Aug-2016 Page 37 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) ambulance system (108 system) in pilot districts Comments (achievements against targets): The Baseline was kept constant at 8,800 based on data available in 2012 due to which the achievement is far higher. The data is from the agency contracted for providing Ambulance services. A.2 Intermediate Results Indicators Component: Strengthening existing government health programs Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Health personnel receiving Number 0.00 25500.00 25500.00 91415.00 training (number) 22-Aug-2006 31-Mar-2016 31-Mar-2016 31-Aug-2016 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Health facilities constructed, Number 0.00 375.00 375.00 392.00 renovated, and/or equipped (number) 31-Mar-2006 31-Mar-2016 31-Mar-2017 31-Aug-2016 Comments (achievements against targets): Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at Page 38 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) Target Completion Percent of government Percentage 0.00 47.00 47.00 47.00 Primary Health Centers (PHCs) providing round-the- 31-Mar-2006 31-Mar-2016 31-Mar-2016 31-Aug-2016 clock services Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Training materials on non- Yes/No N Y Y Y communicable disease activities for clinical staff are 31-Mar-2006 31-Mar-2016 31-Mar-2016 31-Aug-2016 developed Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Guidelines and standards for Yes/No N Y Y Y trauma care services are developed 31-Mar-2012 31-Mar-2016 31-Mar-2016 31-Aug-2016 Comments (achievements against targets): Component: Innovations in service delivery and health financing Page 39 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Births (deliveries) attended Number 0.00 5200000.00 5200000.00 5800000.00 by skilled health personnel (number) 31-Mar-2005 31-Mar-2016 31-Mar-2016 31-Aug-2016 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion number of claims paid to Number 0.00 0.00 120000.00 123462.00 private sector hospitals by the government health 31-Jan-2015 31-Mar-2016 31-Mar-2017 31-Aug-2016 insurance program supported by the project (cumulative) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion number of private hospitals Number 0.00 0.00 50.00 256.00 empaneled by SAST that have submitted at least 100 31-Mar-2006 31-Mar-2016 31-Mar-2016 31-Aug-2016 claims (cumulative) Page 40 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion number of hospitals Number 0.00 0.00 150.00 766.00 empaneled to provide care to road accident victims 31-Jan-2015 31-Mar-2016 31-Mar-2017 31-Aug-2016 under the government program (cumulative) Comments (achievements against targets): Page 41 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) B. KEY OUTPUTS BY COMPONENT Objective/Outcome1 : Improve health service delivery for the underserved and vulnerable groups: 1. Direct project beneficiaries 2. Percent of births/deliveries occurring in a health facility 3. Percent of births/deliveries occurring in a health facility among the poor Outcome Indicators 4. Percent of population receiving outpatient care in government facilities in 7 less-developed districts 5. Number of women screened for cervical cancer in pilot districts 1. Health personnel receiving training 2.Number of health facilities constructed/renovated and/or equipped Intermediate Results Indicators 3. Percent of government Primary Health Centers providing round- the- clock services 1. 91,415 health personnel received training in various areas including organizational development, quality assurance, healthcare waste management etc.; first referral units providing 24 hour services made operational in each district; good clinical practice guidelines and standard bidding documents for health sector developed; Key Outputs by Component 2. A total of 340 civil works undertaken during the original project (linked to the achievement of the Objective/Outcome 1) while 24 civil works were supported through Additional Financing. Food safety equipment supplied; citizen’s help desks at hospitals; pilots on non-communicable diseases and road safety successfully completed 3. Efficient procurement, introduction of Information communication technology initiatives in the sector Page 42 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) Objective/Outcome 2 Public-private collaboration and financing, particularly for the benefit of underserved and vulnerable groups 1. Number of health claims paid by health insurance pilot program benefitting below poverty line households Outcome Indicators 2. Percent change in the number of road traffic accident patients transported by 108 ambulance system who received emergency care at Government Hospitals in pilot districts 1. Number of claims paid to private sector hospitals by government health insurance program Intermediate Results Indicators 2. Number of private hospitals empaneled by SAST that have submitted at least 100 claims 1. Progress report on public-private partnerships 2. Contracting 124 mobile health clinics to deliver services to unreached and underserved areas; Contracting of 42 specialist doctors Key Outputs by Component in 7 underserved districts; Massive training of village health and (linked to the achievement of the Objective/Outcome 2) sanitation committees under the public health competitive fund; Institutional development of SAST for phased expansion of health insurance for below poverty line households 3. Performance-linked financing Page 43 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Supervision/ICR Jorge A. Coarasa Bustamante Task Team Leader(s) Atin Kumar Rastogi Procurement Specialist(s) Anantha Krishna Karur Financial Management Specialist Shreelata Rao-Seshadri Social Safeguards Specialist Lucy S. Lotha Team Member Shankara G. Krishnamurthy Team Member Gopalakrishna Gururaj Team Member Anupam Joshi Environmental Safeguards Specialist Bathula Amith Nagaraj Team Member B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY05 70.791 351,905.79 FY06 44.875 214,221.72 FY07 13.810 74,573.20 FY08 0 70.06 FY09 0 0.00 Total 129.48 640,770.77 Supervision/ICR Page 44 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) FY07 16.126 114,606.76 FY08 20.917 128,851.97 FY09 32.049 204,363.29 FY10 46.127 243,790.78 FY11 52.223 170,311.71 FY12 46.604 189,306.96 FY13 41.000 159,828.64 FY14 45.906 171,505.96 FY15 32.080 91,258.36 FY16 24.369 114,096.84 FY17 26.189 119,918.61 FY18 9.222 47,781.49 Total 392.81 1,755,621.37 Page 45 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Component 1. Strengthening existing government health 0 75.30 0 programs Component 2. Innovations in service delivery and health 0 115.80 0 financing Component 3. Project Management, Monitoring 0 20.70 0 and Evaluation Total 0.00 211.80 0.00 Page 46 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) ANNEX 4. EFFICIENCY ANALYSIS The economic and financial analysis in the Project Appraisal Document was presented in Annex 9. The analysis focused on the distribution of government health spending (including the extent to which the spending was pro-poor), the ability of the government to respond to existing market failures, and the fiscal and economic situation in Karnataka. This included a brief discussion of the financial sustainability and implications of the project on recurrent expenditures. The analysis, however, did not explore the potential consequences or effectiveness of the specific interventions. In this annex, we build on the initial analysis by assessing the major project components on four dimensions of 1) technical efficiency; 2) cost-effectiveness; and 3) equity. The objective of the project was to increase utilization of essential curative and public health services, particularly in underserved areas and among vulnerable groups. The project had three components: i) strengthening existing government health programs ($75.3 million); ii) innovations in service delivery and health financing ($115.9 million); and iii) project management, monitoring, and evaluation ($20.6 million). The project was used as an instrument to fill gaps not covered by other sources of financing, as well as to identify other interventions that could address new challenges faced by the health sector (including the rise of non-communicable diseases). Technical efficiency The project components were designed to address the burden of disease in Karnataka and, as a result, contributed to significant improvements in health outcomes. The infant mortality rate in the State fell from 48 deaths per 1000 live births in 2006 to 31 deaths per 1000 live births in 2013. Meanwhile, the maternal mortality rate declined from 178 deaths per 100,000 live births in 2007-08 to 138 deaths per 100,000 live births in 2012-13. The restructuring and additional financing, in particular, focused on the growing burden of non- communicable diseases (NCDs). The project focused on improving primary health care, as many of the essential preventive services can be delivered at this level for a significantly lower cost than at more specialized facilities. The project invested in the renovation and construction of primary health care facilities. A total of 364 primary healthcare facilities were renovated under the project, which also supported training of 91,415 health personnel with specific focus on organizational development, quality assurance and healthcare waste management. This allowed facilities to expand their patient volume. While less than half (48%) of primary health centers conducted at least 10 deliveries per month in 2007-08, the share rose to 62% by 2012-13. The focus on facilities that had at least 10 deliveries per month is notable, as volume has been found to be positively associated with quality of maternal and obstetric services (Kruk et al., 2016). By improving the quality of services through additional training of human resources, the project resulted in significant uptake of health services. Outpatient utilization increased substantially at government facilities in the seven less developed districts (rising to 78% in 2016 as compared to 46% in 2006). While specific data are not available, this likely contributed to reductions in complications related Page 47 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) to poor quality of care and those related to delays in seeking treatment. In addition, the inclusion of pilots for NCDs and access to emergency services for road traffic accident injuries was clearly aligned with the changing burden of disease in Karnataka. While significant improvements were achieved in MCH outcomes, the growing burden of NCDs presented challenges. In 2012, the main cause of DALYs was cardiovascular disease, prompting the need to invest more in primary health care services to ensure prevention, early detection, and adequate treatment. Figure 1: Causes of DALYS in Karnataka, 2006 and 2012 The introduction of mobile clinics substantially improved access to essential health services at a low cost. The mobile clinics were responsible for the provision of primary healthcare services like curative services for common illnesses, first aid, referral services, family planning services, antenatal and postnatal care, immunizations, counseling particularly on HIV/AIDS, implementation of national health programs, environment sanitation and health education and other health-related services in the allotted area. In total, 107 operational units remained at the end of the project amounting to a total cost of 866,779,153 Rs. The mobile clinics served 7,243,415 patients, resulting in a per capita cost of 116 Rs. Page 48 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) Cost-effectiveness The inclusion of health interventions supported by the project was guided by the changing burden of disease in Karnataka and international evidence of their effectiveness. The project supported a basic package of cost-effective maternal and child health interventions through the improvements in primary health care delivery models. In addition, the project piloted several cost-effective initiatives for NCDs, most importantly in relation to screening of cervical cancer. In 2012, India had the highest age- standardized rate of cervical cancer in South Asia (22 per 100,000). Almost 1.2% of deaths among women in Karnataka were due to cervical cancer in 2012 (IHME, 2017). Cervical screening can significantly reduce mortality by early detection and treatment and has been identified as a highly cost- effective intervention in India, with a cost-effectiveness of $10 per life saved (Goldie et al., 2005). During the project, 27156 women were screened for cervical cancer. The Lives Saved Tool was used to estimate the cost-effectiveness of Component 2A Innovations in service delivery ($85 million). Conservatively, we only accounted for changes in institutional delivery, attributing those directly to the project. Gains were calculated for each year of project implementation as compared to the baseline situation of 2006. Compared to the initial institutional delivery rate of 65%, increases in institutional delivery during the project resulted in 2763 maternal deaths averted. Taking the average age at death and female life expectancy, this can be converted into 121,572 life years. Since the total cost of Component 2 was US$ 85 million this resulted in a cost-effectiveness ratio of $699/life year saved. Using the threshold of Karnataka’s GDP per capita of approximately $3200, this component was highly cost-effective even under the conservative estimates. Table 1: Number of maternal deaths averted as a result of increases in institutional delivery during the project Year Number of maternal deaths averted 2007 72 2008 143 2009 286 2010 283 2011 317 2012 318 2013 321 2014 324 2015 373 2016 326 Total 2,763 Page 49 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) Equity An explicit goal of the project was to increase utilization of health services for the poor and vulnerable populations. As noted in the PAD, at the outset of the project, significant differences in utilization existed between urban and rural areas and the better-off southern and poorer northern districts in the State. The project reduced inequities by focusing on the less developed districts: Bagalkot, Bidar, Bijapur, Chamarajanagar, Gulbarga, Koppal, and Raichur. The construction of primary health care centers in districts where there was a shortfall reduced regional disparities in access to and utilization of health services. As evidenced from household surveys, in 2007-08 the institutional delivery rate ranged from 25% in Koppal to 95% in Udipi district (Figure 2). By 2012-13, this gap narrowed to 71% and 99%, respectively. Despite free care at public facilities, a significant share of the poor and vulnerable populations sought care from private facilities before the implementation of the project due to poor quality and lack of access to public facilities. This resulted in large out-of-pocket expenditures. During the project period, the share of deliveries at public facilities nearly doubled from 35% to 61%, with over two thirds of deliveries in rural areas occurring at public facilities supported by the project. The administrative data also showed notable increases in the use of outpatient services in public facilities in the seven less-developed districts. Figure 2: Institutional delivery (% of pregnant women), 2007 and 2013 by districts in Karnataka 100 99 99 99 99 96 96 96 97 97 97 98 98 96 95 95 96 94 95 92 92 93 94 90 90 91 90 87 87 85 84 82 83 83 80 81 80 80 80 77 77 75 77 73 73 71 71 70 65 67 65 64 64 62 62 60 Percentage 51 48 47 50 46 41 40 30 25 20 10 0 2007-08 2012-13 Page 50 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) Expansion of insurance for the poor significantly reduced out-of-pocket payments and resulted in better health outcomes. The project supported the government insurance program (Vajpayee Arogyashree scheme) that provided free tertiary care to households below the poverty line (BPL) in Karnataka. Results from an independent evaluation suggest lower mortality rates among households below the poverty line covered under the scheme, as well as substantial reductions in out-of-pocket payments with OOP at tertiary level facilities 64% lower for BPL households covered under the scheme than those just above the threshold (Sood et al., 2014). According to the NFHS data, coverage of government sponsored health insurance schemes at household level nearly tripled (from 10% to 28%) by 2015-16 when compared to 2005-6, and this trend was more predominant in rural areas - the main target areas for the project – as compared to urban areas (32% and 23%, respectively). Budget implications The project’s components included both capital investments, such as construction and improvements of primary health care facilities, and programmatic investments, including screening for cervical cancer. These results in recurrent costs and will need to be absorbed into Karnataka’s regular budget. The available government resources for health increased substantially over the project cycle. The project did not represent more than 7% of total annual government health spending in Karnataka. Page 51 of 52 The World Bank Karnataka Health System Development and Reform Project ( P071160 ) ANNEX 5. BORROWER COMMENTS The Bank team shared the Implementation Completion and Results (ICR) reporting process with the project. The objectives of ICR and responsibilities of the Bank and the Borrower for completing the ICR were discussed in detail during the implementation support mission of September 2015. The draft ICR was shared with the borrower on January 08, 2018. The borrower / KHSDRP team have reviewed the ICR and communicated (January 16, 2018) their acceptance with the findings and have no comments. Page 52 of 52