Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005309 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA No 55570-CI, Grant No D0030-CI) ON A CREDIT IN THE AMOUNT OF SDR 23.7 MILLION (US$35 MILLION EQUIVALENT) AND ON A GRANT IN THE AMOUNT OF SDR 23.7 MILLION (US$35 MILLION EQUIVALENT) TO THE REPUBLIC OF CÔTE D’IVOIRE FOR THE Health Systems Strengthening and Ebola Preparedness Project August 21, 2020 Health, Nutrition and Population Global Practice Africa West Region CURRENCY EQUIVALENTS (Exchange Rate Effective March 13, 2020) Currency Unit = CFA Franc (CFAF) 595.39 = US$1 US$ 1.38 = SDR 1 FISCAL YEAR July 1 - June 30 Regional Vice President: Ousmane Diagana Country Director: Coralie Gevers Regional Director: Dena Ringold Practice Manager: Gaston Sorgho Task Team Leader: Montserrat Meiro-Lorenzo ICR Main Contributor: Maria Rosa Puech Fernandez ABBREVIATIONS AND ACRONYMS ACV Contracting and Verification Agency (Agence de Contractualisation and Vérification) AfD French Development Agency (Agence Française de Développement) CAS Country Assistance Strategy CBA Cost Benefit Analysis CBO Community Based Organization CHW Community Health Workers CNAM The Health Insurance Agency (Caisse Nationale d’Assurance Maladie) CNT-FBR National Technical Committee for Performance-Based Financing COGES Health Management Committees (Committee de Gestion Santé) CPF Country Partnership Framework CRI Corporate Result Indicator DAF Directorate of Financial Affairs (Ministry of Health) DDS District Health Directorates (Directions Départementales de Santé) DGS Directorate General for Health DGTCP Directorate General for the Treasury and Public Accounting DHIS District Health Information Software DIIS Direction de l’Informatique et de l’information Sanitaire (Directorate for IT and Health Information) DIPE Directorate of Information, Planning and Evaluation DPPS Director of the Department of Planning and Forecasting DPs Development Partners DRS Regional Health Directorates EA Environmental Assessment ECD Health District Management Teams ESMF Environmental and Social Management Framework ESSAF Environmental and Social Screening and Assessment Framework EU European Union FBR Financement Basé sur les Résultats FM Financial Management GDP Gross Domestic Product GF Global Fund GFF Global Financing Facility HMIS Health Management Information System IBRD International Bank for Reconstruction and Development IDA International Development Association INHP National Institute of Public Hygiene (Institut National de l’Hygiène Public) M&E Monitoring and Evaluation MNCH Maternal, Neonatal and Child Health MOF Ministry of Finance MOH Ministry of Health MSHP Ministry of Health and Public Hygiene (Ministère de la Sante et de l’Hygiène Publique) MSLS Ministry of Health and Fight Against AIDS (Ministère de la Sante et de la Lutte contre le Sida) MSP Ministry of Social Protection NGO Non-Governmental Organization NPBF National Performance Based Financing NPV Net Present Value PAD Project Appraisal Report PBF Performance-Based Financing PDO Project Development Objective PIU Project Implementation Unit PNDS National Health Development Plan PPA Project Preparation Advance PRSSE Health Systems Strengthening and Ebola Preparedness Project (Projet de Renforcement du Système de Santé et de Réponse aux Urgences Épidémiques) RASS Annual Health Statistics Report (Rapport Annuel Statistiques de Santé) RCI République de la Cote d’Ivoire RR Rate of Return SIGFIP Integrated Public Finance Management System (Système Intégré de Gestion des Finances Publiques) SNDS National Strategy for Health Development (Stratégie Nationale de Développement Sanitaire) SPARK Strategic Purchasing and Alignment of Resources and Knowledge in Health Project STEP Systematic Tracking of Exchanges in Procurement UHC Universal Health Coverage UHI Universal Health Insurance (Couverture Maladie Universelle) UNICEF United Nations Children’s Fund UNFPA United Nations Populations Fund WHO World Health Organization TABLE OF CONTENTS DATA SHEET ............................................................................................................................ 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 A. CONTEXT AT APPRAISAL .........................................................................................................5 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE)...................................... 12 II. OUTCOME...................................................................................................................... 15 A. RELEVANCE OF PDOs ............................................................................................................15 B. ACHIEVEMENT OF PDOs (EFFICACY) ......................................................................................16 C. EFFICIENCY ...........................................................................................................................24 D. JUSTIFICATION OF OVERALL OUTCOME RATING ....................................................................26 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................26 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 27 A. KEY FACTORS DURING PREPARATION ...................................................................................27 B. KEY FACTORS DURING IMPLEMENTATION .............................................................................28 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 31 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................31 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE .....................................................32 C. BANK PERFORMANCE ...........................................................................................................33 D. RISK TO DEVELOPMENT OUTCOME .......................................................................................36 V. LESSONS AND RECOMMENDATIONS .............................................................................. 36 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 40 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 56 ANNEX 3. PROJECT COST BY COMPONENT ............................................................................ 59 ANNEX 4. EFFICIENCY ANALYSIS ............................................................................................ 62 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 64 ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ...................................................................... 73 ANNEX 7. KEY INDICATORS IN THE HEALTH DISTRICTS UNDER PBF........................................ 74 ANNEX 8. GEOGRAPHIC AREAS OF PROJECT INTERVENTION ................................................ 85 ANNEX 9. SUMMARY OF THE FIELD VISIT DURING THE ICR MISSION ..................................... 86 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name Health Systems Strengthening and Ebola Preparedness P147740 Project Country Financing Instrument Cote d'Ivoire Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency Republic of Côte d'Ivoire Ministry of Health and the fight against HIV/AIDS Project Development Objective (PDO) Original PDO To strengthen the health system and improve the utilization and quality of health and nutrition services in selected regions. Page 1 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 35,000,000 35,000,000 33,069,390 IDA-D0030 35,000,000 35,000,000 31,309,069 IDA-55570 Total 70,000,000 70,000,000 64,378,459 Non-World Bank Financing 0 0 0 Borrower/Recipient 7,000,000 0 0 Total 7,000,000 0 0 Total Project Cost 77,000,000 70,000,000 64,378,459 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 25-Nov-2014 16-Mar-2015 12-Feb-2018 31-Jan-2020 31-Jan-2020 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 04-Oct-2019 55.02 Change in Results Framework Reallocation between Disbursement Categories KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 09-Feb-2015 Satisfactory Satisfactory 0 Page 2 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 02 12-Aug-2015 Satisfactory Satisfactory 10.00 03 23-Feb-2016 Satisfactory Satisfactory 12.31 04 16-Sep-2016 Satisfactory Satisfactory 14.12 05 05-Apr-2017 Satisfactory Moderately Satisfactory 16.22 06 18-Oct-2017 Satisfactory Moderately Satisfactory 21.79 07 18-Apr-2018 Satisfactory Satisfactory 29.80 08 26-Nov-2018 Satisfactory Satisfactory 39.82 09 28-Jun-2019 Satisfactory Satisfactory 54.87 SECTORS AND THEMES Sectors Major Sector/Sector (%) Health 100 Public Administration - Health 14 Health 86 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Human Development and Gender 100 Disease Control 14 Health Systems and Policies 70 Health System Strengthening 32 Reproductive and Maternal Health 19 Child Health 19 Nutrition and Food Security 16 Nutrition 8 Food Security 8 Page 3 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) ADM STAFF Role At Approval At ICR Regional Vice President: Makhtar Diop Ousmane Diagana Country Director: Ousmane Diagana Coralie Gevers Director: Timothy Grant Evans Dena Ringold Practice Manager: Trina S. Haque Gaston Sorgho Task Team Leader(s): Dominic S. Haazen Montserrat Meiro-Lorenzo ICR Contributing Author: Maria R. Puech Fernandez Page 4 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. Independent since 1960, the Republic of Cote d’Ivoire (RCI) had a population of 24 million people, with sixty ethnic groups and with as many languages. After almost two decades of strong economic growth, RCI experienced a series of economic and political crises in the first decades of the 21st century: a war between 2002-2007 and a six-month conflict in 2010-2011, after the 2010 elections. This turmoil undermined economic development and resulted in widespread deterioration of living standards. Stability returned after mid-2011, accompanied by strong economic growth during the 2012-2015 period, averaging 9 percent annually. Despite economic growth, at the time of appraisal, poverty remained high. A large proportion of the population lived in high vulnerability without any social protection at the time of preparation. The Human Development Index showed the RCI ranked 171 out of 187 countries, way below what would be expected for the country’s Goss Domestic Product (GDP). Table 1 Key Health Indicators RCI at Appraisal 2012 Indicator Value Maternal Mortality ratio 614 death per 100,000 Children under 5 mortality 108 deaths per 1,000 live births Basic immunization rates 65 percent Chronic children malnutrition 30 percent Women use of contraceptives methods 14 percent Source: 2012 Demographic and Health Survey, PAD964 2. The political and economic turmoil took a toll on the health system. In the first phase of the crisis (2002-2010), the majority of health facilities were closed in the north and central parts of the country (over 52 percent of health centers nationally) and only Non-Governmental Organization (NGO) facilities stayed open. During the post-electoral crisis (2011-2012), all health centers in the western part of the country and the city of Abidjan were closed. Nationwide many hospitals and health facilities were looted, decimated and in dire shape. 3. Governance and management of the health system was challenging. The Ministry of Health and Fight against AIDS 1 (in French MSLS) faced challenges at different levels at the time of appraisal. At central level, it lacked strategic planning capability and effective control mechanisms. At subnational level, while decentralization had been initiated, the model of health districts had not been fully implemented, with many districts not operational at the time of appraisal. The health information system was weak and fragmented, failing to generate quality and timely information. Funding for the health sector was highly centralized. Health had not been a high government priority in terms of funding 1At the time of the ICR, the ministry was renamed the Ministry of Health and Public Hygiene (MSHP). The ICR will use this acronym from this section on. Page 5 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) (measured as a percentage of the total government budget, or as a percentage of GDP). The budget was skewed towards tertiary health care, leaving few resources for primary and secondary care. 4. Barriers to the utilization of health services were numerous at the time of appraisal. There were availability constraints on both the demand and supply sides. In terms of access, only 45 percent of the population lived within five kilometers of a health center 2, with a large share of the poor lacking access to a health center or a general hospital. Distance to a health center was a major barrier to maternity care for the poorest women. The President’s Emergency Fund (PPU) and development partner interventions were in place to address access barriers 3. Affordability represented a key barrier to the use of services, in particular for the poorest quintiles. In response, in 2011 the government launched the targeted free health care initiative (TFHCI), which in 2012 was narrowed to provide free health care in public facilities for pregnant women, children aged 0-5, emergency care and malaria. At the time of appraisal, the government had started to work on National Health Insurance 4, looking into the experience of other countries in the region, implementing a strategy of pre-paid health insurance coverage. 5. While utilization of health services had increased after the end of civil unrest numerous problems persisted 5. Among these were: (i) health care facilities 6 were, for the most part, not reimbursed for the provision of free-health care services and reimbursement among health care facilities was unequal, favoring tertiary level facilities, which discouraged primary health care centers; (ii) there was no pre-payment verification system to prevent over-billing; (iii) there were disruptions in the flow of inputs to health facilities; and (iv) staff were demotivated. There were acceptability issues related to the attitude of providers and the perceived value and quality of the services by patients. As there was limited health financing allocated to operational costs, a strategy of financing based on performance was being considered to address this challenge. 6. Ebola epidemic outbreak. In mid-2014, Liberia, Sierra Leone and Guinea experienced Ebola outbreaks. The RCI did not have Ebola cases. However, given the nearby epicenter of the epidemic and the extensive border with Liberia and Guinea, the Government was concerned and reached out to the World Bank, requesting support to improve the country’s level of preparedness to respond to a potential outbreak 7. 7. To respond to these challenges, the Government had developed a National Health Policy and a National Health Development Plan (PNDS) for the 2012-2015 period. The main goal was to guarantee access to quality health care for all citizens, especially the most vulnerable. The PNDS aimed to tackle the above-mentioned barriers to the utilization of health services. The main objectives of the PNDS were to: (i) reinforce governance in the sector and the leadership of the MSLS; (ii) improve supply and the use of quality health services; (iii) improve maternal and child health; (iv) reinforce health prevention, promotion and medical outreach; and (v) reinforce the fight against disease and nosocomial infections. 2 Source: PAD for P147740. 4 November 2014. 3 The World Bank was supporting the country through the Emergency Infrastructure Project, while challenges to meet essential equipment requirements persisted. 4 The Government had proposed a Universal Health Coverage program (Couverture Maladie Universelle - UHI) in 2014. 5 Some of the challenges existed before the civil unrest and others were exacerbated by the civil conflict. 6 Only 1-5 percent of first level health facilities and 25 percent of health facilities in the second and third levels were reimbursed for free health care services. 7 Luckily the country was spared. Page 6 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 8. Donors supported the national health policy. Most donor-supported programs focused on providing inputs to meet the country’s needs. The Government and the World Bank took this support into consideration, as well as the World Bank’s ongoing support in related sectors, during project design. The World Bank-financed Social Safety Net Project 8 supported key areas relevant to the health sector, in particular the design of a targeting mechanism. 9. Sectoral and institutional context. Cote d’Ivoire’s health indicators were amongst the weakest in the region for its income level. At the time of appraisal, the country did not seem able to meet the Millennium Development Goals (MDGs), even though there had been improvements in health indicators in the 1990s, in particular among urban populations, with strong geographic disparities. Cote d’Ivoire had a rapidly growing young population, like other countries in the region (2.6 percent per year in 2013). This pace of population growth put pressure on basic social infrastructure. Other challenges in the health system were the heavy reliance on and higher number of doctors and nurses, compared to midwives and community health workers, relative to other countries in the region. Strategies were needed to attract and retain medical staff in rural and more challenging geographic areas. Unlike other countries in Africa, RCI did not employ community health workers to the same degree, and those it used were not well trained or well prepared to meet the realities on the ground. 10. Considering the barriers mentioned in paragraph 4 and the experience of neighboring countries, the RCI decided to move towards Performance-Based Financing (PBF) in 2014, as laid out in the two policy documents 9. The government had piloted two small projects 10 and was encouraged by their results and the experience in nearby countries (Benin, Burundi, Rwanda and Senegal). The health sector was the first to take on performance-based financing, supported by IDA and several international development partners. PBF was meant to help the country address issues of utilization and quality of health services, to stabilize or reduce the cost of these services; to promote the efficient use of scarce resources, and to improve staff motivation. Theory of Change (Results Chain) 11. The Project focused on interventions to strengthen the country’s health system and to improve the utilization and quality of health and nutrition services in selected regions. The Project supported the country’s goals for the health sector as described in the national policy documents 11, mentioned in paragraph 7. The project’s design complemented the country’s and other supporting donors’ work in the health sector. Given that other donors were providing input- specific support, the project focused on the following complementary areas: (i) providing technical assistance to develop a national health insurance system (UHI) and pilot a universal health care (UHC) approach and assisting in its implementation by covering the poor; (ii) piloting the Performance-Based Financing (PBF) approach to increase the volume and quality of services, targeting activities around maternal and child care, addressing linkages to the UHC; (iii) rehabilitating health facilities and providing equipment to PBF facilities, to support the provision of quality service; and (iv) supporting the development of a robust health management information system and health system management. 8 P143332 9 National Health Policy and a National Health Development Plan (PNDS) for the 2012-2015 period. 10 Abt Associates and the Elizabeth Glaser Pediatric Aids Foundation (EGPAF) led the two pilots. 11 National Health Policy and a National Health Development Plan (PNDS) for the 2012-2015 period. Page 7 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Implicit Theory of Change at Appraisal • Provision of technical assistance to develop a • Provision of Technical Assistance to develop and National Health Insurance System; implement PBF in selected districts: • Rehabilitation and equipping selected health • Training activities on: (i) concept and Activities facilities procedures, (ii) behavioral change education; • Acquisition of essential drugs, equipment, vehicles (iii) supporting health facilities in assessing their for epidemic preparedness; needs and setting up their performance goals • Training on emergency response readiness for (business plans, quality and quantity indicators). government staff; • Information, education and communications • Acquisition of hardware and software for health activities on demand generation and strategic information systems for the development of a communication on PBF; facility-based information system; • Provision of Financial Resources to health • Provision of technical assistance on health providers based on performance management • Pilot of Universal Health Care Approach starting • Manuals prepared (PBF, ACV, for health with the poor; facilities) • Rehabilitated and equipped health facilities to • Capacity building activities completed provide quality health services; • Performance based contracts signed15 districts Outputs • Storage locations fully equipped and furbished to Piloting PBF tackles epidemics; • Communication activities • Trained health staff (on emergency preparedness); • Initial development of electronic patient records; • Facility-based information systems developed, with links to UHI and PBF; • Trained staff at district and regional level (at facility and supervision level) on health management systems (for PBF and UHI). • Improved readiness for to respond to health • Increased deliveries by trained staff emergencies • Increased number of prenatal visits Intermediate • Strengthened stewardship and institutional • Increased number of vaccinated children (DTP) outcomes capacity at central level • Increased number of children receiving Vitamin A • Improved health insurance coverage/national • Increased number of health facilities ready to health insurance system, covering the poor provide an agreed set of health services • Improved Essential Health Infrastructure • Improved health system management capacity; • Improved information systems and quality of data Expected • Strengthened country’s health system • Increased utilization and improved quality of Project health and nutrition services in selected regions Outcomes Page 8 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 12. To increase the utilization and quality of the health services in the country, the project supported implementing a PBF approach. The PBF was to provide financial resources to health providers based on performance, showing improved results measured by indicators of the quantity and quality of health services. The project intended to provide technical assistance and financial support for PBF. Under the Project, the PBF component offered additional financing for health providers against the achievement of better performance. Health providers set the performance goals, based on the assessment of their own needs and on information from quantity and quality indicators, as part of a business plan. The business plan included activities to be undertaken to reach the performance goals. Health providers were to receive training and coaching. This support was to be provided by the PBF Technical Unit within the MSHP, the verification agencies and the Project Implementation Unit (PIU) at different moments of the PBF process. Verification agents would accompany this process, ensuring achievement of targets and provision of accurate and quality data on health services. Once these steps were successfully completed, and verifications carried out, financial resources (referred to as “subsidies”) were to be deposited in the health facilities accounts. These resources (additional to what the primary health care facilities had been receiving until then) were to be used to acquire necessary inputs to improve both the technical capacity and quality, as well as the patient-perceived quality of the services provided and thus increase utilization. The PBF approach was anchored in the hypothesis that a better work environment and incentives to increase utilization would motivate health staff to improve their performance, reduce absenteeism and improve patient reception and experience in the health centers. 13. To enable the implementation of the PBF, the project supported key areas to strengthen the health system. Rehabilitating infrastructure focusing on primary health care facilities was to contribute to support increased access, motivate utilization of health services, and improve both the perceived and the technical quality of services to be measured through technical quality indicators and patient satisfaction. Strengthening the health management information system was key to provide quality and timely health data, to inform policy decisions, to guide implementation of national policies, and to improve health outcomes. Improved health system management was meant to support capacity to deliver better results: upgrading patient information systems and hospital reform were thought to be key management tools to support monitoring the delivery of results. Finally, supporting the development of the UHC was to contribute to implement a policy to ensure greater access and affordability—initially for the poor, but in general for the population—of a minimum set of basic health care and nutrition services. 14. One of the desired outcomes was to strengthen the health system’s capacity to mount a timely and adequate response to the Ebola epidemic, should this be required. The project opted to work with United Nations International Children’s Emergency Fund (UNICEF) and used the same arrangements tested in neighboring countries (Liberia, Sierra Leone and Cameroon) to achieve this outcome. Expenses were to be pre-financed by UNICEF and reimbursed retroactively, to quickly acquire medical equipment, supplies, drugs, and vehicles, recondition the storage spaces to lodge these, to train staff and hire them for Ebola preparedness. These activities were to strengthen the surveillance and intervention capacities of the National Institute of Public Hygiene (INHP). Page 9 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Project Development Objectives (PDOs) 15. The PDO was to “Strengthen the health system and improve the utilization and quality of health and nutrition in selected regions” 12. Though all the project’s interventions were initially focused on a limited group of 14 selected districts, technical assistance and foundational work were to have a national scope. Key Expected Outcomes and Project Development Objective Indicators 16. Although not explicitly mentioned in the PAD, the ICRs authors identified the following key expected outcomes: (i) increased utilization of health and nutrition services, especially of maternal, neonatal and child health services; (ii) improved quality of these services; and (iii) strengthened the health system in several ways: greater capacity to deliver quality health services, enhanced capacity to collect and use information on patients; and stronger human resources. The expected long-term impact would be measured through improved health indicators (in maternal and neonatal and children’s health) 13. 17. The PAD included the following PDO-level indicators to measure achievement: Table 2- PDO and Key Intermediate Indicators Outcomes measured PDO and key intermediate indicators Increased utilization of health Per capita utilization of services and nutrition services Deliveries in a health facility by trained health personnel Severely malnourished children detected and treated Vaccination coverage: DTC Improved quality of health Average quality score for facilities covered by PBF Deliveries in a health and nutrition services facility by trained health personnel Severely malnourished children detected and treated Vaccination coverage: DTC Service readiness at health facilities Number of health facilities under PBF which used their resources to equip themselves Health personnel receiving training People who received essential health, nutrition and population services Children under 5 who receive Vitamin A supplementation Strengthened health system Deliveries in a health facility by trained health personnel for improved performance Severely malnourished children detected and treated Vaccination coverage: DTC Average quality score for facilities covered by PBF Availability of equipment and supplies at the INHP Enrollment of poor people in national health insurance (CNAM) Health personnel receiving training 12 Financing Agreement December 17, 2014. 13 Key expected outcomes were to be measured across the targeted regions, through a formal evaluation, described in Annex 7 of the PAD. The Baseline was carried out in 2017. The field work for the impact evaluation was completed in late 2019. At the time of the ICR, it was expected that the analysis and conclusions would be completed during the second half of 2020. Page 10 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Components 18. The Project included two components: 19. Component 1 – Performance-based Financing (PBF) (total estimated cost at appraisal: US$38.5 million, of which US$31.5 million IDA). This component aimed to increase the volume and the quality of health services, with a focus on maternal, neonatal and child health (MNCH) interventions, through PBF in selected regions. Specifically, it was designed to support: (i) increased utilization of MNCH-related services; (ii) improved clinical practices and health worker motivation; (iii) structural improvements; and (iv) improved management capacity, governance, monitoring and record keeping in health facilities. 20. Component 2 – Strengthening the Health System for Improved Performance (total estimated cost at appraisal: US$38.5 million IDA). This component aimed to finance areas critical for the effective operation of the health sector in general and of the PBF in particular. It included the following subcomponents: a) Improved Health Insurance Coverage (UHI) (Couverture Maladie Universelle 14): This sub-component was to finance technical assistance to support the implementation of the UHI: (a) developing criteria to identify the population living in extreme poverty; (b) assisting in the development of a costing study; (c) supporting analysis on sustainability of the UHI and the PBF approach; (d) supporting information systems linkages necessary for the UHI (included under sub-component (c) below); and (e) engaging long-term technical assistance to support the development of the UHI. b) Essential Infrastructure and Rehabilitation: This sub-component focused on the rehabilitation of selected health facilities 15, the bulk purchase of essential equipment for newly rehabilitated facilities, as well as other facilities in the project regions, and monitoring activities on the availability and use of this equipment. c) Health Management Information Systems (HMIS): This sub-component aimed to support the development of facility-based information systems linked to the UHI, the development of electronic patient records and training and development of staff to use and further develop these systems; d) Supporting Improvements in Health System Management: This sub-component included support for hospital reform and the development of a strategy for community health and other technical assistance to support strengthening the health system both at the district and region level; e) Ebola Preparedness: The Project aimed to strengthen the Government’s institutional capacity and capabilities in emergency response preparedness, with a special focus on communicable diseases, around disease surveillance, case diagnosis and management; and logistics and coordination. The Project intended to support the level of prevention activities and rapid response to an Ebola outbreak by procuring essential equipment, supplies, drugs and vehicles; f) Project Management: The Project included financing for a PIU to manage the overall project. 14 The PAD referred to this program by its French acronym, “CMU.” However, to disambiguate from references to the World Bank Country Management Unit, this ICR will use the English acronym UHI. 15 Both the RCI and other financiers were supporting the rehabilitation of health facilities since the end of the civil unrest. The Bank was supporting the RCI through the Emergency Infrastructure project. At the time, none of the financiers was replacing essential equipment. This changed during the life of the project. Page 11 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets 21. The PDOs and outcome targets were not revised. Revised PDO Indicators 22. The PDO indicators were not revised. Revised Components 23. The Components were not revised, although activities not originally envisaged where undertaken (details in next section). Other Changes 24. In September 2019, the World Bank approved a Level II restructuring revising some intermediate indicators, and reallocating funds across categories. The respective changes are detailed in tables 3, 4 and 5 below. In addition, financing of PBF support activities was extended to five additional districts, and supplemental activities not originally envisaged were undertaken. 25. Adjustments to the results framework: The Midterm Review Aide Memoire outlined the need to modify the results framework. However, since the proposed changes did not affect PDO indicators, the team decided that a formal restructuring could wait, simply reflecting the revised framework in the aide memoire. Taking advantage of the need to restructure the project to reallocate funds three months before the end of the project, the results framework was formally revised as follows: Targets for two Intermediate Results Indicators (IRIs) were revised down; one indicator with two sub- indicators was added to comply with mandated corporate indicators (see Table 3 below) and one IRI was dropped. Table 3 –Revised Intermediate Indicators and Targets Indicator Action Original Target Revised Target Children under 5 who received Vitamin A supplementation Target revised 706,350 300,000 Health facilities constructed, rehabilitated and/or equipped New target 560 97 Proportion of indigents among the total direct PBF beneficiaries Dropped 20% - People who received essential HNP services (CRI) (composite) see New indicator and target 900,381 next 4 indicators a) People who have received essential health, nutrition, and New NA 540,229 population (HNP) services - Female (RMS requirement) b) Number of children immunized New NA 126,401 c) Number of women and children who have received basic New NA 582,771 nutrition services d) Number of deliveries attended by skilled health personnel PDO Indicator 191,209 Not revised Page 12 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 26. Reallocation across categories: Resources across categories were reallocated as detailed in Tables 4 and 5. Table 4 - Changes in amount of Credit allocation to each disbursement category Category Original Revised Allocation (XDR) Allocation (XDR) (1) Performance-based payments under part A.1 of the project 7,900,000 7,040,750 (2) Goods, non-consulting services, consultant services, Training and 4,300,000 5,159,250 Operating Costs under part A.2 of the project (3) Goods, works, non-consulting services, consultant services, Training 10,800,000 11,500,000 and Operating Costs under Part B of the project, except Part B.5 of the Project (4) Goods under Part B.5 of the Project 0 0 (5) Refund of Preparation Advance 700,000 0 Total Amount 23,700,000 23,700,000 Table 5 - Changes in amount of Grant allocation to each disbursement category Category Original Revised Allocation (XDR) Allocation (XDR) (1) Performance-based payments under part A.1 of the project 5,700,000 4,906,846 2) Goods, non-consulting services, consultant’s services, Training and 3,200,000 3,993,154 Operating Cost under part A.2 of the project (3) Goods, works, non-consulting services, consultant’s services, 8,000,000 8,000,000 Training and Operating Costs under Part B of the project, except Part B.5 of the Project (4) Goods under Part B.5 of the Project 6,800,000 6,800,000 Total Amount 23,700,000 23,700,000 27. Changes in activities: Additional activities not originally envisaged included: a) Financing PBF technical assistance, accompanying and verification activities in five additional districts. IDA did not finance subsidies in these new five districts; b) WASH Subproject: An additional convention was signed with UNICEF to provide access to water for health centers. This entailed: (i) the construction of 57 points of access to water, (ii) the establishment of 17 reservoirs, (iii) connection to existing water networks and (iv) the rehabilitation of 43 latrines; c) Nutrition Subproject: Creation of 269 Nutrition Units (Unités Thérapeutiques Nutritionnelles) were in the original 14 PBF districts; d) Additional studies: (i) a study on the quality and factors affecting the use of PBF subsidies and training on financial management for 700 members of the Local Management Committees (Committee de Gestion Santé, COGES) and health staff and study tours and international workshops for the preparation of the UHI; (iii) project preparation studies on cost, human resources, and infrastructure needs, among other topics (Annex 9: a detailed list). Page 13 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) e) Purchasing of additional goods (such as tablets and other IT products) for the INHP. Rationale for Changes and Their Implication for the Original Theory of Change 28. The restructuring did not affect the original theory of change. 29. Changes to the Results Framework: During the mid-term review in February 2018, an in-depth analysis of the Results Framework (RF) was carried out and changes proposed, in order to: (i) clarify the definitions of some results indicators; and (ii) revise the targets for two Intermediate Results Indicators (IRIs), to ensure adequate monitoring and assessment of the Project’s outcomes and results (see Table 3 and Annex 1). 30. The targets for two IRIs (see Table 3 above) were revised downwards for the following reasons: (i) Vitamin A supplementation: The original target was based on the national strategy at the time of project preparation consisting of a twice a year campaign. Early in project implementation the National strategy was revised to using a facility-based supplementation. The new strategy is known to result in high coverage for one-year olds as it is linked to vaccination. Coverage is reduced between ages 1 and 5 years as the vitamin is then only provided to children once they come to health facilities when they are sick; (ii) Health facilities constructed, rehabilitated and/or equipped: After the project was approved, several financing partners came forward with resources to equip health facilities damaged by the civil conflict as well as to carry out some rehabilitations. It was then decided that the project would focus exclusively on civil works (construction and rehabilitation). The target for the indicator grouped together equipment and civil works without explicit targets for each. Additionally, at appraisal there was only an initial survey of the status of health infrastructures. Two years later, the 2016 in-depth survey of these same facilities showed that their physical condition had continued to deteriorate and was worse than foreseen. Lastly, following the post-conflict reconstruction efforts, the rehabilitation unit costs were higher than those estimated at appraisal. The indicator was reformulated as “Health facilities constructed or renovated” and the target lowered to what could be financed given the revised unit costs. 31. The reason to drop the IRI “proportion of indigents among the total PBF beneficiaries” was that data could not be gathered because of delays in the implementation of the electronic records system for patients 16. Finally, as mandated by management, the project results framework was retrofitted to include corporate results indicators (CRIs) as IRIs. 32. Reallocation of funds: As a result of the additional activities, the 2019 Level II restructuring approved the reallocation of funds between disbursement categories. The reallocation of funds was required to ensure adequate financing for PBF verification and counter verification activities in all 19 PBF districts, and for studies to prepare the follow up project. The main changes were: (i) the increase in construction and rehabilitation costs; (ii) the dilapidated status of health centers in the ten districts of the second phase; (iii) the addition of five districts to the PBF, supported by the Global Fund (which entailed higher expenses for verification and training) and (iv) the need to strengthen the financial management capacity of health actors. It is important to highlight that the implementation of the PBF required additional efforts to overcome implementation challenges, and to train and build capacity at the health centers, district and regional levels. For the credit and the grant, funds 17 were reallocated from Category 1 (Performance-based payments under Part 16 The Project was supposed to beginning of the development of the electronic patient records (EPR), which took longer than expected. The pilot will be launched during 2020. The initiative experienced delays in Cabinet decision making. The technical designed was carried out and validated by the DIIH and technical teams. There were infrastructure and technological issues related to electricity and internet connection that had to be dealt with before piloting the EPR. 17 Around US$3.3 million reallocated from Category 1 to Category 2 in the credit. Page 14 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) A.1 of the project) to Category 2 (Goods, non-consulting services, consultants’ services, training and operating costs under Part A.2 of the project). Credit funds were reallocated from Category 5 (“Refund for Preparation Advance”) to Category 3 (Goods, works, non-consulting services, consultant services, training and operating costs under part B of the project except Part B.5 – Ebola sub-component). Table 4 above shows detailed amounts. On May 2018 an amendment for the ceiling of the designated account was approved, amending that aspect of the original disbursement letter. 33. Financing PBF technical assistance, accompanying and verification activities in five additional districts. In 2017, the Global Fund (Fond Mondial-GF) partnered with the Government and IDA in supporting the implementation of the PBF in five additional districts 18. However, the GF was willing to support only the so-called subsidies for those five districts. Therefore, while the GF resources went to cover the subsidies for the five districts, IDA resources financed start-up costs including technical assistance, as well as supervision and follow-up, and verification activities. This translated into an increase in up-front investments and administrative costs under Component 1 (Category 2), required to support 19 (versus the original 14) districts and their health facilities in the implementation of the PBF (see Annex 3 for an analysis of investment and analysis of costs). 34. WASH Subproject: Many health centers in the country do not have a reliable source of water. It was decided to direct the savings realized in the Ebola subcomponent (since no cases materialized in RCI) to provide access to water for health centers. 35. Nutrition subproject: One of the PDO indicators was the detection and treatment of malnutrition. However, originally there were no specific activities for intensive support to reduce malnutrition. Once the policy of creating therapeutic centers was reintroduced, and given the relatively high level of malnutrition, it was decided to redirect some resources from Component 1 category 2 to create Nutrition Units (Unités Thérapeutiques Nutritionnelles) in the original 14 PBF districts. 36. Additional studies: The key reason for the substantial increase in the number of studies was the very long process to access the Project Preparation Advance (PPA) for the preparation of a new project. Since the new project (SPARK) intends to scale up the implementation of the PBF to rest of the districts in Cote d’Ivoire, the World Bank team agreed to use resources under Component 1 (Category 2) for most of these studies. A few other studies were added to address the need to better understand some bottlenecks regarding PBF, and the UHC deployment (Component 2) (see Annex 9 for a detailed list). II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 37. The relevance of the PDOs is considered High. The PDOs are in line with the main country policy documents: The The number of districts was initially three, but an administrative decision by the government further sub-divided them to five: Buyo, 18 Meagui, Soubre, San Pedro and Grand Bassam. Page 15 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) National Development Plan 2016-2020 19 (NDP), the National Strategy for Health Development (SNDS) and the National Performance Based Financing Strategy (NPBF). The NPBF strategy, first elaborated in 2014-2015, was revised in late 2019 to incorporate the experience gained through the implementation of the PBF pilot phase. 38. The PDOs were in line with the Country Partnership Framework (CPF) for FY16-19, in particular with two pillars: (i) strengthening governance and institutions; and (ii) renewing infrastructure and basic services. The CPF, which was assessed at mid-term, was extended until FY21 to align with the country’s electoral cycle and the NDP 2016-2020. As a result of this exercise, priorities were refocused towards two of three key pillars: (i) strengthening public finance management, transparency and institutional accountability; and (ii) building human capital for economic development and social cohesion. The PDOs continue to be in line with the extended CPF 2016-2020. 39. The PDOs were and remain in line with achieving the World Bank twin development objectives of eliminating extreme poverty and promoting shared prosperity. The Project contributed to the achievement of these goals by (i) improving access to basic health and nutrition services (essential services) for the most vulnerable populations (women and children under 5); and (ii) improving the level of financial health protection for these groups by ensuring progress towards greater access to quality care, which was to foster utilization of basic health and nutrition services, reducing the need for out-of-pocket expenses. Furthermore, the PDOs aimed to strengthen the health system’s institutional, strategic and operational capacity to prepare and combat health epidemics. In the face of COVID-19, this dimension of the PDOs remains highly relevant by completion. B. ACHIEVEMENT OF PDOs (EFFICACY) Rating: Substantial 40. The achievement of the PDOs is measured by five outcome indicators, all of which have achieved or surpassed the original targets. Table 6: Project Development Objective Indicators PDO indicators Baseline Original target Achievement Deliveries in a health facility by trained health personnel in PBF 0 191,209 217,818 districts Severely malnourished children detected and treated in health 0 16,220 19,941 facilities in PBF districts Vaccination coverage: DTC, Hep B, Hib3 in PBF districts 97% 99% 100% Average quality score for facilities covered by PBF Baseline survey 30% 35% Per capita utilization of services in PBF districts 34% 56% 58.6% Source: PBF PORTAL information system, RASS 2018 20. 41. Out of 13 intermediate results indicators, 10 were achieved or surpassed and the remaining 3 were over 90 19At the time of appraisal, the PDOs were in line with the main national sector policies, as indicated in paragraph 7. 20The 2019 RASS was delayed due to COVID19. The workshops with decentralized levels were schedule to start in June 2020. Vaccination rates surpassed the 99 percent target in 2017 and 2018, reaching 101 percent and 100 percent respectively. Page 16 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) percent achieved (see Annex 1). These outcomes benefited the population in nineteen districts covered by the project. The efforts to upgrade health facilities to meet standards and to be ready to offer quality basic health services combined with the implementation of the PBF succeeded in reaching the indicators’ targets and achieving the PDO. Assessment of Achievement of Each Objective/Outcomes Dimension 1 PDO: To Strengthen the health system for improved performance. Rating: Substantial 42. By Project completion, the country’s health system has been strengthened in terms of: (i) enhancing monitoring and surveillance capacity for epidemics; (ii) strengthening stewardship and capacity at central level; (iii) enhanced management and planning capacity in the districts under PBF; (iv) strengthened data and health information in the districts under PBF; and (v) strengthened capacity to deliver the services. PDO Outcome indicators (i.e. deliveries in health facilities by health personnel; severely malnourished children detected and treated; and vaccination coverage) show improvements in performance of the health system in the project geographic areas. Implementation of the PBF in 19 health districts (4 districts in August 2016 and 15 districts in 2017) contributed to strengthened accountability, improved human resources capacity and readiness to provide basic health and nutrition services. 43. One of the Project’s outcomes was strengthening the health system’s capacity and preparedness to respond to epidemics. The project provided critical resources to implement the already existing response plan for Ebola, as captured by the IRIs “Availability of equipment and supplies in the National Institute for Public Hygiene”, and “Health personnel receiving training”. By project completion, monitoring and surveillance capacity along the border was strengthened, thanks to the acquisition of IT equipment (tablets) for all the CSE districts and training, which allowed systematic and timely reporting of sicknesses under epidemiological surveillance. The storage capacity was strengthened with the rehabilitation of two pre-existing hangars, which allowed improved management of medications and medical response equipment. The project provided training in the use and storage of equipment and medications. By project closing, multidisciplinary teams were established to respond to health emergencies in all districts on the border. The Project also strengthened response capacity to epidemics by providing access to water to health establishments in the most vulnerable regions (along the north and western borders). 44. Strengthened stewardship role and institutional capacity at central level: By Project completion, the country had: (i) piloted results-based financing in 19 districts; (ii) established the Health Insurance Agency (CNAM); (iii) created a registry of beneficiaries, which included a registry of indigents; and (iv) piloted the UHI in three PBF districts and the reimbursement for health services for indigents. Institutional capacity was strengthened at central level, while implementing these reforms, as shown by the achievement of outcome indicators 1.1, 1.2 and 1.3. The Government adopted the PBF strategy and revised it in light of the pilot experience, acquired during implementation. It approved and revised a PBF manual and the instruments to operationalize the implementation of the PBF. Before completion, the government held a workshop in January 2020 to revise the PBF manual and incorporate the lessons from the pilot in the 19 districts. The establishment of the PBF national technical team in the MSLS under the Directorate for Health – DGS (Cellule Technique Nationale– CTN-PBF) to lead the implementation of the PBF contributed to strengthening the institutional capacity to lead and accompany the implementation of the PBF. The CTN-PBF was continuously supported by the PIU during project implementation. Despite these achievements, weaknesses remain by project closing in the Page 17 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) leadership exercised by the DGS and the clear definition of the role and responsibilities of the CTN-PBF. 45. As part of the strengthened stewardship role, in framework of the National UHI, the CNAM saw its technical capacity reinforced, with staff training and technical studies to support the piloting and management of the UHI. By Project completion, the CNAM, with the Project support, created a registry of beneficiaries for the general population, with an effort to identify the indigents in the country. In October 2019, it launched the UHI starting with the formal sector (civil servants and university students) and the poor. Over 200,000 poor individuals were enrolled by project completion in three PBF districts (as shown by the IRI “Indigents that remain enrolled in health insurance”, surpassing the 150,000 target). 46. By Project completion, the capacity of the MSHP directorates for financial management and information systems (DAF and DIIS, respectively) had been strengthened. The Annual Health Statistics Report (RASS) was produced annually for the 2016-2019 period, shortening the time period to complete it21 (IRI Annual RASS produced). The quality of the data in the report improved gradually with greater involvement of the regional and departmental levels, rendering it more relevant. The Project supported the DAF and DIIS through technical assistance, training and equipment (such as video-conferencing equipment). 47. Building planning and management capacity at decentralized level: The Project contributed to the decentralization efforts in the health sector, transforming health facilities and districts into actors with a degree of autonomy, with capacity to receive and manage financial resources. By project completion, capacity had been built within the health system at decentralized and local levels, which allowed to reach the outcomes measured by the first three PDO indicators mentioned above, as well as the new IRIs “People who have received essential health, nutrition and population services”. The implementation of PBF in 19 health districts (out of 103 districts) contributed to strengthening the health system. Thanks to training, technical assistance and close follow-up from the verification agencies, the district and regional levels, the CTN-FPB and the PIU, human resources in health centers and in the districts and regional levels were strengthened (as shown by IRI “Health personnel receiving training”). By project completion, these actors had acquired skills in financial management, planning capacity and business plan preparation. The financial capacity of personnel at the health centers was strengthened, with the opening of bank accounts, managing budgets, awareness of resources and simplified procedures for using public money. Human resource turnover limited the extent of this achievement. 48. With Project support, performance-based agreements were introduced for health sector operations for 275 service providers (403 including the ones supported by the Global Fund) and regulators in 9 regional directorates (2 of them by the Global Fund) and 19 district directorates (five of those supported by the Global Fund). By completion, there was a strategy for financing health based on performance, under implementation and expansion to the rest of the health districts; a framework (the performance agreements and business plans) in place to establish goals, with quality and quantity indicators to define achievement of results; and enhanced accountability in the use of resources and strengthened governance. Financing (known as “subsidies”) was received after providers fulfilled their performance commitments. Performance agreements strengthened governance at the decentralized levels of the health system. 21 The RASS was prepared within a period of six months after the end of the year, with the Project’s support. Page 18 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 49. During the ICR mission, it was observed that once the Regional Health Directorates (Direction Régionale de Santé, DRS) and, the District Health Directorates (Direction Départemental de Santé, DDS) and the health centers were working under PBF, the performance and results-oriented work mentality continued, even when people moved to a different region in a different capacity than the one they had when they worked under PBF. It was observed that the significant training received under PBF trickled through the health system. Another observation during the ICR mission was the acknowledgement of greater transparency in the use of funds: the MEF and the Ministry of Budget affirmed that, during the PBF pilots, evidence of results against resources was shown and it contributed to the effectiveness of the spending. Among the achievements, it is worth mentioning the simplification of procedures for the use of public resources, which was supported by the MEF by creating the figure of the “regisseur”, an official of the MEF who supervised and accompanied the health establishments as they experimented with a minimum degree of financial autonomy, after being granted the status to open bank accounts in their names. 50. The health system was strengthened by the improvements on the quality of the data in the PBF districts. The quality of health data improved at health centers, district and regional level. These data are the source for the RF updated achievements, shown in the ICR. By project completion, registries at health facilities were properly completed, as they were key to show performance and receive resources. Improved quality data allowed the establishment of the PBF information system and also supported improvements in the national health information system. While the information tools existed, they were not completed properly. The involvement and level of engagement of the service providers, verification agencies and district levels were significant, showing ownership of the data as well as understanding of the importance of generating quality data. The quality data was used to guide decision making and to monitor business plans and performance contracts. The use of reliable data has transformed the way regional and district levels work. For 2017 and 2018 the regions presented health data quarterly. This reflected greater involvement in the data preparation and in its utilization. The regions prepared action plans (along the lines of performance contracts) to improve health indicators and health outcomes. By project completion, norms and regulations were successfully implemented. Before, the regulations and standards were there but were not clearly implemented. During the ICR mission interviews, the improvement of the quality of health data was consistently considered one of the main achievements supported by the Project. 51. By project completion, the management capacity to improve the quality of the health services had been strengthened in PBF districts: Staff in health centers felt empowered to take initiative, make changes and take decisions on the actual work done in their health centers. This fostered creativity and enhanced resourcefulness to find solutions to problems. There were numerous examples evidenced during the ICR mission, as well as other documented best practices. The impact was felt in the transformation from passivity to action. Knowing that the opportunity existed, capacity was created to plan, implement, monitor and evaluate, motivated by accountability and incentives. 52. The physical infrastructure of the health sector was strengthened with the direct rehabilitation of 80 health establishments, 91 health facilities refurbished to provide nutrition services, and upgrades to enable access to water through the construction of hand-propelled water pumps and connections to city water systems. This supported increased access and utilization of health services and improved capacity to meet the health challenges the centers faced. Moreover, a total of 265 health facilities dedicated resources to upgrade their physical infrastructure using their PBF resources. This was perceived as a critical achievement to improve performance at local level. Page 19 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) PDO 2: To Improve the utilization of health and nutrition services in selected regions. Rating: Substantial 53. The utilization of health and nutrition services increased in all of the 14 health districts under PBF: for the 2016 – 2019 period for the first four districts and from 2017 -2019 for the next 15 districts incorporated to the PBF program. The number of consultations increased across the board for urban and rural health centers and for general and regional hospitals, in the first four districts and in the following 15 districts (see Annex VII for an explanation of seasonal peaks observed in the PBF districts). Graph 1: Rate of Utilization of health services – First Four Districts under PBF 70% 59% 59% 60% 50% 52% 52% 47% 47% 50% 42% 42% 44% 43% 37% 40% 32% 25% 30% 20% 10% 0% T3-2016 T4-2016 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T4-2019 2016 2017 2018 2019 ANYAMA COCODY-BINGERVILLE BOUAFLE SINFRA Source: PRSSE. PBF portal. 2020 Graph 2: Utilization of health services – Second Group of Ten Districts under PBF 120% 100% 90% 90% 79% 82% 74% 77% 74% 80% 72% 69% 62% 64% 60% 40% 20% 20% 0% T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T4-2019 2017 2018 2019 BOUNA DABAKALA ISSIA MBAHIAKRO NASSIAN NIAKARAMADOUGOU PRIKRO TIEBISSOU VAVOUA YAMOUSSOUKRO Source: PRSSE. PBF portal. 2020. Page 20 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Table 7 – Annual Growth Rate 2016-17 2017-18 2018-19 Four Districts 2,4% -0,3% 7,7% Ten Districts 1,0% 5,6% 54. The outcome indicators showed achievement of the targets for all of the outcome indicators and most of the intermediate indicators (see Annexes 1 and 7): the number of deliveries in health centers by trained staff by districts reached 217,818 (133 percent over the target); the improvement in the detection and treatment of malnutrition (19,941 children, representing 122 percent of the target) and the level of vaccinations in PBF districts (between 98 percent and 100 percent of target throughout implementation); the regular supplementation of Vitamin A, which health centers took on as a regular activity rather than via annual campaigns. This strategy, together with the implementation of PBF, resulted in reaching 361,042 children under 5 years old in two years of the project, surpassing the -target (120 percent). Targets were surpassed for the number of women using family planning methods and those receiving pre-natal care. The target for the indicator on the number of people receiving a basic health package of health, nutrition and reproductive health services was surpassed as well (132 percent of target). The number of children immunized reached 192,165 (well over the 126,401 target). 55. The implementation of the PBF, supported by the project, greatly contributed to the increase in utilization of basic health and nutrition services. The PBF resources were used to improve the physical appearance of health centers, in particular maternity areas, and ensure access to water, medications and equipment. The PBF resources allowed the health facilities to reach out to their communities, providing resources and flexibility to come up with creative options to attract patients to health centers and to bring healthcare to remote communities as well. 56. Health centers took initiative and got creative, coming up with clever approaches well-tailored to their communities, to meet established goals and achieve quantitative targets. With monetary resources available to them, health centers developed strategies to reach out to communities, fostering prenatal visits and institutional deliveries. Some of the innovative initiatives included: hiring transportation to bring pregnant women close to labor to health facilities, the construction of toilets for women in the maternity area and offering baby kits if the mothers deliver in health facilities. Some health providers provided for free the “carnets des consultations” for pregnant women who were coming to prenatal consultations, eliminating financial barriers. They also used the PBF resources to implement activities to provide mobile health services and go out to more remote communities (visiting and providing services). This strategy to reach out to populations was successful. The health centers sought to reinforce their basic technical capabilities as well, to be ready to provide health services, using PBF resources for the acquisition of equipment and medications as needed, focusing on maternal and child health. All this greatly improved the utilization of service. PDO 3: To Improved quality of health and nutrition services in selected regions Rating: Substantial 57. The quality of the health and nutrition services improved by project completion for the group of districts under PBF. Improvement in quality was measured by a set of indicators defined in the PBF Manual (annex 5 of the manual) for primary health care centers and hospitals. The quality score includes general indicators (regarding registries, business plan and financial management), indicators for vaccinations, prenatal care, nutrition, diarrhea, malaria, IRA and Page 21 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) pharmacy; and indicators on some dimensions of quality such as hygiene, security and environment. As shown in the tables below, the quality score for health districts under PBF experienced an improvement between 2016 and 2019. It worsened during the period of integration to the PBF of the 15 additional districts (10 supported by the project and 5 by the Global Fund). Annex 7 includes additional tables showing the achievements in each of the PBF districts. Graph 3- Score technical quality for all the PF districts (%) 78.9 80.1 82.9 73.5 76.5 62.9 62.7 67.0 60.0 47.8 53.0 52.5 44.6 Integration of new districts T3-2016 T4-2016 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 2016 2017 2018 2019 Graphs 4 and 5- Quality Score by Type of Facility Graph 4 - Evolution of average quality Graph 5 - Evolution average quality score score in Primary Health Facilities (%) in reference hospitals (%) 77.6 80.4 81.9 85.2 85.4 86.9 86.8 78.4 71.7 80.5 76.5 80.5 83.8 85.0 83.1 79.7 63.1 67.6 70.9 70.8 74.2 59.3 72.2 62.7 47.9 82.0 45.7 72.1 75.077.8 79.0 70.7 72.574.6 57.2 59.7 64.4 50.8 52.1 55.9 62.3 66.7 44.2 39.0 48.4 46.6 38.3 48.9 T3-2016 T4-2016 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T3-2016 T4-2016 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 2016 2017 2018 2019 2016 2017 2018 2019 04 districts 15 autres districts 04 districts 15 autres districts The tables below show the achievements in improvement of a limited number of areas regarding the technical quality of health and nutrition services in the health districts under PBF for health centers and hospitals (general and regional). Annex 7 includes graphs for each group of districts of additional dimensions of technical quality. Page 22 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Graph 6- Evolution of Some dimensions of Quality in the ESPC 100.0 88.3 88.7 87.2 90.6 92.6 83.0 80.6 83.5 80.6 84.2 74.6 74.5 77.6 80.0 60.0 40.0 20.0 0.0 T3-2016 T4-2016 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 2016 2017 2018 2019 Hygiène, Sécurité, Environnement Maternité Prise en charge du Paludisme Disponibilité des Médicaments Traceurs Pharmacie Graph 7: Evolution of some dimensions of quality in Hospitals (%) 100.0 90.0 80.0 70.0 60.0 50.0 40.0 30.0 20.0 10.0 0.0 T3-2016 T4-2016 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 2016 2017 2018 2019 Hygiène, Sécurité, Environnement Hospitalisation Bloc Opératoire Maternité Prise en charge des urgences Disponibilité des Médicaments Traceurs 58. The Project supported carrying out the Service Availability and Readiness Assessment (SARA) Survey in 2016. The survey provided timely and relevant information on the status of availability and operational capacity of health establishments to provide quality basic health and nutrition services, information not available at the health information system. It also provided an assessment of the quality of the data generated by health establishments. The SARA survey was crucial in supporting the activities to achieve the PNS 2016-2020 objectives. The quality score was prepared with the information from the PBF information system, since the SARA survey was not continued. Page 23 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 59. The implementation of the PBF fostered improvements in the quality of the provision of health services. PBF resources were dedicated to bringing health facilities up to a specified level. Around 265 health facilities used the resources to acquire equipment, improve infrastructure, and for other activities that allowed them to better provide health services. The improvements in infrastructure and equipment transformed the working environment at the health establishments. Health personnel were more motivated, enabled to provide better quality of services in a cleaner environment, with better access to water and sanitation. During the ICR mission, reports consistently emerged of the significant impact the improvements in infrastructure and equipment had in the perceived and actual quality of health services. Other aspects, such as access to water and water for the maternity areas of health centers, had an analogous impact on women’s perceptions of quality in the provision of services. 60. Patient satisfaction with the quality of care also improved in the health establishments. Community Based Organizations (OBC, in French) were functioning by project completion. They reached out to the patients, verified the services were provided and checked with beneficiaries on their level of satisfaction during their visits to health facilities. During the ICR mission, anecdotal information was gathered from beneficiaries on how their views and experiences had been taken into consideration by health providers (see Annex 10). 61. Staff received training in their specific technical area (midwives, nurses, social workers), which contributed to better performance of services. Training also covered key aspects of PBF, such as health data gathering, which pushed staff to ensure that all the steps in the health visits were covered. PBF contributed to compliance with existing norms and procedures, which were not previously fulfilled, improving enforcement of standards and improving the quality of service delivery. The PBF favoured compliance with and use of protocols and directives issued by the MSLS. The set of quality indicators supported the implementation of existing protocols and directives, as they required more detail and specifications on the health services provided. C. EFFICIENCY Assessment of Efficiency and Rating Rating: Substantial 62. Project resources were used efficiently. Global benchmarks suggest PBF investment needs to be roughly US$3 per capita per year and requires longer implementation time, to produce measurable health impacts. This project, explicitly designed as a pilot, invested substantially less than that (US$0.70 per capita), as it aimed at leveraging existing resources and so it was not expected to generate measurable health impacts by completion. However, this modest investment did lead to measurable improvements in both the usage and quality of services. The literature suggests that, once a PBF program is established, roughly 30 percent goes to operating costs other than performance payments. 63. PBF facilities achieved greater technical efficiency through an increase in inputs, which in turn increased health workers’ motivation. This stood in contrast to non-PBF facilities, where providers highlighted a lack of inputs and poor infrastructure as constraining and demotivating factors 22. PBF achieved technical efficiency gains by inducing healthcare providers to focus on the country’s priorities. Data showed that 78 percent of PBF funds targeted high-impact interventions identified in the country’s investment case, which was designed to improve reproductive, maternal, 22 https://dash.harvard.edu/bitstream/handle/1/40976810/DURAN-DISSERTATION-2019.pdf?sequence=6&isAllowed=y Page 24 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) neonatal, child and adolescent health and nutrition (RMNCAH) outcomes. 64. There are also signs of improved allocative efficiency, as 84 percent of total PBF spending occurred at primary health facilities while only 11 percent was at secondary hospitals and the remaining 5 percent supported regulators. 65. As is to be expected in a pilot program (small scale and short time), the ratio of operational costs to performance payments was higher than 30 percent. In the IDA-financed districts, on average almost 40 percent of the IDA-financed PBF resources were operating costs (not counting upfront launch and investment costs). Despite this, as the number of districts (and the population covered by the PBF) expanded and service utilization increased, the ratio improved, showing a downward trend for operating costs. In the IDA and Global Fund-supported districts operating costs were about 42.5 percent, reflecting the smaller scale (five districts) and shorted implementation time of the PBF scheme. 66. Despite the relatively high operating costs, the substantial impact of the program—as evidenced by the significant improvement in the quality of care, increased service utilization, and increased allocative and technical efficiency—demonstrate the substantial general efficiency of the PBF program. 67. What is more, the project financed additional activities not initially contemplated. This was possible for two reasons. First, the project made efficient use of resources, notably in the case of the savings realized under the Ebola subcomponent. UNICEF’s centralized procurement allowed for savings on medications and equipment making additional acquisitions and the WASH project feasible. Second, the project supported capacity to respond to needs arising during implementation, to achieve the PDOs. This was the case with the establishment of the nutrition units in 269 primary health care centers, the financial management training of 700 individuals in health facilities and the COGES and the support for the SARA survey (see Annex 1.b for a list of these activities). 68. The Project was implemented within the period foreseen in the PAD, committing all the grant and virtually all the credit resources. The difference in the disbursement projections in the Operations Portal between the original Grant and Credit amounts (in XDR) and the final disbursed amounts (in US$) are due to variations in the exchange rate. At the end of the project, a total of US$1.7 million remained undisbursed because the amount was insufficient to cover certain activities, and it was decided to use the new project (SPARK) resources to cover those activities. 69. The pace of implementation improved during the life of the project, as operational challenges were overcome and planning improved (see factors affecting implementation). The limited delays in implementing project activities, such as the rehabilitation of infrastructure and the PBF interventions (see Annex 4), they were justified by the need to build capacity and carry out a sound analysis of the state of infrastructure to plan and act accordingly. Within Component 1, the share of resources allocated to activities to accompanying the implementation of the PBF (other than subsides) was higher than anticipated (see section 1.B and Annex 3): Part of those resources financed the cost of the supervision and verification activities in the additional five districts as well as to support the capacity building and accompanying activities that were key to the successful implementation of the PBF component. Recurrent costs represented around 10 percent of these resources (supervision, coordination meetings, quality evaluation). 70. Benefits from the Project interventions. An element to consider in terms of costs and benefits is the improvements in the score to measure improvements in the quality of health service provision, in particular regarding: (i) hygiene, security and environmental aspects; (ii) technical upgrades to the operating rooms and the maternity blocks; Page 25 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) (iii) admissions and hospitalizations; and (iv) availability of medications. It is safe to assume that the improvements in the quality score (measured as a percentage), while not considered at appraisal, brought benefits to the communities where they took place. These benefits were likely translated into increased utilization of health services and improved health outcomes. D. JUSTIFICATION OF OVERALL OUTCOME RATING Relevance of PDOs High Efficacy Substantial Efficiency Substantial Overall Outcome Satisfactory E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 71. The project focused on neonatal and maternal health and nutrition, monitoring indicators to explicitly account for the outcome of the project activities on women and children. Reducing severe and acute child malnutrition was a priority for the country, which had a nutrition strategy elaborated in 2001 as well as some protocols. The Project supported the implementation of the nutrition strategy, enhancing the impact of ongoing activities. With the Project’s support, the annual nutritional supplementation vitamin A campaigns supported by UNICEF 23 in PBF districts was transformed into a regular health service provided in health centers, to address the population’s need. The project allocated resources to the establishment of 269 therapeutic nutrition units in 14 PBF health districts in primary health care facilities and general hospitals. This activity was critical in the implementation of the already existing national nutrition policy. The project supported training of health professionals, reproduction of materials (educational and demonstration materials), coaching and supervision. While this activity did not foresee the refurbishing of physical space for the nutritional units, health providers allocated PBF funds to upgrade the spaces with the Project’s resources. Health establishments implemented strategies which successfully reached out to women, educating them in using prenatal care in public primary health care centers and family planning. The PBF resources allowed this additional effort to increase the technical capabilities and to meet the needs of the health facilities and those of the population they were serving. 72. The project supported the country’s efforts to improve the quality of health services for women. The project monitored progress and achievements in access and utilization of health services (family planning, timely prenatal consultations, deliveries in health upgraded facilities attended by trained personnel, coaching and educational support on nutrition). Health providers prioritized improving maternity areas in the health facilities, introducing areas to wash and toilets, to meet the needs for women’s privacy and thereby encourage use of health services. Providers under PBF used resources to meet goals that put women at the center of the health outcomes improvements. During the ICR mission, examples emerged of creative and innovative initiatives to reach out to women who lived in remote areas (such as hiring transportation to bring pregnant women to the health facilities, incentives for community workers who regularly go to the farthest away enclaves to be the “ambassadors” for the rural and urban health centers). 23UNICEF distributed stocks of vitamin A provided by donors, namely Canada. A workshop was held annually to decide the distribution of the stock Page 26 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Institutional Strengthening 73. The Project explicitly supported institutional strengthening across the health system as part of its PDOs. Significant achievements were made with the PBF information system. Training of human resources on PBF, on epidemiology activities, and on technical aspects of the UHI (costing, sustainability, comparative implementation experiences) were supported by the Project. The PBF piloting in 19 regions brought structural and operational transformations to the different levels of government. PBF transformed the methodology of work to one based on performance, clear goals, indicators to monitor, accountability, and efficient use of resources. This started at provider level, reaching districts and regional departments. It changed the demand and the supply side of provision of health services. The Project strengthened the operational capacity of regions and districts: it supported the revision of district and regional profiles, defined the conditions for their functioning, developed a training strategy and designed an operational plan that will be implemented under the follow project SPARK. Mobilizing Private Sector Financing 74. During the ICR mission 24, it was observed that the municipal government and a private company had provided support to the health facilities with infrastructure improvements. This support had come after the health providers had been working under the PBF and results in terms of quantity and quality were tangible in their communities. It is likely that similar support has taken place in other regions and districts. Poverty Reduction and Shared Prosperity 75. The Project’s interventions benefited the poorest and more vulnerable population as described in the efficacy section. The piloting of the national health insurance scheme directly benefitted a larger than targeted group of 200,000 very poor individuals (see paragraph 40). The project supported the establishment of a beneficiary registry for the general population with an emphasis on the poor. The achievement of the PDOs likely contributed to the reduction in out of pocket expenses for receiving health services for poor families, thanks to greater availability of and access to quality health services. It is expected that increased utilization of health and nutrition services contributed to better health outcomes (see Annex 1 C for updated health indicators at completion). Other Unintended Outcomes and Impacts NA III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 76. Good coordination with other development partners. During preparation and before effectiveness, IDA and the government coordinated with other actors working on the health sector in the country, to build on each other’s efforts, 24 The mission visited the regions of Marahoue and Haut Sassandra. Page 27 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) avoiding duplications. For instance, in January 2015, IDA accepted the French Development Agency (Agence Française de Développment, AFD’s) offer to use their architect and bio-medical engineer to work on the detailed design of the infrastructure and needs of basic equipment for the rehabilitation of health centers to be carried out under the Project. There was donor coordination during preparation on the geographic areas of focus for the project. Another example was the coordination with AFD, European Union (EU), African Development Bank, UNICEF and World Health Organization (WHO) on the Ebola preparedness activities. Finally, the national strategy for PBF was prepared and validated with national stakeholders and with other financial partners (UNICEF, United Nations Population Funds (UNFPA) and Global Fund). 77. Ebola epidemic during 2014. As the Ebola epidemic emerged during project preparation, the Government decided to allocate US$10 million as a project sub-component, to reinforce capacity to tackle epidemics. The main issues before effectiveness were logistical planning to receive equipment and supplies and storage for the goods procured. Activities under this sub-component were implemented expeditiously, thanks to the very efficient procurement process used by UNICEF and the experience accumulated in dealing with the epidemic in neighboring countries. 78. Difficulties in speedy use of the PPA. There were some difficulties with the use of funds of the PPA; these funds were to finance preparatory work for UHI and the pilot of the PBF. There were delays in opening the project’s account and in submitting the payment request for these funds. This caused delays in the initiation of the implementation activities. While the government and the World Bank worked to move this process forward, the studies planned to be carried out during this period , had to be carried out once implementation started (studies on costing of basic health services and fiscal space for the UHI, study on technical specifications for the rehabilitation and equipment of health establishments). 79. Strong Government commitment. The Government was strongly committed to the Project’s PDOs and the Project’s activities. Institutional arrangements were put into place during preparation. At central level, a Project Steering Committee was established in May 2014 to guide the overall preparation. In July 2014 a technical committee was created by the Minister, to guide the preparation as well. The Government gave both a physical space and US$1 million to begin preparations for creating the UHI. All legislative steps were taken during 2014 to start UHI implementation in 2015. B. KEY FACTORS DURING IMPLEMENTATION 80. Effectiveness was timely, but the project’s takeoff was slow: The initial set of activities for takeoff took time: hiring the FM and monitoring and evaluation specialists, completing the project operational manual, the PBF manual and setting up the committees to manage and supervise the project (Comité de Pilotage du Projet; Comité Technique du Projet; Comité Pilotage PBF; PIU and Cellule Technique National du PBF). There were delays in meeting the disbursement conditions and some deadlines to complete activities were pushed back in time (for the preparation of the Operational Manual for the Project and Manual for PBF). This was normal given the complexity of the new program and the novelty of the interventions, all of which all required more time than expected. The institutional arrangements required several committees to be established by decree, which took some time as well. Resources for these take-off activities were not initially available. The PPA funds were not available, as they were not requested before effectiveness. Meeting disbursement conditions for categories 1-3 took time. There were delays in opening bank accounts for the Project. This translated in delays for launching several studies (on cost, fiscal space, health services, technical specifications for the rehabilitation and equipment of the health centers). All of these activities were delayed until the resources could be disbursed. The Government and the World Bank collaborated to put in place the necessary elements for implementation Page 28 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) during this period. 81. Mobilizing the counterpart funds was a challenge. The disbursement of the counterpart contribution from national budget resources, destined to cover part of the PBF subsidies disbursements, experienced significant delays. While the commitment to the PBF and the project were firm, delays persisted until closing. Budgetary practices also impeded carrying out activities in a timely manner. This was due to the SIGFIP (Système Intégré de Gestion de Finances Publiques), which closes during the November – January period. Implementation of some activities was delayed if resources were not included in the budget, by the Ministry of Budget, despite activities being in the project’s annual PTBAP. Nevertheless, efforts were regularly applied to ensure access to both national, credit and grant resources. This was important due to the pari-passu for component 1, which resulted in delaying the payment of subsidies to the health providers. 82. Use of PBF resources took time: Operational challenges that appeared during the early phases of the PBF implementation were gradually surmounted. Greater support from the CTN-PBF, the PIU and the ACVs managed to unblock bottlenecks to use the subsidies that appeared during early implementation. A study carried out in September 2018 on a sample of health facilities revealed that the main difficulties in using the PBF funds were related to the internal organization of the health facilities and the lack of functionality of COGES, the skills and knowledge of teams in the health facilities, and the insufficient support to the health facilities by the regulators (DDS and DRS). A major cause was the difficulty in preparing a realistic business plan. Another difficulty lay in the delays in the disbursement of the subsidies in the health facilities’ accounts, creating confusion over the period they were being paid. Support to health facilities through training and accompanying was key to overcome the operational difficulties in the use of available funds. The ACVs and districts provided extensive support in the preparation of the business plan. The support, when given, was highly rated by the health facilities. 83. Operational difficulties in the framework of the PBF: There were difficulties to change procedures and align them with the PBF procedures (such as management tools for quality and interpretation of the indicators). There were challenges to master the tools of data collection, and issues of ownership of the process and the changes to the health service providers and decentralized level regulators. Other operational challenges worth mentioning were the delays in signing the Contracting and Verification Agency (Agence de Contractualisation and Vérification, ACV) contracts to hire them, delays in their deployment, difficulties in organizing the community participation, and difficulties in mastering the preparation of the business plan in the time required by the PBF manual. The delays in the payment of subsidies has been an operational element that improved over time but had not been totally overcome—in particular, the difficulty related to mobilizing government resources. The community verification was taking place by project completion, but the information module was not up into the PBF information system. Therefore, there was no data on community verification at project closing. 84. Weaknesses in COGES functioning. Many COGES were not functional and were not able to play the role envisioned for them in the PAD. During implementation, functional COGES participated in the validation of the business plan but seldom in the preparation. Nevertheless, efforts to involve the COGES were applied during implementation by the decentralized levels, with support from the PIU and the World Bank. 85. Institutional challenges were present throughout implementation. Page 29 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) a) There was frequent turnover of leadership at the National Technical Committee for Performance-Based Financing (CTN-FBR) and the DGS. The CTN had five coordinators in four years. The limited tenure of coordinators and, in some instances, the interim character of their appointment were limiting factors for the CTN. This weakened the CTN’s leadership role for implementing the PBF strategy; b) The institutional anchoring of the CTN was constrained when it was placed under the General Directorate of Planning and Forecasting (DPPS). The MTR recommended to place it under the DGS, to enable the CTN to carry out the regulatory role for the PFB foreseen by the PBF strategy; c) There were challenges in smoothly coordinating the roles and responsibilities of the CTN and the PIU during implementation. The CTN did not directly manage the PBF subsidies, which was a source of tension. The implementation of the institutional arrangements for the PBF added to the operational challenges of implementing the PBF for the first time. This caused delays in overcoming operational challenges (such as delays in sending reports, payments of subsidies, delays in hiring the ACVs). A decree was approved on July 2018 to better clarify the roles and responsibilities of the different institutions working to implement the PBF. This decree was key to give the CTN the institutional framework of work; d) There was high turnover of District and Regional directors, which undermined their role as regulators, supervisors, and hand-holders of the health facilities. With the frequent rotations, the changes created disruptions for the districts in question. On the positive side, the DDS or DRS who moved to a new region which might have not been under PBF, brought with him or her the PBF way of working; e) The PIU was transformed into the Coordinating Unit for Health Projects, in May 2018 (Decree N.008211), to expand the previous PIU responsibilities, enabling it to manage all World Bank health projects and other international sources of health financing. 86. Infrastructure interventions changed during implementation. Several factors affected activities under this subcomponent: (i) some donors supported rehabilitation and equipping of health facilities; (ii) an in-depth study on the state of health infrastructure was carried out during implementation which complemented and updated the information available at appraisal. The study showed that some facilities had continued to deteriorate since appraisal, many of which had not been upgraded since their construction; and (iii) the construction costs were higher than anticipated, as these increased in the years post-conflict. Given the limited amount of resources under the Project, a choice was made to focus the available resources on quality of primary health care centers, in line with the sector decentralization efforts. 87. The Project financed some activities important for the SPARK project: such as implementing the SONU network (which was a key issue for launching SPARK activities), the project supported the training on SONU on health facilities (on the cartographic tool). While SPARK was already effective, not all disbursement arrangements were functioning and there was urgency in financing these training workshops. 88. Procurement required support and training: some operational difficulties arose during implementation from a lack of familiarity with IDA procurement procedures and a lack of previous experience by the executing agencies in elaborating terms of reference and technical specifications. Other challenges related to hiring consultants, experts in health, inaccurate estimates of costs of certain activities from beneficiaries, and the delays in the approval of contracts due to the inclusion in the national financial management system (SIGFIP and SIGMA). Page 30 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 89. The Project included two elements for M&E: routine M&E and an impact evaluation. Routine monitoring and evaluation were based on the existing HMIS to ensure timely and ongoing monitoring. As part of the project’s activities, the project foresaw strengthening the existing HMIS, to reinforce the country’s information and data collection systems. The PBF pilot was to reinforce the HMIS through verification and counter-verification of key indicators. The RF was adequately designed and placed emphasis on intermediate outcomes, focusing on accountability in the delivery of maternal and child health services. It included indicators that were regularly collected by the national data collection systems. An impact evaluation of the pilot PBF was to assess how the pilot had contributed to the overall objectives and to guide scaling up of the PBF scheme. M&E Implementation 90. Project monitoring and evaluation was carried out regularly. The RF was updated in a timely manner, contributing to the assessment of the project’s progress and reported in the ISRs. Baselines were collected for those indicators that still required them, once the project was launched. The PIU and the World Bank worked closely to ensure an effective use of the Project’s M&E framework. Most importantly, the national HMIS was fundamentally strengthened through project activities. Thanks to the Project’s support, the Ministry was able to launch and maintain National Annual Statistics Report (Rapport Annuel Statistiques de Santé) or RASS, which had been discontinued for several years. 91. As explained in paragraph 25, during the MTR, an in-depth analysis of all indicators was carried out. The findings of this joint analysis guided the required adjustments to the RF to better monitor the project. They also served to adapt the project activities to better achieve the PDOs 25. As mentioned in paragraph 25, while the RF was formally revised in late 2019 when reallocations of Project resources were needed, the World Bank used the updated RF as agreed during the MTR to monitor implementation progress. The quality of the monitoring data was good and improved throughout implementation. The project supported the creation of a PBF portal (https://www.fbpcotedivoire.org/) with information on quality and quantity indicators for health providers, publications and access to the DHIS2 to explore the PBF data. The only module that remained to be incorporated into the portal was the module to capture the Community- Based Organization verification activities and the analysis of the work carried out. The PBF portal contains quantity and quality indicators and the HMIS contains only quantity. But the implementation of the PBF portal has informed the HMIS and has contributed to quality improvements for the HMIS data. The PBF contributed to the quality of the data, which had been verified and validated 26. 25 That was the case of the support for the UTNs, the use of funds for the WASH project, to set priorities for rehabilitation of health infrastructure, to find solutions for operational obstacles along the implementation of the PBF program, to mention a few examples. 26 The Annual Health Statistics Report (RAAS) was prepared every year since 2016. During this period, the regions and districts became more involved in the validation of health data, contributing to greater quality and to increase the focus on their role in generating quality health data. The DIIM has carried out pilots to integrate the PBF and the HMIS data in the UCH and these pilots went well. All of Page 31 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 92. The impact evaluation was not carried out before project completion. The baseline study was done in 2016 for the first phase districts (Cocody Bingerville, Anyama, Bouafle and Sinfra) and in 2017 for the second phase districts participating in the RBF (Yamoussoukro, Tiebisso, M’Bahiakro, Pikro, Bouna, Nassian, Dabakala, Niakara, Vavoua, Issia). These districts received infrastructure rehabilitation, equipment, participation in a UHI pilot for the poorest and a pilot of autonomous management of medication at the health centers. The baseline was carried out on a limited number of women, a shortcoming at the time. The impact evaluation includes 11 districts that received rehabilitation of infrastructures and 13 other districts did not receive any program or pilot. Data gathering took place in late 2019 and the data were being analyzed at Project closing. The impact evaluation is expected to be completed by the end of 2020, given the delays connected with COVID-19. M&E Utilization 93. Health establishments used PBF data on quantity and quality indicators to prepare their business plans, to update them and to make decisions on the allocation of the PBF resources. Data was used at the district level by those districts under PBF, for their business plans as well and to guide and fulfill the commitments under their performance agreements. The M&E information on the project performance guided decisions such as the establishment of the UTN in PBF districts to implement the already existing National Nutrition Policy, contributing to the achievement of the PDOs. It supported the policy dialogue with the Government on the next steps to support the country sectoral priorities. Namely, the decision to expand the PBF program to the rest of the country, in the context of strategic purchasing and support the implementation of the UHI. Justification of Overall Rating of Quality of M&E Rating: Substantial B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 94. Environmental: The project supported the Ministry of Health (in particular, the Directorate for Public Hygiene and Environmental Health) in the elaboration of a plan to update the national plan for management of health waste for the period 2016-2020. This Plan National de Gestion de Déchets Sanitaires (PNGDS) was part of the National Development Plan 2016-2020. The PNGDS built on the previous plan for the 2009-2011 and on previous lessons. The Environmental and Social Management Framework (ESMF) for the Project was elaborated and launched in October 2015. 95. During the project implementation phase, activities were implemented in compliance with the national legislation and safeguards policies triggered by the project. The environmental and social screening of activities was systematized, and the environmental and social monitoring report was regularly produced. 96. The implementation of both the ESMF and the PNGDS faced challenges during the life of the Project. During the infrastructure rehabilitation activities, substantial supervision was required to ensure that construction firms this contributed to better quality of data to monitor the progress towards the PDOs. The RAAS had been published up to 2011. Page 32 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) incorporated and followed safeguards requirements. The supervision teams (integrated by the PIU specialists with support from the Bank team) carried out intense field work to ensure compliance with the ESMF and the PNGDS, regarding hospital incinerators, rehabilitations of health centers and construction of latrines and water points. 97. Fiduciary compliance: There were no significant issues with the Project’s financial management and the audit reports were clean. There were difficulties with the availability of counterpart funds as mentioned above, which had not been resolved by project completion. There were issues with the transition to national financial management system SIGFIP, which meant additional complications for paying invoices during implementation. 98. Procurement. Procurement activities were satisfactory throughout implementation. The Procurement Plan was regularly revised to reflect how project activities adapted, to better meet the PDOs. The Bank regularly carried out ex- post reviews of the procurement activities and provided training and support to resolve issues that arose. There were some difficulties after the project transitioned from SIGFIP and entered the SIGMAP. A focal point for SIGMAP was appointed, which contributed to overcoming them. The establishment of the Systematic Tracking of Exchanges in Procurement (STEP) system created difficulties as well, which were in the process of being resolved by completion. There were difficulties making the transition on some contracts that had started during the Project and were to be continued during the follow up project SPARK 27. C. BANK PERFORMANCE Quality at Entry Rating: Satisfactory. 99. International knowledge and in-country policies and experience were considered in project design. Project design incorporated lessons from other PBF projects, such as those related to the synergies between PBF and health systems strengthening. It also considered in-country PBF experiences (see paragraph 10) 28 , in particular regarding the lack of management autonomy, which prevented payments from getting to health facilities. Design ensured the project’s strategic relevance and approach: PDOs and components were aligned with the country’s sectoral policies. Project design opted for a selection of proven strategies for maternal and child health outcomes. It ensured the inclusion of poverty, gender and social development aspects: project activities focused on and monitored poor and vulnerable populations (women, children, the poorest). 100. Risk assessment was extensive and realistic, with an overall rating of Substantial. The PAD rightly deemed implementing capacity and governance risks as substantial, as well as project risks in light of the newness of PBF and UHI and the steep learning curve. Given this assessment, extensive and continuous dialogue and technical assistance were contemplated among the project’s activities. The PAD included mitigation measures such as training of and supporting staff in health establishments by the CTN-PBF, the ACVs and the PIU as well as technical assistance by a World Bank team of experts. The World Bank provided support to ensure that the implementation arrangements would 27 These were: the contracts for the 5 verification agencies; the recruitment of an agency for counter-verification of PBF data; hiring a firm to carry out the impact evaluation of PBF; hiring a communication agency to implement the communication strategy. 28 EGPAF and ABT pilots included lessons learned. Page 33 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) function adequately. Significant attention was dedicated to a quality design of the M&E to support project implementation and assessment of outcomes. 101. Preparation was highly participatory. Preparation entailed extensive consultations with key stakeholders, several ministries, donors and NGOs. The Government team and the World Bank held workshops and meetings with stakeholders to explain the PBF early on during preparation. The Government consulted with development partners during preparation, to ensure coordination and collaboration during design. 102. Flexibility to adapt to a changing context. The World Bank responded adequately to the Government request to incorporate a component to support the country’s preparedness efforts in the face of the Ebola epidemic. The World Bank worked with the country to re-allocate resources within the project to finance equipment, medications and capacity building. The Bank built on the experience UNICEF had gathered in Liberia, Sierra Leone and Guinea, supporting the countries to respond to the epidemic. 103. There were minor shortcomings in project design. The World Bank knew enough on the status of health facilities across the country to deem that strengthening health infrastructure was key to achieve the PDOs. The World Bank relied on existing health infrastructure assessments at preparation to cost and set targets for this activity. The shortcoming was to underestimate the costs for civil works, which increased during the reconstruction boom in times of peace and the actual status on some of the health facilities which continued to deteriorate after appraisal. This could have been foreseen, given that many of the health centers had not experienced any infrastructure improvement since their construction. Once an in-depth assessment study was carried out during implementation, the Government and the World Bank established priorities for civil works in light of the main findings and the activities of other partners. Another shortcoming was that while the PAD acknowledged the learning curve for the country for piloting the PBF, it underestimated the time and needs required to train and accompany the service providers. It also underestimated the time it would take to master the operational procedures of the PBF, for the different parts of the PBF to work effectively. The Government and the World Bank worked together to ensure the operational challenges were overcome and best practices shared, getting ready for the scale up of PBF. Quality of Supervision Rating: Satisfactory. 104. The World Bank carried out regular supervision missions, adequately staffed. The World Bank provided training on fiduciary matters and was regularly accessible to the PIU for providing support. While there were three TTLs during the five years of implementation, transition arrangements worked smoothly and TTLs remained involved in the country’s health sector work. The supervision of fiduciary aspects and safeguards was adequate, favored by the continuity of the fiduciary team. The World Bank worked in collaboration with the PIU to ensure that safeguards were complied with during construction activities. 105. The World Bank worked with the Government to introduce activities that would ensure the achievement of the PDOs, with a focus on development impact. Showing responsiveness, the World Bank supported the establishment of the Nutrition Units in 269 health centers to support the PDOs and the achievement of targets on malnutrition and supporting the implementation of the National Nutrition Program. The World Bank worked with the Government to Page 34 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) allocate the savings achieved in the Ebola component to provide access to water for health facilities, contributing to the achievement of the PDOs. 106. The World Bank was open in its reporting on the challenges faced during implementation, proving quality reports on the supervision and regularly monitoring and evaluating performance with the Government. An example of this was the MTR, delayed from the second half of 2017 to February 2018. The MTR entailed an exhaustive independent assessment of the Project, to identify challenges and ensure that processes (targeting, registration and payments for PBF) were working as intended or introduce the needed adjustments. The main findings and the continued collaboration to improve the Project’s performance and overcome the operational challenges, resulted in opting for a new follow up project, instead of additional financing. It is important to highlight that the recommendations regarding the changes to the indicators in the RF were formally adopted in the 2019 (restructuring actually taking place in September 2019) together with the reallocation of funds between disbursement categories. The restructuring was a required formality carried out when a reallocation of credit and grant funds was required. It took longer than envisaged because of the intense supervision work and engagement with a new partner (the GF) in the country’s health sector, as well as the preparation of the new follow-up health project to scale up universal health insurance and the PBF. Justification of Overall Rating of Bank Performance Quality at entry Satisfactory Quality of supervision Satisfactory Overall rating of Bank performance Satisfactory Page 35 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) D. RISK TO DEVELOPMENT OUTCOME 107. Strengths a) The national budget allocated to the health sector for 2020 increased by 16.58 percent compared to the 2019 budget; b) The follow up project – (SPARK P167959)- became effective in July 2019, representing the expansion of the PBF to all districts in the country; c) The ownership of the PBF at the providers and decentralized regulators (districts and regions) levels, who have embraced the procedures, results and outcomes achieved during the pilot and have been providing support to each other, sharing best practices; d) The institutional strengthening and capacity building in the health sector; e) The ongoing budgetary reform process, focused on performance, budgeting for results, accountability and implementation of national sector policy on health, nutrition, emergency response, among others; f) Establishment of a National Platform for health financing. 108. Challenges and Weaknesses a) The commitment and ownership of the follow up project and the reforms it is supporting by the higher levels of Government: leadership and support for PBF and UHI activities will be critical for the institutionalization of these reforms; b) The commitment will be critical to ensure that budget resources will be available in the future to pay for the PBF subsidies; c) The political economy can undermine achievements; d) The COVID-19 pandemic is likely to affect the development outcome in at least two ways. First, it has worsened the country’s macroeconomic situation. This is likely to constrain Government’s capacity to continue increasing the health budget. Second, it has compelled the Government to increase hospital capacity. While this will be vital in managing COVID-19, experience suggests that once the epidemic has passed, the additional installed capacity is likely to become a recurrent expense. This new capacity would need to be maintained, decreasing budget allocative efficiency. V. LESSONS AND RECOMMENDATIONS Preparation 109. Importance of carrying out complete analysis during preparation. In the case of the Project, the overall analysis was extensive and aligned with the main national policies. Regarding infrastructure, the difficulties on making funds from the PPA available, delayed the study to carry out an in-depth diagnosis of the infrastructure needs. The PAD relied on available assessment studies to cost and establish targets for the infrastructure activities. This resulted in underestimating the conditions of health facilities, the rehabilitation costs and overestimating the number of health facilities that could be rehabilitated with the resources available. The Government and the World Bank collaborated with other donors and established priorities for interventions, focusing on high impact health centers and high-quality rehabilitation. Page 36 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 110. Designing a gradual approach pays off. Opting for a gradual approach when designing a component supporting a program such as PBF, to be implemented for the first time is sensible, considering the need to define procedures, institutional arrangements and assess the training involved to ensure smooth implementation. In the case of this Project, the Government and the World Bank opted for setting two groups of health districts, and sequencing implementation. This allowed the PIU and the CTN-PBF as well as the verification agencies, the regional and district health authorities to focus on a small group of districts and work on overcoming bottlenecks, which benefited the next group of health districts. Staff who worked in districts under PBF and moved to districts not working under that modality contributed to pollinate and disseminate PBF procedures and culture. The central level of the MSLS contributed to disseminate best practices by fostering collaboration between the districts under PBF and the DIIS during the preparation of the annual health statistics report (the RASS). 111. Importance of designing within the country national development policies. The project included activities in support of the main national sector policies, which sought to strengthen the health system, improve its performance and efficiency; and improve access to quality basic health care and nutrition services, in particular for the most vulnerable populations. The inclusion of technical assistance for the universal health coverage policy and the piloting of performance- based financing aimed at supporting the country reform efforts. Implementation 112. Collaboration with Development Partners is crucial for successful implementation. The Government and the World Bank worked closely with development partners during design and implementation. Collaboration with development partners informed setting priorities, adapting to changes in the implementation context and introduce new activities that were needed to achieve the PDOs. Examples under the project included the definition of priorities regarding infrastructure rehabilitation; the support for administrative and verification activities for additional health districts under PBF supported by the GF and the decision to finance water and sanitation activities in the northern districts of the country to better prepare them to tackle health epidemics. Collaborating partners showed flexibility, commitment and long-term engagement reaping the fruits of their long-term work. 113. Commitment to M&E to guide Project Implementation pays off and supports long-term decisions. The Project focused on monitoring and evaluation to assess implementation progress and guide decisions to adjust project activities and reallocate resources. The Government and the World Bank focused the MTR to carry out a deep analysis of the RF indicators to decide how to move forward on the country’s health sector priorities. An impact evaluation is key to provide critical lesson for scaling up a program. Earmarking resources for an impact evaluation is key to ensure carrying out a sound and timely baseline and evaluation analysis. In the case of the Project, availability of Government resources for the impact evaluation and COVID-19 contributed to delay the completion of the impact evaluation before completion. On Performance based financing: 114. Training and capacity building are crucial for successful implementation: When projects have training among their activities to ensure adequate implementation, building capacity and bringing teams up to speed takes longer than expected. Capacity building is important for the good foundation of a program. In the case of this Project, despite the training initially provided, more training was needed in areas or topics (such as financial management or preparation of a business plan) to overcome operational challenges. The ACVs as well as the CTN-PBF needed to furnish more support. All Page 37 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) this capacity building was required to master operational procedures in the PBF program, and this took longer than envisaged to master. When designing a project, time needs to be built in, in a realistic manner, if activities and programs are new to the country. The study on the use of subsidies recommended that the ACVs train providers and management committees (COGES) on financial management, insisting on budgetary procedures and nomenclature as well as on a realistic planning of activities. This is a must when a program and activities are new (such as PBF with all its instruments and procedures). Training in other areas such as procurement (on STEP for instance), and safeguards, is relevant for the strengthening of the health sector. While the gradual approach, the needed up-front investment and capacity building translated into a slower trend of disbursements early in project implementation, this trend changed with implementation and disbursements accelerated with the know-how acquired. The implementation experience will benefit districts under PBF in the next scaled up phase. 115. Role of the verification agencies as agents of change: The ACVs were an integral part of the PBF as designed in Cote d’Ivoire. As the PBF was piloted in the country, it was clear that the ACVs played a key role supporting the transformation of health care facilities. They built capacity, by accompanying and coaching health providers all the way. They also helped change the work of regulators in PBF districts, contributing to a transformation in work culture, accountability, use of quality data and performance. Contracts for ACVs need to include incentives for ACVs knowledge transfer to all levels (health facilities, districts and regions), to ensure that the accompanying and coaching work is considered as a key component of their work with health facilities. 116. Need to factor in the composition and functioning of local actors: In the case of the COGES, they did not have the capacity or the incentives to carry out the functions foreseen in the PBF manual. Their composition represents the administration and Government, rather than the community. It was foreseen to have them participate in the business plan and in the decisions on the uses of funds. But as many of them were not operational at the beginning of the Project, the Project did not introduce the changes required to make them functional. Later on, in implementation, they were trained in financial management. The COGES do not have legal independent status, which was a limiting factor as well. This lesson was considered when SPARK was designed and considered the role and collaboration of local actors. 117. Importance of clearly defining roles and responsibilities. While the CTN-PBF and PIU worked together to implement the Project, this collaboration would have been more effective if their roles and responsibilities would have been further detailed. In case of changes in management in any department, a clear description of functions can contribute to overcome conflicts and delays. This is crucial at a moment when the PBF is expanding to the rest of the health districts in Cote d’Ivoire. It is important that the MSHP enables the CTN-PBF to carry out its job well, reinforcing its capacity with adequate human and financial resources (with a budgetary line). Ensuring clear institutional roles and responsibilities is key for the sound governance of any program, whatever the institutional architecture selected by the country. 118. Mastering operational procedures takes time: there is a steep learning curve when implementing PBF. The PBF brings about a change in mentality at all levels of health service delivery and administrative and operational transformations. The country needed time to establish the pieces of the PBF system (steering committees, CTN-PBF, PBF tools, coordination with other relevant ministries, etc.), at the central level and from the central level, to the providers and to the regional, district and health center levels. Training and support are crucial to build knowledge and capacity, and to figure out how to resolve unforeseen obstacles and to disseminate best practices with other actors. Training needs were also underestimated. Training is truly necessary, not just as a one-off activity, but in a continuous form, provided by key actors, such as the ACVs, the CTN-PBF, the PIU and the district and regional offices of the MSHP. Page 38 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 119. Disseminating best practices can help with future implementation: Creating communities of practice is important to capitalize on the experiences during a pilot phase. The dissemination of the solutions to obstacles and the innovative activities that flourished is key to passing on the know-how acquired in a pilot phase in a program or project implementation. In the case of this project, numerous actors in healthcare centers overcame obstacles in the implementation of the PBF program. Examples were the lessons derived from the lack of enough options to put together a short list in their area, displaying initiative in seeking and organizing training to enable the provision of a health service; etc. During the ICR mission, examples were shared to show how unforeseen challenges had been overcome and how actors had learned how to best use the resources to meet their needs. Creating such an exchange mechanism for experiences could support the expansion of the PBF program to the rest of the districts in Cote d’Ivoire. . Page 39 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: PDO Indicators Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Deliveries in health facilities Number 0.00 191209.00 217818.00 by trained health personnel 04-Nov-2014 04-Nov-2014 31-Jan-2020 Comments (achievements against targets): Surpassed Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Severely malnourished Number 0.00 16220.00 19941.00 children detected and treated 01-Feb-2015 02-Feb-2015 31-Jan-2020 Comments (achievements against targets): Page 40 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Surpassed Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Vaccination coverage: DTC Percentage 97.00 99.00 100.00 01-Feb-2015 02-Feb-2015 31-Jan-2020 Comments (achievements against targets): The yearly targets were surpassed in years 2015, 2017 and 2018, and reached in 2016. There is is no final data for 2019 Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Average quality score Number 0.00 30.00 35.00 improvement for facilities covered by RBF 01-Feb-2015 02-Feb-2015 31-Jan-2020 Comments (achievements against targets): Surpassed Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Page 41 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Per capita utilization of Number 0.35 56.60 58.60 services 01-Feb-2015 02-Feb-2015 31-Jan-2020 Comments (achievements against targets): Surpassed A.2 Intermediate Results Indicators Component: Intermediate Results Indicators Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Women using modern Percentage 13.80 15.60 15.77 family-planning methods 01-Feb-2015 02-Feb-2015 31-Jan-2020 Comments (achievements against targets): Surpassed Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Children under 5 who Number 0.00 706350.00 300000.00 361042.00 Page 42 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) received Vitamin A 01-Feb-2015 02-Feb-2015 15-Oct-2019 18-Jun-2019 supplementation Comments (achievements against targets): Achieved with new target. indicator was revised to reflect the change in strategy on the country. Originally the target was supposed to be achieved through mass campaigns. However, the national strategy was modified to facility based distribution, consequently national targets where also lowered. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Pregnant women receiving Number 0.00 368048.00 411264.00 antenatal care during a visit to a health provider 01-Feb-2015 02-Feb-2015 31-Jan-2020 (number) Comments (achievements against targets): Surpassed Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion People with access to a basic Number 0.00 7433532.00 7788773.00 package of health, nutrition, or reproductive health 01-Feb-2015 02-Feb-2015 31-Jan-2020 services (number) Page 43 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Comments (achievements against targets): Surpassed Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Health personnel receiving Number 0.00 4720.00 4893.00 training (number) 01-Feb-2015 02-Feb-2015 31-Jan-2020 Comments (achievements against targets): Surpassed Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Direct project beneficiaries Number 0.00 7583532.00 7580420.00 01-Feb-2015 02-Feb-2015 31-Jan-2020 Female beneficiaries Percentage 0.00 60.00 62.10 Comments (achievements against targets): Achieved Page 44 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Proportion of indigents Percentage 0.00 20.00 0.00 among the total direct RBF beneficiaries 01-Feb-2015 02-Feb-2015 15-Oct-2019 31-Jan-2020 Comments (achievements against targets): The indicator was eliminated at project restructuring in October 2019, because it could not be measured. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Indigents that remain Number 0.00 150000.00 201354.00 enrolled in health insurance 01-Feb-2015 02-Feb-2016 31-Jan-2020 Comments (achievements against targets): Surpassed Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Annual RASS produced Yes/No N Y Y Page 45 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 01-Feb-2015 02-Feb-2015 31-Jan-2020 Comments (achievements against targets): Achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Health facilities constructed, Number 0.00 526.00 97.00 138.00 renovated, and/or equipped (number) 01-Feb-2015 02-Feb-2015 04-Oct-2019 31-Jan-2020 Comments (achievements against targets): The target for this was revised during the 2019 restructuring to reflect that as the country resumed normal operations, new partners resumed their support and agreed to finance equipment for most of the the 515 facilities initially targeted, as well as rehabilitating several of the 200 facilities targeted for rehabilitation. In addition, the health facilities turned out to be in worse shape than originally identified. Moreover, the substantial uptake in construction after the civil unrest, coupled with limited supply of qualified construction firms, resulted in higher unit costs of civil works than anticipated. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Service readiness of health Percentage 0.00 30.00 29.00 facilities 01-Feb-2015 02-Feb-2015 31-Jan-2020 Page 46 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Comments (achievements against targets): Partially achieved, Data from the PBF quality scorecard. This indicator considers the following aspects: availability of essential drugs; availability of blood products; laboratory functioning; hygiene, security, and environment; emergency external consultations; functionality of radiology services; functionality of surgery services. These data come from the 2016 SARA survey Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Equip./supplies available in Percentage 0.00 75.00 88.00 the National Institure for Public Hygiene (INHP) 01-Feb-2015 02-Feb-2015 31-Jan-2020 Comments (achievements against targets): Surpassed Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion People who have received Number 0.00 900381.00 900381.00 1186062.00 essential health, nutrition, and population (HNP) 02-Feb-2015 04-Sep-2019 04-Sep-2019 31-Jan-2019 services People who have received Number 0.00 540229.00 540229.00 735358.00 essential health, nutrition, and population (HNP) services - Female (RMS Page 47 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) requirement) Number of children Number 0.00 126401.00 126401.00 192165.00 immunized 02-Feb-2015 04-Sep-2019 04-Sep-2019 31-Jan-2019 Number of women and Number 0.00 582771.00 582771.00 885422.00 children who have received basic nutrition services 02-Feb-2015 04-Sep-2019 04-Sep-2019 31-Jan-2020 Number of deliveries Number 0.00 191209.00 217381.00 attended by skilled health personnel 02-Feb-2015 02-Feb-2015 31-Jan-2020 Comments (achievements against targets): Surpassed. This is a corporate indicator that was added in October 2019, at the end of the project to comply with management directives to retrofit such indicators in every project whenever a project was restructured. Two sub-indicators were already part of the project's results framework, and two more were added (Women and children receiving nutrition services, and Number of deliveries attended by skilled personnel). For these last two sub-indicators, we have used the same number for the end target as for the original target, because they were added to the results framework three months before the end of the project. Page 48 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Notes on the indicators: 1. Annual RASS Produced: The value for this indicator is not cumulative. This indicator was achieved and surpassed in 2017 and 2018, when it reached 100%. The value is provided by the 2018 RASS. The 2019 RASS experienced delays in its completion, due to the COVID19 crisis. The regional ateliers to prepare the RASS suffered delays due to the public health crisis. 2. The adjustments to the results framework were agreed upon to improve the relevance of the monitoring and evaluation to track changes supported by the project. The analysis carried out during the mid-term review resulted in an agreement to clarify the definition of some indicators to ensure correct and accurate calculation (see Annex 1 for changes to each indicator). The targets for two IRIs (see Table 3 main text) were revised downwards for the following reasons: (i) Vitamin A supplementation: The original target was based on the national strategy at the time of project preparation consisting of a twice a year campaign. Early in project implementation the National strategy was revised to using a facility-based supplementation. The new strategy is known to result in high coverage for the first year as it is attached to vaccination and reduced coverage afterwards as Vitamin A is only provided to children coming to health facilities when they are sick; (ii) Health facilities constructed, rehabilitated and/or equipped: The indicator was confusing as it grouped together equipment and civil works without targets for each. After the project was approved, several financing partners came forward with resources to equip health facilities damaged by the civil conflict. It was decided that the project would focus exclusively on civil works (construction and rehabilitation). In addition, a 2016 study provided an in-depth survey of the status of health infrastructures, which was not available at appraisal; the study showed that the physical condition of health facilities was worse than foreseen, and the rehabilitation unit costs estimated at appraisal were low. The indicator was reformulated as “Health facilities constructed or renovated”. Furthermore, the IRI “proportion of indigents among the total PBF” was dropped because data could not be gathered, due to delays in the implementation of the electronic records system for patients 29. 3. The Mid-term review looked in depth at the evolution of all the indicators in the results framework. The analysis showed that there were: a) 10 out of 18 indicators that had been met or surpassed their targets; b) 5 indicators that had not yet reached their targets but were on a path to achieve the targets proposed in the PAD; and c) 3 indicators that had not reached their targets and were not on track to likely achieve them. These three indicators were indicator 1.2 (Severely malnourished children detected and treated); indicator 2.2 (Children under 5 who received Vitamin A supplementation); and indicator 2.10 (Health Facilities constructed, renovated and/or equipped). 29 The Project was supposed to beginning of the development of the electronic patient records (EPR), which took longer than expected. The pilot will be launched during 2020. The initiative experienced delays in Cabinet decision making. The technical designed was carried out and validated by the DIIH and technical teams. There were infrastructure and technological issues related to electricity and internet connection that had to be dealt with before piloting the EPR. Page 49 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 4. Based on this analysis, the MTR provided recommendations to adjust the definition of certain indicators (2.4 “Population with access to health services”; 2.6 “Direct beneficiaries of the Project”; 2.9 “Annual preparation of the RASS”; 2.11 “Health facilities built, rehabilitated or equipped by the project” and 2.11 “Service readiness of health facilities”) and/or define new targets to ensure an adequate monitoring and assessment of the project progress and achievements. This was the case of indicators 2.2 “Children under 5 who received Vitamin A supplementation”; indicator 1.2 “Severely malnourished children detected and treated” and indicator 1.5 “Per capita utilization of services”. 5. Children Under 5 who received Vitamin A Supplementation: The changes in this indicator were needed to reflect the change in policy regarding the administration of Vitamin A. The baseline was based on the numbers from vitamin A campaigns, while this indicator aimed to measure routine vitamin A supplements, delivered at health facilities on routine checkups. 6. Average Quality Score for Facilities covered by the PBF: This indicator focused on the 14 districts originally included in the PAD. Page 50 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1 To Strengthen the health system (for improved performance). 1. Severely malnourished children detected and treated 2. Deliveries in a health facility by trained health personnel Outcome Indicators 3. Vaccination Coverage DTC-HepB-HiB3 4. Per capita utilization of services 5. Average quality score for facilities covered by PBF 1. Children under 5 who received vitamin A supplementation (capacity at health centers to provide this basic service) 2. Pregnant women receiving antenatal care during a visit to a health provider (capacity to provide the service, strengthen management at local level) 3. People with access to a basic package of health, nutrition or reproductive health services (Number of People, in particular women and children, who have received basic nutrition services) 4. Health personnel receiving training Intermediate Results Indicators 5. Direct project beneficiaries (and female beneficiaries) 6. Annual RASS produced (strengthened health information system) 7. Health facilities rehabilitated and equipped 8. Number of health facilities under PBF who used their resources to equip themselves 9. Service readiness of health facilities (strength of the health system) 10. Equipment/supplies available in the National Institute for Public Hygiene (INHP) 1. Technical assistance provided to the MSHP: Hire international expert to support the PBF pilot implementation phase. 2. Developed and launched a health management information system with data from the health centers under PBF, Key Outputs by Component accessible by service providers, regulators and the general public. (linked to the achievement of 3. Training of health care providers on simplified financial management (building human capital). the Objective/Outcome 1) 4. Performance contracts signed by district and regional levels with the CTN-FBP/ MSHP (outcome: it allowed regulation and strengthening of the governance of health system through supervision and regular monitoring of quality). COMPONENT 1: Performance - 5. Development of managerial capacity and accountability by the heads of health centers. Based Financing 6. Development of Manuals and instruments/tools: MPBF, Project Operational Manual, Business plan guidelines. 7. Creation of the Nutrition Units in 14 PBF districts, allowing for the implementation of approved Nutrition Policy. Page 51 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Ebola Preparedness 1. Equipment and medications acquired and delivered: 26 ambulances; 52 other 4x4 vehicles; 200 motorcycles; 4 infrared body scanners; 400 portable infrared thermometers; laboratory reagents; hand washing equipment and supplies for health facilities, schools and universities, the military and the police; Medications, disinfection materials and products; Personal Protective Equipment (PPE); gloves and masks; 55 computers. 2. Rehabilitation of two INHP warehouses: in Daloa and in Adiopodoume, allowing storage of equipment and medication for tackling an epidemic such as Ebola. 3. Construction of 43 latrines VIP, 8 modern latrines, 57 points of access to drinking water (forages and manual pumps), 17 wells and connections to existing water systems in 86 health centers and 3 border posts in the western part of the country: Project WASH in Health (UNICEF). 4. INHP capacity strengthened: acquisition of 2 photocopiers, 32 video projectors, and Training. 5. Equipping District and Regional actors, at operational level (DRS and DDS) to enable systematic reporting of cases, on-site coaching on the use of tablet technology for epidemiology surveillance. COMPONENT 2: Strengthening Health Information System the Health System for 1. Pilot patient electronic records in general hospitals in two districts (Anyama, Bouafle); Improved Performance 2. Development of facility-based information systems; 3. In PBF health centers, staff regularly analyzed health data for planning the business plans and decision-making; 4. Survey on Prevalence of Diabetes in Côte d’Ivoire completed; 5. Completion of the SARA survey in 2016 (The Service Availability and Readiness Assessment of Health Facilities); 6. Elaboration of RASS 2015-2016-2017-2018 (Rapport Annuelle de la Situation Sanitaire); 7. Reproduction of the tools for health data collection (health registry); 8. Health facilities requesting electronic version of these tools for printing themselves; 9. Pilot phase for integration of data from university health centers (CHU), and semi-public institutions (army, police, customs, taxes, etc.) into the national health information data (SNIS) (not foreseen in the PAD); Improving Health Management Systems 1. Activities to support health systems at district and region level, related to PBF and UHI; 2. Revision to improve (and align with PBF responsibilities) the norms regulating the organization and functioning of the Health Regions and Districts. Acquisition of visual equipment to connect the DGS to the Regional Offices (allowing to hold regular virtual meetings); Page 52 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 3. Technical assistance to the MSHP cabinet on health financing and resource mobilization (financing of a senior technical consultant for the cabinet); 4. Electronic Patient Record: (i) Study carried out on the development and implementation of the Electronic Patients Record in two reference hospitals; and (ii) Acquisition of hardware and software for Electronic Patients Records 5. Community Health Strategy developed (and training under this activity); 6. Studies on the (i) functioning of the COGES in the health facilities; (ii) the adequacy/relevance of the training available for the top management of the health sector and the demand for skills and competencies in the MSHP (l’adéquation de l'offre de formation pour les cadres supérieurs de la santé et la demande en compétences du Ministère de la Santé) ; (iii) the liberalization of training for health personnel in Côte d’Ivoire and preparation of the specifications (cahier de charge) for the liberalization; 7. Elaboration of PNDS 2016-2020; 8. Revision and updating of the National Plan for Medical Waste Management (déchets bio médicaux) 2016-2020. Infrastructure, Rehabilitation and Equipment for Health Centers 1. Study on the Status of the infrastructure and equipment of health centers (for 25 districts and 560 health centers); 2. 80 health facilities rehabilitated (great support for the launching of the UHI) ; 3. Acquisition and installation of medical and biomedical equipment for health facilities in 4. 14 districts (other development partners); 5. 10 incinerators for health centers (to support the implementation of the National Planfor Sanitary Waste) in the districts of Bouna, Nassian, M’Bahiakro, Vavoua, Bouafle, Yamoussokro. Universal Health Coverage (UHI) 1. Four studies completed. (*see at the bottom of the table); 2. International Symposium held in Abidjan to share international experiences in universal health coverage (UHI) on health risk in Africa; 3. Pilot project to identify the poor in the 14 PBF districts supported by the project; 4. Validation of the strategy to identify and incorporate the agricultural and informal sector into the UHI; 5. Co-financing of regional workshops to prioritize training for health centers offering SONU; 6. Piloting the implementation of the UHI in three health districts under PBF districts (Yamoussoukro - Issia - Babakala). Page 53 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Project Management 1. Acquisition of equipment, hardware and software for the PIU; 2. PRSSE Communication strategy prepared and disseminated; 3. PRSSE Operational Manual prepared; Objective/Outcome 2 To Improve the utilization and quality of health and nutrition services in selected regions. 1. Deliveries in a health facility by trained health personnel 2. Severely malnourished children detected and treated Outcome Indicators 3. Vaccination Coverage DTC-HepB-HiB3 4. Average quality score for facilities covered by PBF 5. Per capita utilization of services 1. Women using long-term family planning methods; 2. Children under 5 who received vitamin A supplementation (regular visits to health centers) 3. People with access to a basic package of health, nutrition or reproductive health services (number of People, in particular women and children, who have received basic nutrition services); Intermediate Results Indicators 4. Pregnant women receiving antenatal care during a visit to a health provider (quality and utilization) 5. Health personnel receiving training (quality) 6. Direct project beneficiaries (and female beneficiaries) 7. Proportion of indigents among the total direct RBF beneficiaries (Dropped) 8. Service readiness of health facilities (quality) 1. Document on the national FBP strategy (revision of the document in third quarter 2019; participatory workshops to prepare the draft and consolidate it into an strategic document); 2. PBF Operational Manual completed; 3. PBF studies completed: (i) costing of health services delivery in public, semipublic and private health centers in the Key Outputs by Component context of the UHI; (ii) status of health infrastructure in 25 districts; (linked to the achievement of 4. Agencies for Verification hired and carrying out its responsibilities; the Objective/Outcome 2) 5. Community-based organization hired and working in the PBF districts; 6. Acquisition of nutritional inputs for 14 Therapeutic Nutrition Units (UNT) in the PBF districts; 7. Opening bank accounts and payment of subsidies to health centers, ensuring access to resources to improve functioning, foster motivation and improve quality of services delivered; Page 54 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 8. Availability of a “Bonus start up” for health centers to support them in improving their technical capacity; 9. Staff training on the utilization of the management tools for the health service supply under the UHI; 10. Provision to all 598 public health facilities with 366 black trash cans with covers (dimensions of 240 liters), 782 black trash cans (dimensions of 27 liters with pedal), 1,564 yellow trash cans (dimensions of 27 liters with pedal); 9,807 security boxes; 12,8710 black trash bags sacs (300 liters); 12,8710 black trash bags of 50 liters; 20,3840 yellow trash bags of 50 liters and 200 yellow carts for bio-medical trash transportation; 11. Innovative initiatives lead by health centers under PBF carried out, for improving the utilization of health services (examples): a. organization of mobile health visits to remote communities b. hiring motorcycles services to bring pregnant women who live in remote villages to the health centers for their prenatal visits and for deliveries; c. competitions and prices to acknowledge and reward mothers who respect the vaccination schedule and the four prenatal consultations; d. health providers organized a system of calls to reach out to pregnant women who stopped visiting the health centers and/or have missed vaccinations’ appointments for their children. 12. Health centers used the PBF subsidies FBP for improving the quality of the health services they provide, such as: a. building latrines and spaces providing access to water for the maternity areas; b. acquisition of emergency kits and beds; c. rehabilitation and equipping of the hospital rooms to enhance patients’ comfort; d. acquisition of medical materials (réactifs and films) for clinical exams. 13. The PBF fostered and accelerated the establishment of services that were not available in certain health centers: a. Unités nutritionnelle thérapeutiques to tackle malnutrition b. Family planning c. Tuberculosis diagnosis center *The four studies are : (i) Calcul du cout de production des actes médicaux et des services de santé; (ii) Mobilisation des ressources additionnelles pour la UHI ; (iii) Cout global de mise en œuvre de la UHI ; and (iv) Capacité contributive des ménages. Page 55 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Responsibilities Preparation Dominic Haazen TTL, Senior Health Specialist Ibrahim Magazi TTL, Senior Health Specialist Maurice Adoni Senior Procurement Specialist Rianna L. Mohammed-Roberts Senior Health Specialist Helene Barroy Economist Jean Charles Kra Amon Senior Financial Management Specialist Abdoulaye Gadiere Senior Environmental Specialist Faly Diallo Finance Officer Andrea E. Stumpf Lead Counsel Marie Roger Augustin Legal Analyst Aissatou Chipkaou Operations Analyst Sybille Crystal Senior Operations Officer Amba Denise Sangara Team Assistant Oumou Collobaly Team Assistant Akoua Gertrude Tah Team Assistant Kouassi Lambert Procurement Consultant Supervision Dominic Haazen TTL Senior Specialist Health Policy Ibrahim Magazi TTL Senior Specialist Health Emre Ozaltin TTL (Senior Economist) Moulay Driss Zine Eddine El Idrissi Senior Health Economist Aissatou Chipkaou Operations Analyst Haoua Diallo Program Assistant Damien de Walque Senior Economist - RBF Impact Evaluation Page 56 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Cedric Ndizeye Health Specialist Ayodeji Oluwole Odutolu Senior Health Specialist Lombe Kasombe Operations Analyst Sandhya Sukumaran Consultant Monserrat Meiro-Lorenzo TTL, Senior Health Specialist Opope Oyaka Tshivuila Mataka TTL, Health Specialist Maurice Adoni Senior Procurement Specialist Jean Charles Kra Amon Senior Specialist Financial Management Abdoulaye Gadiere Senior Environmental Specialist Oumou Kassi-Coulibaly Program Assistant ICR Monserrat Meiro-Lorenzo TTL /ICR supervisor M. Rosa Puech Operations and Evaluation Consultant/ ICR Main Contributor B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY14 21.867 199,496.03 FY15 30.441 190,034.05 FY16 .250 617.90 FY19 0 20,994.64 Total 52.56 411,142.62 Supervision/ICR FY15 3.925 34,958.17 FY16 15.372 125,167.58 FY17 20.351 117,201.52 Page 57 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) FY18 38.738 376,650.54 FY19 40.003 885,546.10 FY20 23.235 302,337.23 Total 141.62 1,841,861.14 Page 58 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Results-based Financing 38.5 36.50 0 Health System Strengthening 38.5 38.50 0 Total 77.00 75.00 0.00 Total Project cost: US$77 million: The Borrower’s contribution, estimated US$7 million, was allocated to Component 1 as commitment towards the Performance-Based Financing activities. Table 8: Total Project Cost by Financier (IDA and Government) in FCFA* Source of Total Amount (disbursed and to be Original Budget Financing disbursed before end of grace period) IDA 38 806 458 398 37 070 523 026 Borrower 4 060 000 000 3 935 316 934 TOTAL 42 866 458 398 41 005 839 960 Source: Service de gestion administrative et financière, PRSSE, 31 May 2020 * Note: The table above is in FCFA at an exchange rate used by the PIU of FCFA 580/ 1 US$ Page 59 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Table 9: Cost of Component 1 - Cost of the Performance-Based Financing in FCFA Cost financed by the Cost financed by Cost financed Cost financed by IDA in original Cost financed by IDA in 5 GF Component Government in original 14 Government in 5 GF by GF in 5 Total 14 districts districts districts districts districts Up-front Up-front Up-front Up-front Recurrent Recurrent Recurrent Recurrent Recurrent investment investment investment investment Subsidies 5 922 258 111 1 502 509 408 - 215 022 490 - 32 253 374 2 647 543 267 10 319 586 650 - - Support to PBF 1 877 973 593 3 831 960 641 331 407 105 520 719 681 625 991 198 1 887 383 599 110 469 034 256 473 872 9 442 378 723 implementation - Total 1 877 973 593 9 754 218 752 331 407 105 2 023 229 089 625 991 198 2 102 406 089 110 469 034 288 727 246 2 647 543 267 19 761 965 373 Source: Service de gestion administrative et financière, PRSSE, 31 May 2020 Table 10: Actual Expenditures by Component 2 (FCFA) Expenditure IDA Expenditure Government Total Amount % Sub-components 2.1: UHI 1 222 615 334 18 231 900 1 240 847 234 5,8% 2.2: Rehabilitation of essential Infrastructures 8 903 641 259 40 857 196 8 944 498 455 41.6% 2.3: Health management 914 933 268 2 000 000 916 933 268 3,83% information system 2.4: Improvements health 1 786 117 271 3 209 555 1 789 326 826 8.4% management system 2.5: Ebola preparedness 5 983 504 531 5 983 504 531 25% 2.6: Project Coordination 3 899 121 732 1 117 185 809 5 016 307 541 23.3% Total 22 709 933 395 1 181 484 460 23 891 417 855 100% Source: Service de gestion administrative et financière, PRSSE, 31 May 2020 Page 60 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Cost structure under Category 2: Support to the Implementation of PBF 1. The table below shows the nature of the different activities financed under Category 2 of the credit and grant. Category 2 financed (i) investment costs to launch the PBF program (which included equipment, training on the PBF and on financial management); (ii) the costs of all verification activities and (iii) recurrent costs such as supervision, quality evaluation and coordination meetings). Regarding the cost of the verification activities, it is worth pointing out that 33 percent of those costs covered the implementation of the PBF in the five additional districts supported by the Fond Mondial. The five additional districts represent 33.6 percent of the total population in the 19 districts supported by the Project. Structure of costs financed under Category 2: Support PBF implementation 2016-2020 70% 60% 50% 40% 30% 63% 20% 27% 10% 10% 0% a. Up-front investment costs b. Verification Costs (ACV, OBC) c. Recurrent costs (Supervision, Quality (Preparation of strategy and tools, PBF evaluation, coordination meetings) initial training and FM, equipment, baseline studies) Page 61 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) ANNEX 4. EFFICIENCY ANALYSIS Cost Benefit Analysis at Appraisal. 1. The PAD included a cost-benefit analysis (CBA) to measure the Project’s economic performance and assess the returns against the alternative of the status quo. The CBA focused on the PBF program and the infrastructure activities, which amounted to around US$69 million of the total project budget (US$77 million or 89 percent). 2. The CBA considered only the direct benefits and costs of both activities. For the PBF program, direct costs meant the project cost for purchasing the services. The CBA did not include indirect costs due to the challenge to assess and monetize those. Direct benefits meant the total gains generated from the use of health services delivered to the beneficiaries. The analysis did not account for quality upgrading and indirect benefits were not accounted for (such as user’s or health professionals’ behavior changes). The Project was expected to be implemented in a five-year period. The first year was expected to consist of setting up the mechanisms for the pre-piloting PBF scheme, with the PBF scheme being up and running for all 14 districts in the pilot, starting year 2. 3. The analysis showed a rate of return of approximately 25.88 percent (and a net value of US$9.4 million). The CBA showed that the NPV and the RR would not vary, because the prices of the services purchased were not be modified over the course of the project. The costed minimum benefit package (excluded HIV/Aids treatment) was estimated at US$73.4. Actual Costs and Benefits at Completion 4. Changes to the CBA assumptions. The main changes during implementation that might have affected the NPV and the RR were: a) the activities to launch the PBF took longer than expected. The first four health districts started in August 2016 and the next 10 health districts started in February 2017 and subsequent months; b) The infrastructure activities to rehabilitate primary health care centers took longer than anticipated, given that the study on the state of situation of the health infrastructure in the 25 health districts was carried out in mid-2017. This delayed realizing the benefits of infrastructure strengthening; c) The number of health centers rehabilitated under the infrastructure strengthening sub-component was smaller than anticipated; d) A total of 265 health facilities used the resources provided under the PBF to carry out upgrades in the physical infrastructure, within the PBF manual guidelines, contributing to strengthening the technical capacity and quality of the provision of health services in their communities; e) The number of beneficiaries was higher than estimated given that: (a) three more districts (subdivided later into five for administrative reasons) were part of the PBF pilot program; (b) additional key activities were carried out within the resources of the credit and grant, such as the establishment of 269 UNT in the PBF districts, enhancing the capacity to tackle malnutrition in these regions. 5. Additional benefits from Project interventions. Another element to consider in terms of costs and benefits is the improvements in the score to measure improvements in the quality of health service provision, in particular regarding: (i) Page 62 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) hygiene, security and environmental aspects; (ii) technical upgrades to operating rooms and maternity blocks; (iii) admissions and hospitalizations; and, (iv) availability of medication. It is safe to assume that improvements in the quality score (measured as a percentage), while not considered at appraisal, both brought benefits to the covered communities and drove the increases in utilization of health services and, potentially, improved health outcomes. 6. It is likely that the Project had positive effects on human capital. The project directly supported achievements in nutritional status (by detecting and treating malnutrition), which in turn improves education outcomes, and in family planning, all of which impact human capital 30. 7. The Project focused on mothers and children as vulnerable groups. The project contributed to reduce malnutrition and improve nutritional status, which are recognized to impact children cognitive development. It contributed to improve the health of pregnant women in PBF districts, increase institutional labor, and family planning. It is recognized that family planning contributes to increased use of preventive health care, higher education of children and increased employment of women. A regional analysis31 showed that for every US$1 invested in family planning US$3 would be saved in other development sectors that contribute to the Millennium Development Goals. 8. The project enabled creative financing initiatives to provide access to health services to remote communities living outside a 5-kilometer radius of a health centers, without an ambulance service or transportation (this was the case for the rural health center of Broma, in the district of Issia, visited during the ICR mission). Literature 32 aggregating evidence for low-income countries, finds that strengthening health workers capacity, community level health systems, and health information systems are all associated with increased utilization and reduced mortality and morbidity. 9. The Project strengthened health care workers’ performance. The Project financed training on different areas (PBF instruments, financial management and planning, surveillance instruments, etc.); supported additional staff compensation and higher budget for the health centers, and improved planning and management using quantitative and qualitative indicators. All of these measures can be presumed to have brought benefits. 10. Among the benefits from rehabilitated health facilities, it is important to point out: attracting people to the health centers, increasing utilization of health services (some of which are preventive care, such as vaccinations and check up on newborns and children under 36 months); lowering infection rates, improving the use of medical equipment; increasing staff’s motivation and morale; increasing patient satisfaction. Improving access to and rehabilitation of health care facilities translates into health indicators, as it was the case of the Project. 11. Among the benefits from strengthening the information system: it is likely that the strengthening of the HMIS, including the establishment of the PBF MIS, contributed to better decision making, planning and use of resources, generating benefits for the health system in the PBF districts and overall. 30 PAD SPARK -HEALTH 31 PAD SPARK-HEALTH and Family Planning: Francophone West Africa on the Move. A call for action. 32 PAD SPARK-HEALTH and Hatt, Laurel, Ben Johns, Catherine Connor, Megan Meline, Matt Kukla, Kaelan Moat, June 2015. Impact of health systems Strengthening on Health. Bethesda, MD. Health Finance and Governance Project Abt Assotiates Page 63 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS 1. The Borrower comments were introduced in the document as relevant. The borrower’s own assessment of the project is included in the list of supporting documents and the project’s files. The executive summary is included below is summarized below. Executive Summary 2. The Cote d’Ivoire, represented by the PIU for World Bank Projects, as the coordinating unit for the Health Systems Strengthening and Ebola Preparedness Project (Projet de Renforcement du Système de Santé et de Réponse aux Urgences Épidémiques, PRSSE), for the 2015-2019 period, requested the preparation of a completion and results report for the PRSSE. The report covers the ensemble of the project activities carried out under all components and subcomponents and the process of overall coordination of the project. 3. The methodological approach followed to prepare the report had several stages: a) Review of Aide-memoires, documents and reports prepared throughout the project implementation; b) Interviews with key actors and beneficiaries who participated in the project’s implementation (implementation partners, operational actors during implementation, such as the regional and departmental directorates, health care providers and COGES leaders); c) Interview with key actors in the ministries responsible for implementation of project activities (Cabinet Director of the Ministry of Health, Director General for Health, Ministry of Economy and Finances and Ministry of Budget); d) Interviews with technical and financial partners (UNICEF, WHO, MSHP, World Bank), e) Field visit to several health districts benefiting from the project’s interventions (Bouaflé and Issia) and interviews with the local health authorities (Regional Directors and the health management teams at district level, hospital managers and managers for urban and rural health primary health care centers); f) Finally, an analysis of data, triangulating the information to ensure the coherence and consistency of the data and information used to prepare this report. 4. This report describes the main results of each of the component and subcomponents (results- based financing, universal health coverage (UHC), rehabilitation of health facilities, health information systems, health management systems and prevention and response to epidemiological responses). The project’s implementation generated tangible results, with programmatic successes that can undeniably be attributed to the interventions of the PRSSE 5. Regarding Component 1 – « Performance-based Financing » (PBF), the implementation of the pilot phase achieved actual results that allow to contemplate a more successful implementation during the scale of the PBF program: a) Increased readiness of health facilities to provide health services (from 5 percent in 2015 to 26 percent in 2019), ensuring capacity to effectively react and admit patients for care (permettant d’assurer la réactivité et une prise en charge efficace dans les structures sanitaires); Page 64 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) b) Increased number of pregnant women assisted by health staff for pre-natal consultations (from 30,656 in 2015 to 384,539 in 2019), demonstrating increased access to and availability of medical services (the capacity to treat patients - la prise en charge); c) Increased number of children suffering malnutrition treated in health facilities; d) Strengthened human and technical capacity of the operational actors regarding financial management (Regional Directors, Department/District Directors and head to health facilities); e) Emergence of a culture of governance, transparency, teamwork, with stronger involvement of the COGES; f) Utilization of part of the subsidies received by health facilities to strengthen the technical capabilities of health centers and working conditions, generating and reflecting collective ownership and more participatory choice of investments, to be carried out to meet the real needs of the population they serve; 6. Beyond the programmatic achievements, the PBF brought a real revolution to the allocation of resources in the health sector and a change in paradigm, regarding the management of health facilities. The implementation of the PBF strategy showed the need to understand and focus on governance, participation, equity, inclusion, ethics, fostering the efficiency of health services as the guiding values for a well performing health system. 7. The PBF strategy was a trigger that called for and fostered reforms of the health system and its overall transformation. These should encompass the operationalization of the different reforms undertaken by the Government (la tutelle), such as the program budget, the hospital reform, the human resources reform in the health sector, the community strategy reform, UHC, the decentralization process with the operationalization of the health districts through the clear and precise definition of the roles and responsibilities of key actors (Regional health director, Departmental director, hospital directors, head of health centers, COGES, among others). 8. Regarding Component 2 « Strengthening the health system for improved performance», the report’s evaluation noted tangible results, allowing to see conditions for strengthening the health system and improve its performance. Sub-component « Improving Universal Health Coverage (UHC) », 9. The PRSSE offered an excellent opportunity to better understand the difficulties, challenges and issues of the implementation of an social insurance system (assurance sociale) through capacity building and strengthening (training and sharing of best experiences and practices) and carrying out studies on critical aspects of the implementation of the UHI (on the costs of provision of specific health services, the cost of global coverage for the eligible population for the UHI, the contributive capacity of households, establishing a pilot project to identify the indigents). 10. The main conclusions and recommendations of the different studies provided the CNAM with a set of elements crucial for the analysis of the potential financial viability of the UHI; it provided elements to consider costs control and devise measures to foster efficiency in the provision of health services and avoid unnecessary expenditures and wasting resources; it allowed to understand the numerous challenges linked to the enrollment and provision of health care (prise en charge) for the poor. The studies highlighted the need for institutional arrangements and strategic collaborations with Page 65 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) other ministries, in order to guarantee the provision of quality health care services, social mobilization and stakeholders’ engagement in the search of opportunities to mobilize additional resources. Sub-component « Rehabilitation of essential Infrastructures » 11. Despite the weak budgetary envelope allocated to this sub-component, eighty-three (83) primary health care centers (ESPC in French), in twenty-five (25) health districts, were rehabilitated, improving geographical access, increasing utilization of health services and improving the quality of patient care. The rehabilitations facilitated and enabled the implementation of the PBF. Additionally, in the framework of the fight against Ebola, two (02) warehouses were rehabilitated, to store the goods acquired in the framework of strengthening the national system to respond to epidemic emergencies and ninety (91) primary health care centers (ESPC) with remaining resources from the Ebola subcomponent were rehabilitated as well. Sub-component « Health Information Systems » 12. The PRSSE supported the elaboration of the Annual Reports of the National Health Status (Rapports Annuels de la Situation Sanitaire -RASS) for 2015, 2016, 2017, 2018, contributing to the availability of updated data to support decision making based on evidence. 13. The PRSSE supported the development of a pilot model for « Electronic Patient Records» (« Dossier patient informatisé »), an initiative that permits to make available quality data for the Regular Health Information System (Système d’Information Sanitaire de Routine - SISR) related to regular health care activities carried out by healthcare facilities. This pilot was implemented in two general hospitals (Hôpitaux de référence): The General Hospital of Bouaflé and the General Hospital of Anyama. The pilot of the patient electronic record contributed to the provision of personalized and adapted patient care to patients. It allowed strengthening the hospitals management information system, through the operationalization of administrative, medical and para-medical management. The electronic record addresses key information challenges and needs regarding the implementation of the UHI. Sub-component «Improving Health Management System» 14. There were significant results under this transversal sub-component, thanks to the financial support of the PRSSE: a) Support for the elaboration of the National Health Development 2016-2020 (Plan National de Développement Sanitaire - PNDS). The PRSSE financed the process of elaboration of this national strategic policy document, that defines every five years the main priority and strategic lines (axes) and allows to have a synergy of actions between the national actors and the technical and financial partners who support the health sector. b) Support the intensification of nutrition activities in those districts covered by the PBF, through the provision of nutritional supplements, of anthropometric equipment and materials and culinary demonstration materials, together with capacity building activities for 303 staff and the reproduction of data gathering tools. This collaboration, by providing financial support to the Ministry of Health (MSHP), enabled and accelerated the MSHP’s engagement in the revision of the strategy to fight malnutrition by going from campaigns to Page 66 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) distribute Vitamin A, to regular distribution of the vitamin in the health centers. The project’s support contributed to the provision of health care services to a significant number of severely malnourished children. c) Financial support to carry out three key studies: i) Analysis of the functioning of the COGES in general hospitals and primary health care centers; ii) Liberalization of training for health providers (agents de la santé); iii) Adequacy of the available training for health managers, in the context of the country’s health needs. 15. The studies’ findings provided the Government with facts and data information that were used to support the different reforms the Government has undertaken. 16. Finally, as part of capacity building within the MSHP, the project supported hiring in 2018 a senior technical advisor, responsible for health financing and mobilization of resources. This advisor supported the MSHP in strategic thinking regarding health financing and mobilization of domestic resources. Sub-component «Preparedness against fighting the Ebola virus» 17. This sub-component directly contributed to strengthening the Ivory Coast’s readiness capacity to address a major health crisis such as Ebola or Coronavirus, through the following: a) Acquisition and distribution of equipment: these included individual protection equipment, mobilization equipment (such as pick-up trucks, vehicles, motorcycles, ambulances), medical materials and accessories (syringes, catheters, first aid), and medications (Amoxicillin Ceftriaxone, etc.); b) Strengthening the logistical capacity within the National Institute for Public Health (INHP): through the acquisition of pick-up trucks, motorcycles, computers and printers, tablets for health centers in the districts to enable systematic monitoring and reporting on diseases under epidemiologic surveillance; c) Increasing access to drinking water and improving sanitation and hygiene conditions in 91 health facilities in 10 health districts in the north west of the country, bordering the neighboring countries who had experienced Ebola in 2014 (Guiglo, Blolequin, Toulepleu, Duekoué, Bangolo, Kouibly, Danané, Man, Zouan- Hounien, Ouangolodougou. Challenges 18. Despite the above-mentioned achievements and results, the evaluation observed institutional, programmatic and operational challenges. Namely: a) The mobilization of counterpart funds suffered a remarkable delay. It took over around a year to make the funds available to pay for the subsidies to the health establishments implementing the PBF; b) The project had to support the implementation of the PBF in five health districts financially supported by the Fonds Mondial, due to lack of enough resources to cover implementation of the PBF in the five districts. The PRSSE displayed significant efforts to cover the financial needs (in payment of subsidies and implementation of the PBF) for these additional five districts; Page 67 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) c) The national budgetary principles and practices, in particular the procedures of the Integrated Management System of Public Finance (Système Intégré de Gestion des Finances Publiques - SIGFIP), created difficulties at certain points during implementation, to render available financial resources. In fact, the SIGFIP closes its activities during the November-January period, and activities planned during that period can’t be executed if resources can’t be mobilized; d) The underestimation of the budget for the sub-component on « rehabilitation of essential infrastructures » did not allow to achieve the expected results (560 health facilities rehabilitated), despite the adjustments undertaken to ensure an optimal use of the resources. 19. The implementation of the Performance Based Financing Strategy (PBF) faced challenges as well: a) The institutional changes experienced by the CTN-FBP did not allow for a global ownership of the roles and responsibilities by the CTN-FBP; This undermined its full involvement and coordination process during implementation. This situation limited its role in anticipating and resolving operational limitations and obstacles, mainly: i) The weak collaborative synergies between the different actors, to ensure a constructive complementarity during implementation; ii) The delay experienced in launching the work of the Verification Agencies (Agences de Contrôle et de Vérification), due to the delayed signature of the contracts; iii) The difficulties to incorporate the results of the community-based verification, into the calculations of the health establishments performance, due to the lack of parameters in the online data management portal for PBF; iv) The difficulties that health facilities experienced to complete their business plans and transmit them within the time period established in the PBF operational manual. Sustainability 20. Despite the challenges encountered, there are observable elements (as achievements) that allow us to expect the sustainability of the PRSSE interventions. 21. In terms of the political dimension, there is evidence of the strong and sustained government engagement and commitment to the PNDS objectives, after nine years. This was shown by the sustained trend to transfer the choice of investments to the health establishments at local level and to transfer a larger allocation of budgetary resources to the regions and districts, under the impulse of the PBF. This commitment was shown as well through approval of an exemption granted to health centers for the opening of their own bank accounts to receive resources, to facilitate the implementation of the PBF. This constitutes a strong political signal to ensure the sustainability of the PBF. 22. Finally, the government’s commitment to ensuring the sustainability of the project’s achievements was reflected in the new health Project SPARK, which provides continuity and follow-up to the PRSSE’s interventions. The SPARK’s development objectives focus on improving the utilization and quality of health service to reduce maternal and child morality in the Ivory Coast Côte. The Government is committed to reducing the weight of external financing in the health sector, implementing different strategies to mobilize domestic financial resources. Page 68 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 23. Another element supporting the sustainability of the PRSSE interventions is the engagement, involvement and availability of stakeholders and beneficiaries throughout implementation. This reflects the joint willingness to change their way of doing things and to contribute respectively to the achievement of the project objectives. The work-in-synergy involving all different actors (the top management of the regional and departmental health directorates, the heads of the health facilities, the COGES and the ACVs) during the implementation of the PBF open up a path to sustainability of the Project’s interventions. 24. The implementation of the PBF created an opportunity to redefine the COGES mission, with their greater involvement in the management of health care centers (participating in the elaboration and implementation of business plans, accompanying in the monitoring of expenses, overseeing the distribution of subsidies for individual « bonuses », and resources for regular functioning of the health centers). It is worth pointing out that reforms and new regulations will be required to give technical means and capacity to the COGES, to ensure they can carry out their roles and new responsibilities. This process brought to the center of the discussion the reflections concerning community participation in the process defining the health needs within the community and the ownership of the interventions related to the reforms, what benefits the implementation of the strategies and ensures the sustainability of the actions. 25. Regarding health providers, the PRSSE, through the PBF, represented an opportunity to foster leadership, a culture of performance, accountability, responsibility, to integrate the notion of quality in the provision of health services and to strengthen financial management capacity at this level. It supported the compliance with and implementation of existing protocols and directives, issued by the health authorities (mainly the MHSP), regarding patient care. 26. Regarding local regulators (DDS and DRS), the Project set at the center of the discussions, yet again, the reflections around the roles and responsibilities of the regions and districts, in the implementation of health reforms. It brought/meant the reaffirmation of the stewardship mission (des missions régaliennes) of the decentralized levels: governance, oversight, analysis of health data to make evidence-based decisions. 27. In terms of human resources, the PRSSE supported the execution of a capacity building plan, to strengthen certain capacities and know-how of different actors involved in implementation with the acquisition of new knowledge and skills on PBF (regulations and procedures on procurement, preparation of business plans, etc.). The strengthening of the supervision frequency and capacity during the life of the project contributed to a transformation of professional practices, emphasizing the quality of care for patients. This allowed for the sequenced transfer of responsibilities, contributing to consolidate the project achievements, through ownership, acquisition of new know-how, a culture of « public good », a transformation of vision with the sector, an understanding of the future challenges and a capacity to adapt to a changing context of health reforms such as the PBF. Strengthening stakeholder’s capacity consolidates ongoing support and foresees the achievement of the desired changes in the medium and long term. 28. The PRSSE implementation was an opportunity to strengthen the health infrastructure, through the rehabilitation of a number of primary health care centers, two warehouses to store equipment and medications, in the context of fighting Ebola, belonging to the INHP. The rehabilitations are part of the efforts to ensure sustainability, to ensure capacity to respond to the health needs of the population in particular to the needs to implement the Universal Health Coverage (UHI). Finally, the subsidies received under the PBF strategic framework, were used by most health establishments, to Page 69 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) strengthen and upgrade their technical capabilities and to improve the working conditions, depending upon their specific needs. 29. All these interventions aimed to strengthen the six pillars of the health system, which consolidates the achievements to streamline and sustain the interventions. Lessons Learned 30. Several lessons have been learned from the implementation of the PRSSE and they can be capitalized on, to strengthen the implementation of future projects. The project’s implementation relied on government/state institutions, which drew a lesson through planning. The planning of project interventions was based on national strategic plans, themselves inspired by the guiding orientations of the Plan de National de Development Sanitaire (PNDS). This contributed to carrying out activities that were aligned with the national health objectives, while capturing in a specific way, the needs to be fulfilled in each area of the health system (health infrastructures, SIS, social protection, governance, etc). This approach allowed regulatory/execution structures to strengthen their human resources and equipment capacities. This capacity building consolidated their institutional status and enabled them to fulfill their stewardship mission. 31. In line with its functions and principles, the PBF enabled the adoption of autonomous management of financial resources. In order to facilitate implementation in line with the PBF principles, the Ministry of Economy and Finance granted management autonomy through a special exception, giving health establishments the right to open a bank account. This derogatory provision (disposition dérogatoire) represents a remarkable step in the transfer of management autonomy to health establishments. 32. The lessons learned were observed during the project implementation, through best practices in the context of PBF framework. These best practices are to be built upon, to reinforce the results during the expansion phase, planned during the implementation of SPARK project. The example of the health centers in Bonon, Klan et Kayeta, in the health districts of Bouaflé and Sinfra are perfect illustrations. With the subsidies received, these centers developed strategies to improve the health of mothers and children, by encouraging women to do fours prenatal control visits; to give birth in the health centers, to ensure their children (0 to 11 months) received all of the vaccinations. The centers foresaw rewards when the protocols were respected, such as: a) Free distribution of the mother-child family book, for those women who went to the health centers during their first three months of the pregnancy; b) Free baby kits for those women who gave birth at the health centers, right after birth; c) An annual celebration for mothers (la fête des mères) to reward those women who attended the four prenatal controls during pregnancy, until labor, as recommended by WHO; d) Organization of a christmas party each (un arbre de Noël) for those children who have received all the required vaccines for the 0-11 months, in time. 33. This strategy contributed to improve several indicators: (i) the percentage of deliveries in health centers, assisted by qualified staff went from 40 percent in 2016 to 59 percent in 2018; (ii) the coverage of CPN4+ increased from 44 percent Page 70 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) in 2016 to 65 percent in 2018; (iii) the coverage of children fully vaccinated was over 92 percent in 2017. The year 2018 experienced a 100 percent coverage in the Klan health center. 34. The second lesson learned refers to the strategic collaborations crafted with other technical and financial partners during the project implementation. A strategic collaboration was established with UNICEF to acquire, equip and rehabilitate health facilities to implement the Ebola sub-component and the nutrition activities. These strategic partnerships contributed to achieve tangible and efficient results, during the implementation of the project. This is a result to build on and should be an inspiration for another project’s implementation. 35. Regarding the sustainability of the PBF strategy, while one may praise the possibilities offered by the PBF approach, it is important to acknowledge the challenges and constraints that weight over its implementation, which are not obvious to overcome. The PBF is built on a variety of instruments that, while they appear to be technical, are extremely sensitive and political. Thus, its institutionalization will pass by several profound mutations, such as the transformation of administrations, the adjustment of budgetary frameworks in public finances, new laws and decrees establishing the public prerogatives (les prérogatives des missions publiques) and the need to integrate the PBF with other ongoing health sector reforms such as the hospital reform and the UHI, which are a significant challenge for sustainability. 36. The scaling up of the PBF will require a strong mobilization of financial resources, not only to pay for subsidies, but also to cover the costs of bringing the health facilities up to standards and in compliance with the norms, and to be able to have the means for l’achat stratégique. It is undeniable to note that the contribution of the World Bank has been beneficial. Nevertheless, in the long run, there is a need for an appropriation of financial resources by the State, politically of strategic importance to ensure the sustainability of the PBF. This remains still a challenge, even if the Government is engaged and the mobilization of resources remains a priority. The recommendations below are part of this spirit of commitment to the success of the PBF strategy during the expansion phase. Recommendations to Authorities responsible for stewardship and supervision (autorités du tutelle) 37. The evaluation showed a less than optimal functioning of the CTN-FBP. The challenges of the PBF scaling up are high and a failure would be hard to deal with. Creating he conditions for optimal functioning of the CTN-PBF will require taking into account the decree N°0100/MSHP/CAB, dated 09 July 2018, responsible for the creation, attribution and functioning of the bodies/institutions implementing the performance-based financing: a) The first gap to be filled will consist in clarifying its mission and responsibilities, to ensure management stability; strengthening of its human resources staff with proven skills and professional profiles for the units described in the afore-mentioned decree; b) The General Directorate for Health, given its role in the PBF, must ensure that the units which compose the CTN-FBP can be quickly be set up and ensure their functionality; c) The General Directorate for Health must provide the cellule with the financial and logistical resources required for its operation (a budget line) to ensure it properly can carry out its responsibilities. The optimal functioning of the CTN-FBP will contribute to overcome the operational difficulties observed during the implementation of the PBF and it will contribute to achieve the expected results during the expansion phase. Page 71 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Recommendations for the PIU for World Bank Projects 38. The evaluation observed the considerable efforts carried out by the PIU on the coordination of the implementation of the project’s activities. The results achieved are the proof of this good coordination. Having stated this good performance, there are areas for improvement, such as strengthening the internal teamwork, strengthening the work among the different teams, in particular, the monitoring and evaluation team and the rest of the PIU units. Efforts should be made on procurement, to reduce the need for modifications to service contracts and cancellations of contracts. Page 72 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) 1. Project Appraisal Document (PAD964) 2. Credit Agreement IDA-5557 3. Grant Agreement IDA-D0030 4. Aide Memoires (2012-2019) 5. Project Restructuring Document September 2019 6. Implementation Status and Results Reports for the Project 7. Country Partnership Strategy 8. Cadre de Partenariat de Pays (CPF 2016-2019) 9. Multisectoral Nutrition and Child Development Project FY18 (P161770) 10. Plan National de Développement 2016-2020 11. Plan Nationale de Développement Sanitaire 2016-2020 12. Rapport Annuel de la Situation Sanitaire 2017 et 2018 13. UNDP Human Development Report (2018) 14. Project Operation Manual 15. Operational Manual for the Performance Based Financing (PBF) 16. PBF manuals prepared by the Agences de Contractualisation et Vérification 17. Cost Study on Universal Health Care (co-financed with WHO) 18. National PBF Strategy 19. Baseline survey for PBF 20. Plan du Travail et du Budget Annuel (PTBA) 21. Annual Review of PBF 20-23 January 2020. Yamousoukro 22. Survey SARA – 2016 23. Evaluation Study of the COGES – 2016 24. Étude sur la Qualité et les Facteurs d’Utilisation Rapide des Subsides FBP en Côte d’Ivoire (Septembre 2018) – PRSSE. 25. Étude relative à la libéralisation de la formation des agents de la sante ; 26. Étude sur l’adéquation de l’offre de formation des cadres de la santé avec les besoins sanitaires du pays. Studies financed by PRSSE for SPARK 1. Costing of Couverture Maladie Universelle 2. Study on population capacity to pay for health insurance premiums 3. Study on cost of service provision in public and private facilities 4. Infrastructure and equipment need of all facilities in 25 districts 5. Health Human resources training and distribution assessment; 6. A fiscal space for Universal Health Coverage analysis and 7. Qualitative and quasi-experimental impact evaluation of PBF. Page 73 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) ANNEX 7. KEY INDICATORS IN THE HEALTH DISTRICTS UNDER PBF 1. The graphs in this annex were prepared by the PIU monitoring and evaluation team, based on the data collected by PBF information system. They complement the ICR section II on Outcomes. I - Quantity indicators – Analysis of the indicators to measure the quantities of health services provided by health facilities under PBF. 1. Utilization of Health Services by Population Graph A - Utilization of Health Services by Population in the first 4 Districts under PBF 70% 59% 59% 60% 50% 52% 52% 47% 47% 50% 42% 42% 44% 43% 37% 40% 32% 25% 30% 20% 10% 0% T3-2016 T4-2016 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T4-2019 2016 2017 2018 2019 ANYAMA COCODY-BINGERVILLE BOUAFLE SINFRA Graph B - Utilization of Health Services by Population in the next 10 Districts under PBF 120% 100% 90% 90% 79% 82% 74% 77% 74% 80% 72% 69% 62% 64% 60% 40% 20% 20% 0% T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T4-2019 2017 2018 2019 BOUNA DABAKALA ISSIA MBAHIAKRO NASSIAN NIAKARAMADOUGOU PRIKRO TIEBISSOU VAVOUA YAMOUSSOUKRO Page 74 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 2. These tables are prepared using the number of consultations in each of the districts under PBF and the population as estimated by the National Institute of Statistics. Similar trends of utilization are observed for both groups of districts. The district of Bouafle experienced the greater increase (from 25 percent in 2016 to 59 percent during the last quarter of 2019). The district of Cocody-Bingerville experienced the weakest increase in the rate of utilization of health services. This district is located in the greater area of Abidjan, where there are numerous private health facilities, competing with public ones. Despite this, the rate of utilization of health services in public health care facilities under PBF increased in Cocody- Bingerville. Annual Rate of Increase Group of Districts 2016-2017 2017-2018 2018-2019 04 DISTRICTS 2,4% -0,3% 7,7% 10 DISTRICTS 1,0% 5,6% 2. Evolution of number of deliveries by trained health personnel in PBF Districts Graph C- Average number of deliveries per 1000 – First 4 Districts under PBF 40 35.25 37.65 34.43 33.11 35 26.86 28.32 30 25.27 25.93 26.56 23.99 23.32 23.87 23.77 25 20 17.27 15 10 5 0 T3-2016 T4-2016 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T4-2019 2016 2017 2018 2019 ANYAMA COCODY-BINGERVILLE BOUAFLE SINFRA Annual Growth Rate Group of districts 2016-2017 2017-2018 2018-2019 04 DISTRICTS 4,6% 5,9% -1,4% 10 DISTRICTS 5,3% 5,3% 3. The tables are calculated based on the number of deliveries to the population covered, estimated by the National Institute of Statistics. The graphs present the average number of deliveries per 1,000 people. There is a similar evolution in the average number of deliveries in the two group of districts. One observes a seasonal peak at the second trimester of the year, that coincides with the nine months after the second period of rains in the country. Anyama and Nassian are the two districts in the groups with the highest levels of assisted deliveries. Page 75 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Graph D - Average number of deliveries per 1000 – Next 10 Districts under PBF 45 37.87 38.47 40 34.86 36.11 32.51 33.92 35 31.39 31.56 29.23 30 24.66 25.77 25 20 15 10 5 0 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T4-2019 2017 2018 2019 BOUNA DABAKALA ISSIA MBAHIAKRO NASSIAN NIAKARAMADOUGOU PRIKRO TIEBISSOU VAVOUA YAMOUSSOUKRO 3.Evolution of the Number of Children fully vaccinated in District under PBF Graph E – Average number of children fully Vaccinated per 1,000 – First 4 districts under PBF 35 30 25 20 15 10 5 0 T3-2016T4-2016T1-2017T2-2017T3-2017T4-2017T1-2018T2-2018T3-2018T4-2018T1-2019T2-2019T3-2019T4-2019 2016 2017 2018 2019 ANYAMA COCODY-BINGERVILLE BOUAFLE SINFRA Page 76 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Graph F - Average number of children fully Vaccinated per 1,000 –Next 10 districts under PBF 40 36.96 35.02 35.98 35 31.79 30.62 30.66 30 27.86 25.16 25 20 16.50 16.69 15 11.68 10 3.62 5 0 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T4-2019 2017 2018 2019 BOUNA DABAKALA ISSIA MBAHIAKRO NASSIAN NIAKARAMADOUGOU PRIKRO TIEBISSOU VAVOUA YAMOUSSOUKRO Annual Growth Rate Group of districts 2016-2017 2017-2018 2018-2019 04 DISTRICTS 12,4% 19,3% -4,8% 10 DISTRICTS 27,3% 19,4% 4. The graphs show that the growing trends since the implementation of PBF. The first 4 districts saw the average annual growth rate slow down between 2018 and 2019, while the same rate continued to grow in the other 10 districts under PBF. Together, the average annual growth rate reached 14 percent between 2018 and 2019. The district of Bouafle presents better achievements, as it is supported by a very engaged regional directorate, very involved in PBF activities. Page 77 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 4. Evolution of the number of Children receiving Vitamin A in PBF districts Graph G- Average number of Children receiving Vitamin A – First 4 PBF districts 120 95.62 98.38 100 87.53 83.93 79.58 81.39 80 56.04 60 33.67 40 6.94 7.26 11.52 20 0.07 1.51 4.45 0 T3-2016 T4-2016 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T4-2019 2016 2017 2018 2019 ANYAMA COCODY-BINGERVILLE BOUAFLE SINFRA Graph H- Average number of Children receiving Vitamin A – Next 10 PBF districts 200 172.63 180 166.69 160 140 107.27 108.02 120 100 73.19 80 60 28.59 40 20 0.00 0.00 0.00 0.00 0.00 2.20 0 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T4-2019 2017 2018 2019 BOUNA DABAKALA ISSIA MBAHIAKRO NASSIAN NIAKARAMADOUGOU PRIKRO TIEBISSOU VAVOUA YAMOUSSOUKRO Annual Growth Rate Group of districts 2016-2017 2017-2018 2018-2019 04 DISTRICTS 5,0% 74,8% 28,7% 10 DISTRICTS 64,8% 58,4% 5. The graphs show the number of children who received vitamin A supplement to the population of each district. The supplementation of vitamin A for children ages 6-59 months was done in annual campaigns until 2017. The remaining Page 78 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) doses were distributed during regular health visits. With PBF implementation, the national nutrition program started in 2018 a pilot to administer vitamin A during regular checkups. This approach was piloted in PBF districts supported by the Project, resulting in improved nutrition results in these districts. 5. Evolution of the number of women who had Prenatal controls during the first trimester by PBF district Graph I - Number of Women who had their first prenatal control during the first trimester of pregnancy per 1,000 - First 4 PBF Districts 45 39.04 37.49 40 34.01 35 27.53 30 22.83 23.13 25 20.54 17.39 16.72 16.99 16.63 17.37 20 14.54 12.41 15 10 5 0 T3-2016 T4-2016 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T4-2019 2016 2017 2018 2019 ANYAMA COCODY-BINGERVILLE BOUAFLE SINFRA Graph J - Number of Women who had their first prenatal control during the first trimester of pregnancy per 1,000 - Next 10 PBF Districts 40 33.91 34.97 35 32.37 30 25 18.94 18.79 20 15 9.00 9.23 10.00 9.84 10.10 8.56 10 5 0 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T4-2019 2017 2018 2019 BOUNA DABAKALA ISSIA MBAHIAKRO NASSIAN NIAKARAMADOUGOU PRIKRO TIEBISSOU VAVOUA YAMOUSSOUKRO Page 79 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 6. Initially, there were some difficulties with the comprehension of this indicator, which was not properly reported. This was overcome during implementation and it did not have effects on the amounts of the subsidies. II - Quality indicators – Analysis of the technical quality of the health care and services in the health facilities under PBF. 7. To analyze the evolving trend of the technical quality of health services, the PIU monitoring and evaluation team carried out a linear adjustment of the data series of average quality scores. This adjustment generated an R2 for each of the two series (for the first 4 districts and the next 10 districts under PBF). The higher the R2 (close to 1), it is possible to conclude that the series is well adjusted, with this straight linear regression. A. Evolution of overall technical quality Graph K. Average quality score for all districts under PBF (in percentage) 100.0 y = 2.4838x + 55.442 R² = 0.8808 90.0 80.0 70.0 y = 4.5699x + 24.218 Average quality score 60.0 R² = 0.9205 50.0 04 districts 40.0 10 autres districts 30.0 20.0 10.0 0.0 1 3 5 7 9 11 13 15 Period: Trimester 1 to 15 8. The quality of health facilities has been measured every quarter since the beginning of the PBF implementation, for 15 trimesters (August 2016 – January 2020). The analysis showed a R2 at 0.9 percent. Thus, the quarterly improvement rate for the first 4 districts went from 2.48 to 88 percent and from 4.56 to 83 percent for the next 10 PBF districts. The first 4 districts were monitored individually, given their number and the evaluation tools were revised during the period of incorporation to the PBF by the next 10 districts. Page 80 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) B. Evolution of Quality indicators by type of health establishment Graph L. Evolution of quality score in Primary Healthcare Facilities (%) 100.0 y = 2.4338x + 57.58 R² = 0.8428 90.0 80.0 Average quality score 70.0 y = 4.4872x + 26.8 60.0 R² = 0.9014 50.0 04 districts 40.0 10 autres districts 30.0 20.0 10.0 0.0 1 3 5 7 9 11 13 15 Trimesters Graph M - Evolution of the quality score in General hospitals (%) 100.0 y = 2.221x + 57.53 R² = 0.6322 90.0 80.0 70.0 Average quality score 60.0 y = 3.51x + 30.01 R² = 0.952 50.0 04 districts 40.0 10 autres districts 30.0 Recruitment of new ACV 20.0 Civil servants strike Revision of tools 10.0 0.0 1 3 5 7 9 11 13 15 Trimesters Page 81 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 9. A similar analysis was carried out to show the evolution of the average quality score by type of health facility. For primary healthcare facilities, the quarterly improvement rate for the average quality score was 2.43 in the first 4 districts and 4.48 for the next 10 districts. The score evolved to reach 88 percent during the last quarter of 2019 for the first 4 districts and 83 percent for the next 10 districts. Analyzed individually, the trend evolves faster for general hospitals than for primary healthcare centers. The reason is there are a number of primary healthcare facilities that masque the trend exhibited by the hospitals under PBF. For the general hospitals, the improvement rate of the average quality score was 2.21 for the first 4 districts and 3.51 for the next 10 districts. By Project completion, the score was 90.1 percent for the first 4 districts and 75.3 percent for the next 10 districts. The main reason for the lower score is that certain general hospitals did not have the capacity to offer the full minimum package of services, but they were still evaluated using indicators of service availability (for example: functional operating rooms or laboratories). This was the case of some urban primary health care facilities that were re-categorized as general hospitals without the full capacity to provide all of the services in the minimum package. 10. Some of the events that negatively affected the evolution of the quality score were: (i) the 2017 civil servants’ strike; (ii) the revision of the evaluation tool to incorporate needs and observations by some programs and consider key aspects of the health system. The indicators incorporated were linked to the availability of blood, planning, emergency admissions, radiology and dental care; and (iii) the time elapsed between the end of the contract of the first ACV and hiring the new local ACVs. III – Evolution of Quality indicators for some aspects of Quality in Primary Healthcare Facilities Graph N - Quality in Primary Healthcare Facilities – First 4 districts under PBF 100.0 87.8 90.0 81.9 82.3 84.7 81.4 78.7 77.5 80.0 73.8 68.5 70.0 61.2 61.8 61.8 57.6 60.0 47.5 50.0 40.0 30.0 20.0 10.0 0.0 T3-2016 T4-2016 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T4-2019 2016 2017 2018 2019 Disponibilité des Médicaments Traceurs Hygiène, Sécurité, Environnement Maternité Pharmacie Prise en charge du Paludisme Graph O - Quality in Primary Healthcare Facilities – Next 10 districts under PBF Page 82 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 120.0 94.0 95.6 100.0 88.5 91.8 90.2 91.1 86.8 79.6 80.3 75.0 72.2 80.0 69.4 60.0 40.0 20.0 0.0 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T4-2019 2017 2018 2019 Disponibilité des Médicaments Traceurs Hygiène, Sécurité, Environnement Maternité Pharmacie Prise en charge du Paludisme 11. The graphs show the evolution of some relevant areas related to the quality of health care for primary health care facilities for the two groups of districts. Each quality area shows an improvement trend for all districts. This reflects the use of PBF resources to improve healthcare facilities: through acquisition of trash cans, illumination equipment, soap, access to water and latrines. The ACVs accompanied the health centers, contributing to consolidate the achievements. It is important to highlight that the treatment of malaria experienced a significant improvement, reflecting that the training activities carried out by the malaria program were reinforced by the Project with support to health districts to accompany health facilities to implement activities and carry out this work. Graph P - Quality in general Hospitals – First 4 districts under PBF 70% 59% 59% 60% 50% 52% 52% 47% 47% 50% 42% 42% 44% 43% 37% 40% 32% 25% 30% 20% 10% 0% T3-2016 T4-2016 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T4-2019 2016 2017 2018 2019 ANYAMA COCODY-BINGERVILLE BOUAFLE SINFRA Page 83 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) Graph Q - Quality in general Hospitals –Next 10 districts under PBF 120.0 94.0 96.0 94.0 95.0 100.0 88.5 83.9 80.6 81.5 80.0 72.5 69.5 68.5 68.5 60.0 40.0 20.0 0.0 T1-2017 T2-2017 T3-2017 T4-2017 T1-2018 T2-2018 T3-2018 T4-2018 T1-2019 T2-2019 T3-2019 T4-2019 2017 2018 2019 Consultation Externe/ Urgences Disponibilité des Médicaments Traceurs Maternité 12. As it was the case for primary healthcare, general hospitals experienced an improvement trend for quality of health care. With the revision of the set of quality indicators in the second half of 2017, the score went down for hospitals in the first four districts. Despite challenges, several hospitals implemented strategies and used the PBF funds to improve the quality of health care, which translated in improve trends. Page 84 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) ANNEX 8. GEOGRAPHIC AREAS OF PROJECT INTERVENTION Intervention Districts 1 Intervention Districts 2 Full Package REGIONS Partial Package PBF + Rehabilitations Rehabilitations 1. Anyama 15. Abobo Ouest ABIDJAN 2 2. Cocody Bingerville 16. Koumassi-Port Bouet - Vridi 3. Bouaflé 17. Zuenoula MARAHOUE 4. Sinfra 5. Issia 18. Daloa HAUT-SASSANDRA 6. Vavoua 7. Bouna 19. Bondoukou BOUNKANI-GONTOUGO 8. Nassian 20. Tanda 9. M’Bahiakro 21. Bocanda N’ZI IFOU 10. Prikro 22. Daoukro 11. Tiébissou 23. Didiévi BELIER 12. Yamoussoukro 24. Toumodi 13. Dabakala 25. Katiola HAMBOL 14. Niakaramadougou Districts added during Implementation, supported by the Fonds Mondial Regions Districts 15. Buyo NAWA 16. Meagui 17. Soubre. SAN PEDRO 18. San Pedro SUD COMOE 19. Grand Bassam Notes: (i) the first four districts started to work under PBF in 2016; (ii) the last five districts added during implementation received support (subsidies), initially by the Global Fund l and later on by the Project. The Project supported these five districts throughout the life of the project through capacity building activities, technical assistance, monitoring and evaluation. These five districts were initially three, but were subdivided in five districts, after an administrative decision. These five districts did not received funds for rehabilitations of health facilities. Page 85 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) ANNEX 9. SUMMARY OF THE FIELD VISIT DURING THE ICR MISSION Mission ICR - Visite de Terrain (27 et 28 janvier 2020) I - Région de Marahoué – Mardi 27 janvier 2020 Réunion à Zuenoula 1. La mission s’est déplacée à Zenoula pour rencontrer le Directeur Régional et les Directeurs des Départements de Bouaflé et Sinfra, qui étaient en réunion de bilan et planification pendant la dernière semaine de Janvier. L’équipe de l’ACV locale a participé également dans la réunion. Bouaflé et Sinfra sont deux districts qui ont fait partie des districts PBF dans la phase pre-pilote qui a démarré en août 2016. 2. Le Directeur Régional était arrivé récemment à la région de Marahoué, mais il avait travaillé auparavant dans une autre région sous PBF. Les Directeurs départementaux ont été dans leur postes depuis le début de l’expérience PBF. 3. Les régulateurs (DDS/DR) ont évalué très positivement les réalisations de la mise en œuvre du PBF. Les principaux commentaires étaient : a) Le PBF a été révolutionnaire en ce qui concerne la performance, la prise en charge des patients et la qualité ; Travailler avec un plan de affaires a transformé la qualité des services d’une façon mesurable et identifiable. b) Le PBF a permis des changements appréciables dans : (i) les conditions de travail, avec la réhabilitation d’infrastructures dans des établissements qui n’avait eu aucune intervention depuis leur construction initiale ; (ii) la disponibilité des intrants pour traiter les patients ; (iii) l’amélioration des conditions d’hygiène dans les établissements. c) Parmi les aspects plus révolutionnaires, les régulateurs ont mentionné la prise d’initiative par les établissements de santé, créant des stratégies pour augmenter l’utilisation des services dans la population et achever leur objectives. Parmi les exemples mentionnés : l’installation des panneaux pour annoncer la proximité des centres, l’engagement des services de transporte (en moto) pour emmener les femmes enceinte aux centres de santé, le déplacement du personnel a un hôpital pour apprendre comment performer un traitement ou intervention (selon leur initiative), cherchant la formation d’une façon active. d) Les établissements ont placé beaucoup d’efforts sur la mise au niveau des bâtiments, pour augmenter l’hygiène, l’intimité pour les femmes enceintes et satisfaire les besoins des patients, avec : (i) la création des toilettes fonctionnelles et divers, pour le personnel et les patients ; (ii) l’établissement de zones pour que les femmes qui viennent d’accoucher puisse se laver et laver le linge. Ces changements ont eu un effet direct sur l’augmentation des accouchements dans les centres. Ils ont contribué à attirer les femmes vers les établissements de santé, en les motivant a passer les contrôles prénatales. e) Une fois qu’ils disposaient des ressources, les établissements ont pris des initiatives pour réaménager les espaces de travail et trouver de la place pour les différents services (par exemple, service de nutrition, les maternités) et rapprocher le dispensaire et la maternité dans certains établissements pour rendre plus facile l’accès et l’utilisation des services aux patients. f) L’amélioration de la gestion de déchets a été mentionné comme l’une des aspects révolutionnaires, soutenu par le PBF. Page 86 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) g) Le PBF a contribué au renforcement des capacités du personnel : au début avec la formation de base pour la démarre du PBF et l’accompagnement des ACVs. Les compétences acquises et/ou renforcées se concentrent en matière de planification d’activités, préparation des plans d’affaires, suivi et évaluation. h) Le PBF a apporté une amélioration de la gouvernance : il a contribué à développer un sens de responsabilité face au travail dans le personnel des établissements, appuyés par les vérifications des ACV. i) Le PBF a contribué au changement d’attitude du personnel : plus motivés, et prêts à donner un meilleur accueil à la population. Les vérifications de présence ont aidé, mais le PBF a créé un intérêt qui n’était pas présent avant. Avec le PBF, les établissements ont adopté des évaluations individuelles, en utilisant la grille du Ministère de Sante, avec de critères d’évaluation et présence claires. j) Fidélisation des agents de santé, qui ont choisi rester dans le système de santé et dans les établissements sous PBF. Les régulateurs ont mentionné qu’ils connaissent des cas d’étudiants qui veinent de compléter leurs études, qui ont choisi d’aller aux centres sous PBF, plus éloignés géographiquement– un choix, avant PBF, inimaginable. k) Le PBF a aidé à renforcer la planification d’acquisition et entretient des médicaments dans les établissements : avec la création et gestion des inventaires des médicaments, en réduisant les ruptures et les médicaments périmés. 4. Parmi les défis les plus remarquables, les régulateurs ont mentionné : a) La gestion financière des ressources : la réception et l’utilisation des subsides ont présenté des défis. Les difficultés opérationnelles ont influencé la gestion financière. Il y fallu surmonter les obstacles pour dépenser l’argent, en trouvant des solutions pour les difficultés qui se sont présentés à cause de la réalité dans le terrain (exemple de manque d’offres de certains services, pour avoir trois propositions dans la liste courte). La formation dans la gestion financière simplifiée et l’accompagnement des régisseurs ont aidé à maitriser les obstacles. Il y a eu des retards a tous les niveaux. Le nouveau manuel du PBF contient des éléments pour prévenir les retards (par exemple, sanctions). Visite du Centre de Sante Urbain (CSU) à Bonon : 5. Le CSU de Bonon a commencé à travailler sous PBF depuis 2017. La mission a maintenu une réunion avec le directeur du centre et son équipe et elle a visité en détail les espaces physiques du centre. Parmi les principaux acquis, grâce à la mise en œuvre du PBF, les suivants ont été relevés : a) Les ressources PBF ont permis au centre d’améliorer de façon notable, l’infrastructure et les conditions du travail. Selon la direction du CSU, la réhabilitation de l’infrastructure était une priorité pour augmenter l’utilisation et la qualité des services de santé. Les travaux ont amélioré la salle d’accouchement (qui était délabrée et n’avait pas accès d’eau) ; les ressources PBF ont permis l’installation de carrelage (pour améliorer l’hygiène en général), l‘équipement des bureaux, l’installation de la climatisation et l’amélioration de la qualité du raccordement électrique). La mission a observé l’aménagement d’une zone extérieure avec accès à l’eau pour laver et sécher le ligne. Le bâtiment de maternité a été visiblement réhabilité, maintenu propre, avec de bureaux individuels pour chaque sage-femme, ce qui a permis de l’intimité pour les femmes enceintes, pendant leurs visites. C’est aménagement a été très apprécié par la population. Après les travaux, la maternité a des Page 87 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) toilettes pour les femmes patientes et pour le personnel. C’est « upgrade » a été, également, très apprécié. Les changements dans la maternité ont contribué à augmenter les accouchements dans le CSU. La direction du CSU a entretenu des séances de sensibilisation du personnel sur l’importance de maintenir l’infrastructure. b) La qualité des donnes a beaucoup amélioré, selon l’équipe du CSU. Les donnes ont été utilisés pour la planification et le suivi les progrès et les objectives, les indicateurs dans le plan d’affaires. La mission a remarqué l’importance octroyé a cette activité. c) Parmi les défis, l’équipe de management a mentionné les activités de planification. La planification était difficile la première année ; il manquait l’expérience pour préparer le plan d’affaire ; et au début, c’était un défi utiliser le plan d’affaires. Mais l’année suivante, l’équipe a choisi un nombre limité d’indicateurs comme cible pour commencer a travailler, pour atteindre les objectives établis. Après ça, le personnel est sorti à chercher la population dans leur communes (en travaillant avec les ASC, le COGES, et les autorités dans la ville et communautés plus éloignées). d) La prise en change était un défi : mais ils ont avancé dans la stratégie, et ils continuent à travailler avec détermination. e) Le CSU a construit des alliances avec des acteurs publics et privés pour renforcer la capacité du centre à offrir des services de santé de qualité. La visible bonne performance du CSU a séduit la mairie qui a décidé de financer l’encerclement de la propriété. Également, le CSU a recu du financement du secteur privé pour la construction d’un bâtiment dans le contexte du CSU. f) Travailler sous PBF (FBP) a déclenché la créativité et l’initiative dans le personnel du CSU pour se rapprocher de la population (comme la reproduction des calendriers pour l’année 2020, avec les services de sante offerts par le CRU et information de santé en général – UHI, vaccinations, visites CPN, etc. Le CSU s’appuie régulièrement sur les ASC, en leur donnant des matériaux pour l’ASC et pour emporter aux communautés, lesquelles son de accès difficile. g) Le CSU a profité et cherché des synergies parmi toutes les ressources possibles : Les ressources PBF ont aidé à catalyse d’autres fonds et elles ont financé d’autres projets (par exemple, un projet de sante maternelle ou les ressources ont appuyé la formation des sages-femmes. h) Le CSU a aménagé un espace pour la prestation des services de nutrition : cela a permis s’occuper des enfants qui souffraient de malnutrition. i) La collaboration avec l’ACV s’est améliorée depuis le début. Le CSU a bien travaillé avec l’ACV locale depuis Octobre 2019. j) L’ACV a confirmé la grande appropriation du CSU de Bonon du PBF. A Bonon, tous les ressources (PBF, ressources propres, budgétaires) ont été inclus dans le plan d’affaires. Le CSU fait un rapport d’utilisation des ressources. II- Région de Haut Sassandra Réunion à Issia 6. La mission s’est déplacée à Issia pour rencontrer le Directeur Régional de Haut Sassandra et les Directeurs des Départements de Issia et Vavoua, qui étaient en réunion de bilan et planification pendant la dernière semaine de Janvier. L’équipe de l’ACV locale a participé également dans la réunion. Issia et Vavoua ont commencé à travailler sous PBF en 2017. Page 88 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 7. En général, les observations sur les acquis du PBF et du PRSSE ont été positives. Parmi les acquis, les participants ont mentionné : a) Il y a eu un Renforcement des capacités : avec plus des ressources disponibles pour les établissements de santé, ils ont réussi a améliorer les conditions de travail (infrastructure, climatisation, intrants, motivation) ; b) Les PBF a créé des conditions pour des Innovations dans les établissements : le PBF a permis un espace pour la gestion dans les centres de santé qui n’existait pas avant. Et les équipes de gestion dans les centres ont déployé leur initiative et créativité pour achever leur objectives. c) La Motivation du personnel s’est améliorée : ce qui donne come résultant que le personnel reste au travail pendant la journée, arrive ponctuelle et déployai un meilleur traitement aux patients. d) L’utilisations des services a augmenté. e) Au niveau des régulateurs (DRS et DDS) : il y a eu plus d’accompagnement aux centres de santé ; la qualité de leur travail s’est améliorée également ; les régulateurs suivent l’évolution de la qualité, et ils utilisent les donnes de qualités pour informer leurs décisions a future. Comment ont les donnes été utilisés ? Pour la suivie et l’accompagnement des structures sanitaires, pour leur donner appuie dans la qualité des donnés qu’ils préparent ainsi que l’appui dans l’atteinte des résultats. f) Parmi les défis : Il y a eu de la flexibilité pour faire comprendre à la Cellule Technique National et aux régisseurs quand il fallait faire une activité financée par des ressources PBF. Il a fallu trouver comment flexibiliser et apprendre à travailler dans les guides opérationnelles mais, en s’adaptant à la réalité de chaque localité. g) Le PBF a facilité un rapprochement des populations, l’amélioration de la qualité des soins : avec la disponibilité des intrants, parmi d’autres éléments. 8. La DDS de Issia a remarqué comme achèvements : a) La net amélioration de la performance des prestataires, avec l’utilisation des outils de gestion. Avec la disponibilité de meilleurs outils de gestion : les prestataires ont réussi à faire un meilleur remplissage des outils et un meilleur archivage. Spécialement dans le cas des pharmacies, il y a eu une gestion du stock de médicaments, avec un bon entretien des inventaires. Il y a eu une meilleure gestion des déchets dans les hôpitaux et les centres de santé en général. Par conséquent, le taux de fréquentations de la population s’est amélioré. (Exemple de CPN 1 : qui a passé de 0% A 10%). En général, les indicateurs de quantité et de qualité ont amélioré. Ainsi que les capacités du personnel qui préparent les donnes. b) Travail avec l’ACV : l’expérience du travail avec l’ACV a été positive en général. Le personnel des ACV est local et ils ont bien travaille avec les centres de santé dans les activités de planification. Il a fallu plus d’accompagnement que prévu initialement. c) Les ACV ont travaillé de prêt avec les centres de santé dans la préparation, approbation, validation des plans d’affaires. Les plans d’affaires ont permis de travailler ensemble (aux centres de santé, ACV staff, CTN-PBF), dans un contexte complètement nouveau. Les acteurs ont travaillé sur la planification et l’utilisation des ressources, qui sont verses sur les comptes des centres de santé, pour la provision des services. L’ACV a accompagné la mise en œuvre. d) A la question sur la possibilité d’améliorer la qualité dans les établissements de santé, qui ne sont pas sous PBF, la réponse a été : il faut avoir les incentives que le PBF porte pour renforcer et améliorer les conditions Page 89 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) du travail, améliorer le plateau technique et pour motiver le personnel. Le PBF a apporté les ressources pour pouvoir améliorer la performance. e) Travail avec le COGES : la participation du COGES était difficile au début, mais il s’est amélioré à fur et à mesure de la mise en œuvre. Il a fallu plus d’accompagnement du personnel de l’ACV, qui est local. Comment ils se connaissent, ils travaillent bien ensemble. f) En tant que défis, ils ont souligné : (i) la mobilité du personnel a été un défi ; (ii) les ressources du PBF sont arrivés en retard, ce qui a pénalisé les établissements. Les ressources pour l’évaluation de la qualité ne sont pas reçues en temps, et les établissements ont utilisés leurs propres moyens pour cette activité ; (iii) il y a eu des centres qui n’ont pas dédié assez de ressources pour améliorer les équipements techniques ; (iv) difficultés dans l’utilisation des ressources PBF. des difficultés opérationnelles initiales ont été surmonté avec l’appui et l’accompagnement de l’ACV (avec les ordres de paiement, rejetées par le régisseur). g) Leçons : La mise en œuvre du PBF a contribué à la transformation des pratiques dans les établissements de santé (suivi évaluation, gestion de la qualité). Il faut documenter les bonnes pratiques déjà mise en place, créés par des acteurs, dans cette phase initiale du PBF en Côte d’Ivoire. On a discuté sur la possibilité d’avoir une plateforme ou similaire, ou partager les expériences pendant la phase pilote, pour accompagner la mise en échelle. Pendant la phase pilote, les acteurs ont acquis une expérience significative dans la qualité en réhabilitation, accueil, attitude vers la population, importante pour les résultats, la satisfaction et la fidélisation des patients. Plusieurs exemples ont été mentionnés : l’accompagnement de l’arrivée aux centres de santé, la tenue du personnel, pour rendre les structures sanitaires plus humaines. Centre de Sante Rural (CSR) a Broma 9. La mission a visité le CSR au village de Broma, dirige para un infermière chef, avec une équipe intégrée par un infermière et une sage-femme. Pendant la visite, la mission s’est réunie avec le président du COGES et le chef du village ainsi qu’avec de femmes avec leur enfants, présentes au centre pour la vaccination. Le Projet PRSSE a construit une pompe a motricité humaine (PMH), puis que le CSR était privé d’eau potable. L’infirmière chef était au CSR depuis six mois mais il venait d’un autre établissement qui travaille avec PBF. 10. Les principaux acquis avec les ressources PBF: a) La mise au niveau de l’infrastructure de ‘établissement (climatisation, équipement mobilier, TV pour faire de sessions formatives, printers, aménagement de la salle d’accouchement, intrants médicaux, panneaux de signalisation pour le centre sur la route, plusieurs éléments d’hygiène). b) Depuis ces améliorations, le centre est plus fréquenté avec 400-500 visites dans le mois de janvier 2020. c) Un acquis important pour le centre de santé est l’amélioration de la qualité des données. L’équipe a beaucoup travaillé sur ce point, ce qui a rendu la préparation des rapports beaucoup plus facile (rapports des dépenses à transmettre à la DD). L’équipe a bien compris les différences dans la préparation des indicateurs pour le système d’information PBF et pour système d’information de sante national. d) L’expérience de travail avec l’ACV s’est bien passe : par exemple, l’ACV a bien bonne explique les ressources supplémentaires de location (et cela a évité des problèmes avec la présentation des données). Page 90 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) 11. Parmi les défis et besoins, ils ont mentionné: a) L’infirmier chef n’avait été forme dans la gestion financière (la mission recommande que l’ACV accompagne et forme l’infirmier. La sage-femme a également besoin de formation. b) Le centre a des problèmes à maintenir la gratuite des accouchements, à cause du manque de kits d’accouchements. c) La communauté a des problèmes d’accès à l’eau et il manque une ambulance. Le CSR couvre plusieurs communautés éloignés et d’accès difficile à cause des voies. d) Le CSR a eu des problèmes au début pour utiliser les ressources additionnelles du PBF, ce qui est normale (des défis opérationnels qui arrivent quand on utilise des nouvelles procédures). 12. Parmi les acquis et changements positives, on a mentionné : a) Le CSR a commencé à faire des Consultations foraines pour se rapprocher de la population. b) Il s’est appuie sur les ASC pour connecter avec les femmes et les informer, pour les convaincre de venir au CSR pendant le premier trimestre de grossesse. Hôpital General de Issia 13. La mission a visité L’hôpital général d’Issia, qui a travaillé sous le PBF depuis 2017. Les principaux acquis mentionnés pendant la visite ont été : a) L’amélioration de la gestion des déchets avec la construction des fosses a placenta et une autre fosse pour d’autres déchets dangereux et la fosse à bruler des déchets non-dangereux. Il est possible pour des établissements dans la région d’envoyer des déchets à cet hôpital. b) Le changement des pratiques avec le remplissage des outils des données, ce qui a achevé la meilleure qualité des données de l’hôpital : L’équipe de gestion a vu une différence, au moment de prendre des décisions. c) Le rapport avec la ACV fonctionné correctement et ils se sont appuyés sur l’avis technique et l’accompagnement dans la mise en œuvre des activités. d) Le PBF a permis l’amélioration des conditions de travail et des conditions d’hygiène. Cela était assez évident dans les zones extérieures et dans certaines salles a l’hôpital. Pendant qu’Il reste encore du travail à faire, en particulier dans les salles ou les patients sont accueillis. Les conditions observes doivent être améliorées, comme la mission a discuté avec les responsables de gestion. Néanmoins, les ressources additionnelles ont permis l’Installation des moustiquaires dans les salles. e) En tant que défi, l’hôpital a eu des retards dans l’utilisation des ressources depuis le début de la participation dans le PBF. Pendant la visite, la mission a constaté que le retard persiste. f) Quand les ressources PBF ont été utilisés : l’hôpital a démontré capacité de décider les priorités auxquelles dédier les ressources. L’équipe de gestion a remarqué que l’utilisation des ressources est plus facile et rapide qu’avec les ressources traditionnelles de l’État. La provision des services est réalisée dans un délai de temps plus court, et les entreprises savent que le paiement sera plus rapide. g) L’hôpital a eu moins ruptures avec l’inventaire des médicaments et une meilleure gestion, avec moins de médicaments périmés. Page 91 of 92 The World Bank Health Systems Strengthening and Ebola Preparedness Project (P147740) h) Les fonds PBF ont permis : (i) faire des badges d’identification pour le personnel ; (ii) améliorer l’arrivée de l’eau a l’hôpital ; (iii) améliorer les conditions de travail, d’hygiène, en motivant le personnel ; (iv) Plus de collaboration avec le COGES et (v) achever plus de références d’autre centres de santé. Page 92 of 92