Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004327 IMPLEMENTATION COMPLETION AND RESULTS REPORT (TF-96812, TF-15054, TF-12691, IDA-H8770) ON AN AFRICA CATALYTIC GROWTH FUND GRANT IN THE AMOUNT OF SDR 18.47 MILLION (US$ 25.69 MILLION EQUIVALENT) AN INTERNATIONAL DEVELOPMENT ASSOCIATION GRANT IN THE AMOUNT OF SDR 9.35 MILLION (US$ 13.0 MILLION EQUIVALENT) AND A MULTI-DONOR HEALTH RESULTS INNOVATION TRUST FUND GRANT IN THE AMOUNT OF SDR 3.60 MILLION (US$ 5.0 MILLION EQUIVALENT) TO THE REPUBLIC OF SIERRA LEONE FOR THE REPRODUCTIVE AND CHILD HEALTH PROJECT - PHASE 2 ( P110535 ) 8-Dec-2017 Health, Nutrition & Population Global Practice Africa Region CURRENCY EQUIVALENTS (Exchange Rate Effective {June 30 2017}) Currency Unit = Leone (Le) Le 7427 = US$1 US$ 1.39 = SDR 1 FISCAL YEAR July 1 - June 30 Regional Vice President: Makhtar Diop Country Director: Henry G. R. Kerali Senior Global Practice Director: Timothy Grant Evans Practice Manager: Gaston Sorgho Task Team Leader(s): Irina Aleksandra Nikolic ICR Main Contributor: Adanna Deborah Ugochi Chukwuma ABBREVIATIONS AND ACRONYMS ACGF Africa Catalytic Growth Fund AFDB Africa Development Bank ANC Antenatal Care CDC Centers for Disease Control and Prevention DFID Department for International Development DHIS District Health Information System DHMT District Health Management Team DPPI Directorate of Policy, Planning, and Information DSDP Decentralized Service Delivery Program EA Environmental Assessment ESMP Environmental and Social Management Plan EVD Ebola Viral Disease FA Financing Agreement FHCI Free Health Care Initiative FIT Facility Improvement Team FM Financial Management GDP Gross Domestic Product GFATM The Global Fund to Fight AIDS, Tuberculosis and Malaria GoSL Government of Sierra Leone GRM Grievance Redress Mechanism HMN Health Metrics Network HNP Health, Nutrition, and Population HRIS Human Resource Information System HRITF Health Results Innovation Trust Fund IEC Information, Education, and Communication ICR Implementation, Completion, and Results Report IDA International Development Association IPAU Integrated Project Administration Unit IHPAU Integrated Health Project Administration Unit IP Implementation Progress IPFMRP Integrated Public Financial Management Reform Project ISR Implementation Status and Results Report JAS Joint Assistance Strategy LC Local Council LGFD Local Government Financing Department LLIN Long-Lasting Insecticide-Treated Mosquito Net M&E Monitoring and Evaluation MCH Maternal and Child Health MDG Millennium Development Goal MMEIG Maternal Mortality Estimation Inter-Agency Group MMR Maternal Mortality Ratio MoFED Ministry of Finance and Economic Development MoHS Ministry of Health and Sanitation MSF Médecins Sans Frontières MWM Medical Waste Management MWMP Medical Waste Management Plan NERC National Ebola Response Center NGO Non-Governmental Organization NHSSP National Health Sector Strategic Plan NHMIS National Health Management Information System ODI Overseas Development Institute OPM Oxford Policy Management PAD Project Appraisal Document PBF Performance-Based Financing PCMH Princess Christian Maternity Hospital PDO Project Development Objective PEHS Package of Essential Health Services PHU Peripheral Health Unit PRSP Poverty Reduction Strategy Papers RCHP Reproductive and Child Health Project RCHSP Reproductive and Child Health Strategic Plan RF Results Framework SDG Sustainable Development Goal SL Sierra Leone TTL Task Team Leader UN United Nations UNFPA United Nations Population Fund UNICEF United Nations Children Fund UNMEER UN Mission for Ebola Emergency Response WAHO West African Health Organization WB World Bank WHO World Health Organization TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 7 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ..................................................... 13 A. CONTEXT AT APPRAISAL ....................................................................................................... 13 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) ..................................... 18 II. OUTCOME .................................................................................................................... 22 A. RELEVANCE OF PDOs ............................................................................................................ 23 C. EFFICIENCY ........................................................................................................................... 26 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 28 E. OTHER OUTCOMES AND IMPACTS ......................................................................................... 29 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 30 A. KEY FACTORS DURING PREPARATION ................................................................................ 30 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 31 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 33 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 33 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 36 C. BANK PERFORMANCE ........................................................................................................... 37 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 39 LESSONS AND RECOMMENDATIONS .................................................................................... 40 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 43 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 55 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 57 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 58 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 61 ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 77 ANNEX 7. SIERRA LEONE COUNTRY MAP.............................................................................. 79 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name REPRODUCTIVE AND CHILD HEALTH PROJECT - PHASE 2 ( P110535 P110535 ) Country Financing Instrument Sierra Leone Specific Investment Loan Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Related Projects Relationship Project Approval Product Line Additional Financing P127527-Sierra Leone - 20-Sep-2012 Recipient Executed Activities Reproductive and Child Health Phase 2 ACGF Supplement P132753-Sierra Leone - 30-Aug-2013 IBRD/IDA Reproductive and Child Health II Project - Second Additional Financing Organizations Borrower Implementing Agency Ministry of Finance and Economic Development Ministry of Health and Sanitation Project Development Objective (PDO) Original PDO The PDO is to increase utilization of a package of essential health services by pregnant and lactating women and children under the age of five. Page 7 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) Revised PDO To increase utilization of a package of essential health services by pregnant and lactating women and children under the age offive and support the emergency response needed to contain and control the Ebola crisis. FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing P110535 TF-96812 20,000,000 20,000,000 19,978,240 P110535 TF-12691 5,690,614 5,687,584 5,687,584 P110535 IDA-H8770 13,000,000 13,000,000 12,678,975 P110535 TF-15054 5,000,000 5,000,000 4,999,935 Total 43,690,614 43,687,584 43,344,734 Non-World Bank Financing Borrower 0 0 0 Total 0 0 0 Total Project Cost 43,690,614 43,687,584 43,344,735 KEY DATES Project Approval Effectiveness MTR Review Original Closing Actual Closing P110535 12-Aug-2010 01-Oct-2010 30-Aug-2013 31-Oct-2016 30-Jun-2017 P132753 30-Aug-2013 Page 8 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 28-Oct-2013 18.22 05-Aug-2014 22.52 Change in Project Development Objectives Change in Results Framework Change in Components and Cost Reallocation between Disbursement Categories Change in Disbursements Arrangements Change in Institutional Arrangements Change in Procurement 09-Mar-2015 24.11 Change in Results Framework Change in Loan Closing Date(s) 15-Mar-2016 28.56 Reallocation between Disbursement Categories 20-Oct-2016 30.32 Change in Loan Closing Date(s) Reallocation between Disbursement Categories KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Moderately Satisfactory Modest RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 24-Nov-2010 Satisfactory Satisfactory 0 02 17-Apr-2011 Satisfactory Satisfactory 7.86 03 09-Jul-2011 Highly Satisfactory Highly Satisfactory 8.14 04 13-Dec-2011 Highly Satisfactory Highly Satisfactory 11.08 05 23-Apr-2012 Highly Satisfactory Satisfactory 11.27 06 10-Dec-2012 Highly Satisfactory Satisfactory 13.20 07 18-Jun-2013 Satisfactory Moderately Satisfactory 15.25 08 30-Dec-2013 Satisfactory Moderately Satisfactory 18.22 09 14-Jun-2014 Satisfactory Moderately Satisfactory 21.87 Page 9 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) 10 22-Dec-2014 Moderately Satisfactory Moderately Satisfactory 23.64 11 15-Jun-2015 Moderately Satisfactory Moderately Satisfactory 26.32 12 15-Dec-2015 Moderately Satisfactory Moderately Satisfactory 26.79 13 08-Jun-2016 Moderately Satisfactory Moderately Satisfactory 29.03 14 16-Sep-2016 Moderately Satisfactory Moderately Satisfactory 30.32 15 24-Mar-2017 Moderately Satisfactory Moderately Satisfactory 30.67 Moderately 16 10-May-2017 Moderately Satisfactory 30.67 Unsatisfactory 17 27-Jun-2017 Moderately Satisfactory Moderately Satisfactory 30.67 SECTORS AND THEMES Sectors Major Sector/Sector (%) Health 100 Health 100 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Public Sector Management 0 Public Administration 20 Municipal Institution Building 20 Human Development and Gender 0 Health Systems and Policies 80 Health System Strengthening 20 Reproductive and Maternal Health 40 Child Health 20 ADM STAFF Role At Approval At ICR Regional Vice President: Obiageli Katryn Ezekwesili Makhtar Diop Page 10 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) Country Director: Ishac Diwan Henry G. R. Kerali Senior Global Practice Director: Eva Jarawan Timothy Grant Evans Practice Manager: Eva Jarawan Gaston Sorgho Task Team Leader(s): Laura L. Rose Irina Aleksandra Nikolic Adanna Deborah Ugochi ICR Contributing Author: Chukwuma Page 11 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) Page 12 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. Despite emerging from a decade-long civil war, since 2000 Sierra Leone has been on a path of reconciliation, reconstruction, and stabilization of its economy and governance. This recovery continued for 9 unbroken years such that real Gross Domestic Product (GDP) grew at 5.5 percent in 2008. Since 2002, the country has also held elections deemed to be free and fair. However, the global financial crisis of 2007-2008 reduced revenue flows through private capital, remittances, foreign aid, and commodity sales, constraining funding available to finance improvements in social sectors including health care. 2. At Project Appraisal, Sierra Leone had one of the highest maternal and under-five mortality rates in the world and a low coverage of essential health services. The Maternal Mortality Estimation Inter- Agency Group (MMEIG)1 estimated a maternal mortality ratio (MMR) in Sierra Leone of 1,630 per 100,000 live births. The under-five mortality rate was 160 per 1,000 live births per the United Nations (UN) Inter-agency Group for Child Mortality Estimation. 2 In nationally-representative surveys, the use of several essential maternal and child health interventions that prevent morbidity and mortality, remained low. For example, in 2008, the prevalence of skilled birth attendance among mothers and the use of long-lasting insecticide-treated mosquito nets (LLINs) among under-five children were 42 percent and 26 percent respectively. 3 3. The cost of care and harmful cultural practices were identified as demand-side barriers to the use of essential health care for women and children in Sierra Leone. There were cultural barriers to health service use among women, including harmful traditional birthing and breastfeeding practices. 4 A nationally-representative household survey in 2007 also found that among those who felt the need to consult a health provider, cost was considered the main barrier to care use. 5 While the 2010 National Free Health Care Initiative (FHCI) eliminated formal user charges for maternal and child health services in public facilities, a lack of flexible resources for effective delivery at frontline facilities led to the persistence of informal user charges. 1 World Health Organization (2015). Trends in Maternal Mortality: 1990 to 2015. Estimates by WHO, UNICEF, UNFPA, World Bank Group and the United Nations Population Division. 2 Estimates from “United Nations Children’s Fund (2015). Levels and Trends in Child Mortality: Report 2015 Estimates Developed by the UN Inter-Agency Group for Child Mortality Estimation.” Estimates cited in the Project Appraisal Document (PAD), put maternal and under-five mortality rates in 2008 at 857 per 100,000 live births and 140 per 1,000 live births respectively. 3 Project Appraisal Document, RCHP-2, Report No.: 42945-SL. 4 Project Appraisal Document, RCHP-2, Report No.: 42945-SL. 5 Statistics Sierra Leone (SSL). Core Welfare Indicators Questionnaire 2007. Page 13 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) 4. Health service delivery also faced challenges related to management, fragmentation of actors, and otherwise poor quality of care, that may have also discouraged health care use. An institutional assessment of the Ministry of Health and Sanitation (MoHS) in 2008 noted management gaps in finances, health information, procurement systems, and facilities. 6 A 2009 assessment of supply chains for essential commodities also documented the continued lack of capacity in the MoHS or District Health Management Teams (DHMTs) to forecast, procure, distribute or manage essential drugs. 7 Other supply-side barriers to service provision identified at appraisal include a shortage of skilled health staff, proliferation of unqualified health staff, constraints in management control of health staff appointment and distribution, and poor coordination between the government, development partners and Non-Governmental Organizations (NGOs) in the health sector. 5. In the National Health Sector Strategic Plan (NHSSP) 2010-2015 and the Second Poverty Reduction Strategy Paper (PRSP-II) 2008-2012, the Government of Sierra Leone prioritized interventions to reduce child and maternal mortality. These mortality reductions were to be achieved through improvements in the access to and use of reproductive and child health services. The Government intended to develop and enforce appropriate regulations, improve monitoring and evaluation, ensure facilities functioned efficiently, improve health system governance, and coordinate work across all levels of the health system. 8 The President of Sierra Leone also launched the Reproductive and Child Health Strategic Plan (RCHSP) in 2008 aiming to reduce maternal, under-five, and infant mortality rates by 30 percent of the 2005 values by 2010 through a focus on primary health care, strengthening peripheral health services, and decentralization of health service management to the district level. 9 A key element of the RCHSP was the introduction of a package of essential health services (PEHS)10, which would have the greatest impact on major health problems affecting pregnant women and children in a cost-effective manner. 6. This Implementation Completion and Results Report (ICR) assesses the second phase of the Reproductive and Child Health Project (RCHP-2). The objective of Reproductive and Child Health Project Phase 1 (RCHP Phase 1) was to help the government address immediate constraints to reducing longer-term maternal and under-five mortality and to strengthen readiness for Phase 2 implementation. Specifically, RCHP Phase 1, financed (US$ 7 million) by the Africa Catalytic Growth Fund Grant (ACGF), involved decentralized financing and management of integrated health care delivery in health facilities, complemented by outreach services and a network of community health workers. It was expected that at the end of Phase 1, at least ten of thirteen local councils (LCs) would achieve specific targets relating to antenatal care use, child vaccination, and Integrated Child Survival 6 Project Appraisal Document, RCHP-2, Report No.: 42945-SL. 7 Phyllis Benonia Ocran 2009. Essential Medicines and Consumables Quantification Exercise, Supply Chain Assessment, May 2009. Freetown, Sierra Leone: UNICEF/ MoHS. 8 National Health Sector Strategic Plan (NHSSP) 2010-2015 and Second Poverty Reduction Strategy (PRSP-II) 2008-2012 9 Project Appraisal Document, RCHP, Report No.: 40881-SL 10 A full description of the PEHS can be found in Annex 4, Project Appraisal Document, RCHP, Report No.: 40881-SL. Page 14 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) Forms. However, the RCHP Phase 1 ICR rated achievement of the PDO as moderately unsatisfactory, as targets on two of the three outcome indicators were met by less than ten LCs. However, Phase 1 enabled the identification of weaker performing LCs that were targeted to receive intense supervision in Phase 2. 7. Rationale for the RCHP. As noted in the Project Appraisal Document (PAD), the RCHP (Phase 1 and 2) were selected for support by the ACGF because of Sierra Leone’s qualification as a transformational country11 following emergence from civil war, the innovative elements in the Project design including the first nationwide rollout of performance-based financing in Africa, and the Project’s multisectoral approach through strengthening links between the health system and decentralized governance. 12 The Project also aligned with pillar one of the Joint Assistance Strategy (JAS), with the African Development Bank, which focused in part on decentralized delivery of reproductive and child health services, and with a key focus area of the World Bank’s 2005 Africa Action Plan, that is strengthening national health systems. 13 Project Development Objectives (PDOs) 8. In the Financing Agreement (FA) of the RCHP-2, the PDO was to increase the utilization of a package of essential health services by pregnant and lactating women and children under the age of five. Key Expected Outcomes and Outcome Indicators 9. The RCHP-2 aimed to contribute to the National Reproductive and Child Health Program objective of reducing maternal and under-five mortality in Sierra Leone. 10. The Project Appraisal Document (PAD) identified expected outcomes linked to the PDO, including a) Increase in the percentage of children under five who slept the previous night under a LLIN to 75 percent; b) Improvement in the percentage of children receiving Penta-3 vaccine before 12 months of age to 78 percent; c) Vaccination of at least 379,484 children under the project; d) Increase in the percentage of births delivered in facilities to 55 percent; e) Increase in the percentage of pregnant women attending at least 2 antenatal care (ANC) visits to 75 percent; f) Provision of ANC during a visit to providers for 225,907 pregnant women under the project; g) Project benefit to 1,433,143 people directly; and h) At least 78 percent of direct project beneficiaries to be female. 11 Sierra Leone was considered transformational because of macro policy conducive to sustained medium-term growth and credible reform strategies evidenced by improvements in Country Performance Institutional Assessment ratings. Project Appraisal Document, RCHP-1, Report No.: 40881-SL. 12 Project Appraisal Document, RCHP-2, Report No.: 42945-SL. 13 Joint Assistance Strategy (JAS) Report No. 52297-SL, March 4 2010 and “Strengthening the Development Partnership and Financing for Achieving the MDGs: An Africa Action Plan”, September 2005 Page 15 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) 11. Annex 3 of the PAD also specified intermediate outcome indicators of changes in infrastructure, supply of essential commodities and skilled health workers, accountability, information and behavior change programs, and management capacity. Components 12. The original components of the Project are summarized in Table 1 below: Table 1: Components of RCHP-2 Component Sub-Component Description Cost Component 1: Sub-Component 1.1: a) Quarterly grants to all 19 LCs in the country US$ 12 million Strengthening of Service Grants to Local Councils to finance facility inputs such as vehicles, Delivery equipment, drugs, facility rehabilitation, and health worker training. b) Performance-based financing (PBF) to all 19 LCs for facilities based on attainment of a set of maternal and child health service coverage outputs with respect to facility- based deliveries; ANC attendance by pregnant women; use of LLINs by children under five; penta-3 immunization coverage rates of infants under the age of one; consultations for children under five years of age. Sub-Component 1.2: a) Support through the Local Government US$ 1.5 million Local Council Grant Finance Department (LGFD) and Integrated Administration Project Administration Unit (IPAU) for LC grant management. b) Support for performance contracting between LCs and PHUs, external results verification, capacity building in PHUs for PBF. c) Capacity building of DHMTs and LCs for procurement management; of LC environmental committees in medical waste management (MWM); and community sensitization on MWM. Sub-Component 1.3: US$ 5.69 million Purchase and Distribution of Bed Nets. 14 Component 2: Capacity Sub-Component 2.1: a) Capacity building for monitoring and US$ 3.5 million Building Supervision, Monitoring, evaluation (M&E) in the health system; and Evaluation strengthen District Health Information 14 This sub-component was added to the Project on request by the Government of Sierra Leone (GoSL) after Project approval but before the FA was signed and is not recorded in the Operations Portal as a Restructuring (Aide-Memoire, March 3-15, 2013). Page 16 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) System (DHIS); and support introduction of a Hospital Information System. b) Finance independent data quality audits and verification of PBF; support population health surveys. Sub-Component 2.2: a) Build capacity in Njala University and US$ 3.0 million Medical Training University of Sierra Leone for pre-service and post-graduate medical training. b) Conduct needs assessment for upgrades of the physical infrastructure of the teaching facilities and dormitories for students. Total US$ 25.69 million Theory of Change (Results Chain) 13. An overview of the Project results chain, derived from Annex 3 of the PAD, is shown in Table 2 below: Table 2: Results Chain Activities Outputs Project Development Health Impact Objective Financing inputs – vehicles, a) Improved health facility Increased utilization of Reductions in preventable equipment, supplies infrastructure, supply of package of quality essential child and maternal mortality (including LLINs), training of drugs, and consumables. health services by pregnant health workers b) Improved supply of and lactating women and skilled health workers. children under five. Performance-based financing a) Improved accountability of service delivery outputs for facility and LC including facility-based performance. deliveries, antenatal care b) Improved motivation for attendance, immunization service delivery coverage, under-five improvements among consultations health workers. Management capacity a) Improved facility building including grant management capacity . management and MWM b) Improved MWM capacity. Support for health a) Improved capacity for information system managing routine health strengthening information systems. b) Improved capacity for verification of PBF. Page 17 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) 14. The assumptions behind the translation of project outputs to the PDO are that the Project addressed the important barriers to the use of maternal (pregnancy and lactating) and child health care in Sierra Leone and that the barriers to care use that were not addressed by the Project and other ongoing activities in the health sector (such as geographical location of health facilities) did not eliminate the positive effect of the Project. The assumption that the PDO would translate to the long-term health outcomes specified in the Results Framework (RF) is based on peer-reviewed research evidence that links increases in use of essential health services among pregnant or lactating mothers and children to improved health outcomes. 15 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) 15. The Project received two Additional Financings (AFs) and was restructured six times as described briefly below. a) Between Project Approval and the signing of the FA, the Government of Sierra Leone (GoSL) requested for re-allocation of funds (US $ 6.7 million) to the large-scale purchase and distribution of LLINs targeting 80 percent of the population and for the introduction of retroactive financing for funds used by the Borrower to procure these LLINs prior to Project Effectiveness. 16 The re-allocation and introduction of retroactive financing were not recorded in the Bank system, as these occurred prior to the signing of the FA. b) The first AF (US$ 5.69 million), financed by the ACGF, was approved in September 2012. This AF replaced the funds that had been reallocated from both Sub-Components of Component 2 to Sub-Component 1.3 (Purchase and Distribution of Bed Nets), on the request of the GoSL. c) A second AF was approved in August 2013, financed from the Health Results Innovation Trust Fund (HRITF) (US$ 5 million) and International Development Association Credit (IDA) (US$ 13 million). 17 The key additions and changes introduced include: i. Component 1 (AF of US$ 17.06 million): Input-based grant financing (IDA US$ 3.06 million); expanding PBF to private PHUs and from two to eight hospitals (IDA US$5.0 million and HRITF US$5.0 million); supporting the procurement and distribution of LLINs (IDA US$3.0 million); and local council grant administration (IDA US$ 1 million). ii. Component 2 (AF of US$0.94 million): A new Sub-Component 2.3: Technical Assistance, focused on establishing the Integrated Health Project Administration Unit (IHPAU) to 15 Kerber et al (2007). Continuum of care for maternal, newborn, and child health: From slogan to service delivery. Lancet. 370: 1358-69. 16 Aide-Memoire, Implementation Support Mission to the Reproductive and Child Health Project-Phase 2, March 3-15, 2013. 17 Project Paper, Report No. 78269-SL. Page 18 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) improve the capacity of the MoHS to carry out management and oversight of externally-financed projects and to provide technical assistance towards the introduction of a social health insurance scheme for Sierra Leone. iii. A complaint handling and grievance redress mechanism (GRM) was to be introduced to ensure that feedback from Project beneficiaries could be received and acted upon. iv. Modification of the Project RF: The Project revised the definition of one outcome indicator (“Percentage of pregnant women attending at least 2 ANC visits” to “Percentage of pregnant women attending at least 4 visits”); added two intermediate outcome indicators (“all health workers included in human resources information system” and “annual health sector report available”), dropped one indicator (“percentage of graduates having received a six-month training in pediatrics and obstetrics”), and revised target values to account for extended time and coverage (Annex 1-C of ICR). d) In October 2013, the Project was restructured to extend the Closing Date for the ACGF grants which were co-financing the Project by 18 months from October 31, 2013, to April 30, 2015. The extension was necessary to allow for the completion of facility rehabilitation (Component 1) and capacity building activities (Component 2) that had been delayed due to re-allocation of funds for LLIN purchases. 18 e) The Project was restructured in August 2014 to support the response to the Ebola Virus Disease (EVD) crisis. The key changes are summarized below. 19 i. Modification of PDO: The revised PDO of the RCHP-2 was to increase utilization of a package of essential health services by pregnant and lactating women and children under the age of five and to support the emergency response needed to contain and control the Ebola crisis. ii. Reallocation of funds: A total of US$ 6 million, from the IDA grant, was reallocated from Component 1 to Component 3 to support the response to the EVD crisis via NERC, largely by financing the hiring of international medical workers to support control efforts and training of local health personnel. iii. Change in the RF: With the modification of the PDO to support the response to the EVD crisis, one new PDO Indicator was added to the RF: “to contract 130 doctors and other medical workers to support the Ebola control efforts and/or provide back-up medical services”. In addition, another outcome indicator, the targeted number of direct project 18 Restructuring Paper, Report No.: RES12071 19 Restructuring Paper, Report No.: RES15875. Page 19 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) beneficiaries was increased to 2,000,000 (from 1,433,143) to account for service provision to individuals affected by the crisis. As per the Intermediate Results Indicators, the targeted number of health personnel receiving training under the project was increased to 3,770 (from 1,440) to account for increased service provision needs and associated health worker training for the Ebola response. iv. Change in Project Components: The introduction of Component 3, “Emergency Response to Ebola Crisis”, supported key activities in line with the emergency response plan of the GoSL through the National Ebola Response Center (NERC) which coordinated implementation of the response plan across multiple partners including the MoHS, World Health Organization (WHO), Centers for Disease Control and Prevention (CDC), Institute Pasteur, Médecins Sans Frontières (MSF), the West African Health Organization (WAHO), and other United Nations (UN) agencies. v. Modification of Institutional Arrangements: To improve the coordination of the Ebola crisis response, an inter-ministerial task force instituted by GoSL was made responsible for developing and implementing the Accelerated Ebola Virus Disease Outbreak Response Plan (including Component 3); and at the regional level, an Ebola response center in Conakry was responsible for coordination and resource use optimization across countries and partners. vi. Changes in Disbursement Arrangements: The Project introduced the possibility to use UN Advance procedures20 as the disbursement mechanism for Component 3 to facilitate activities in support of the response to the EVD crisis and enhance efficiency via complementarities between UN Agencies, other international organizations, and NGOs in providing emergency health services. vii. Change in Procurement: To facilitate efficiency in responding to the Ebola crisis, provision was made for UN agencies, other international organizations, and/or NGOs to be contracted on a single-source basis by the Borrower responding to emergency situations, or when these organizations were uniquely qualified to provide technical assistance and services in their area of expertise. f) The Project was restructured in March 2015. The changes are summarized below. 21 20 The specific wording in the Project Paper: “In the event of direct contracting with a UN agency/ international organization/ NGOs, the Government will sign a contract/ convention with the UN agency/ international organization/ NGOs. The Bank will create a UN Advance Account and advance the agency directly. Once the funds are utilized, the agency will be required to submit a utilization report to the bank, for documentation purposes.” 21 Restructuring Paper, Report No.: RES16953. Page 20 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) i. The RF was revised by dropping 2 PDO-level and 10 intermediate indicators considered duplicative by the team and by adjusting target values of indicators (Annex 1-C). ii. Following a request from the GoSL, a no-cost extension of the two ACGF grants (TF- 96812 and TF-12691) was approved to provide additional time for implementation of activities delayed during the emergency response to the EVD crisis. The Closing Dates were extended by 18 months from April 30, 2015 to October 31, 2016 to align with the IDA financing Closing Date. g) The Project was restructured in March 2016 to reallocate funds (US$ 3 million) from financing LLINs procurement (Sub-Component 1.3) to financing PBF (Sub-Component 1.1). 22 The gap in funding for LLINs was met by the Global Fund and the Department for International Development (DFID). h) The Project was restructured in October 2016 to extend the Project Closing Date and to reallocate IDA grant funds between disbursement categories. 23 A no-cost extension of the Project completion date by 8 months from October 31, 2016 to June 30, 2017, following a request from the GoSL, to allow for implementation of activities delayed by the EVD crisis, including verification and payment for PBF results and assessment of Project achievements and lessons for future operations. Implication of Changes on the Original Theory of Change 16. The original theory of change was modified because of the EVD crisis as shown below in bold italic font. In summary, the changes were as follows: Component 3 funded international health workers to shore up control efforts and the other capacity improvements that may have influenced support for the response to the EVD crisis; the PDO was expanded; and the anticipated health impacts extended beyond mothers and children. The assumptions remain as described in the original theory of change. Table 3: Revised Theory of Change Activities Outputs Project Development Health Impact Objective Financing inputs – vehicles, a) Improved health facility a) Increased use of a) Reductions in equipment, supplies, training of infrastructure, supply of drugs, package of quality preventable child health workers and consumables. essential health and maternal b) Improved supply of skilled services by pregnant mortality health workers and lactating women and children under 22 Restructuring Paper, Report No.: RES22755. 23 Restructuring paper, Report No.: RES25090. Page 21 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) Financing service delivery a) Improved accountability for five, including LLINs, outputs including facility-based facility and LC performance immunizations, facility deliveries, antenatal care b) Improved motivation for service deliveries, and attendance, immunization delivery improvements antenatal care. coverage, under-five consultations Management capacity building a) Improved facility management including grant management capacity and MWM b) Improved MWM capacity Support for health information a) Improved capacity for managing system strengthening routine health information systems b) Support for the b) Reductions in child b) Improved capacity for emergency response and adult verification of PBF needed to contain mortality due to Hiring of international medical a) Improved human resource for and control the Ebola Ebola crisis workers health supply given the Ebola crisis crisis II. OUTCOME 17. Because of the changes made to the Project, a split rating methodology has been applied to assess overall achievement of project outcomes. Although there were multiple changes made during implementation, for the purposes of a split rating analysis, 3 distinct phases are used, in line with the new ICR Guidelines (July 2017). Phase A is from Project Effectiveness to August 2014. At the end of Phase A, the PDO was changed, new PDO indicators were introduced, and the target values for two indicators were adjusted: direct project beneficiaries and number of health workers trained. Phase B covers the period of September 2014 to March 2015 when the target values for multiple indicators were reduced. Phase C covers the period of April 2015 to the Project Closing Date. The assessment of results is summarized below and a summary of output by component is detailed in Annex 1. To apply the split rating methodology, numerical outcome ratings in each Phase were weighted by the share of Project disbursements in that Phase. 24 The split rating in part mitigates against favorable project ratings in this case from using the lower final indicator targets across the Project life. 24 The actual disbursements in the ICR datasheet are system-generated and only include the Project Trust Funds because the Project was Recipient-Executed. For the purposes of split rating, however, we use total disbursements, which include Trust Funds and the IDA Grant. Page 22 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 18. Rating: High for all 3 Phases. 19. Justification: The PDO aligns with the current Joint Assistance Strategy (JAS) which aims to promote decentralized delivery in health and to provide support to reproductive and child health to address preventable mortality. 25 A key indicator of JAS success is increase in coverage of essential maternal and child health services, which is directly related to the PDO. Thus, there were no shortcomings in the relevance of the PDO to the current JAS. The PDO also aligns with the Sierra Leone Basic Package of Essential Health Services (2015-2020) and the Health Sector Recovery Plan (2015-2020), both of which prioritize essential services for maternal and child health. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 20. RCHP-2 was commenced as the GoSL implemented FHCI, eliminating cost barriers to maternal and child health care use. Both programs had a complementary26 and national scope, so that separating their distinct impacts on health care use is empirically difficult. As per timing, RCHP-2 commenced simultaneously across all 19 LCs in Sierra Leone. In the absence of LCs that can be used as comparison units to LCs that received RCHP-2, we assume that service coverage levels would have remained at baseline levels. The absence of DHIS data from prior pre-intervention years prevents the examination of pre-project trends. Where the target value of an indicator has been changed via restructuring, we used the updated target value to assess achievement of that outcome. 21. The achievement of each of the two PDOs in each Phase was assessed based on the proportion of outcome targets that were achieved. See Annex 1-C for RF with achievements against all target values. We assigned ratings and a score to each outcome indicator as follows: High (exceeded or fully achieved objectives or is likely to do so) if up to 100 percent achieved; Substantial (almost fully achieved objectives or is likely to do so) if 75-99 percent achieved; Modest (partly achieved or is expected to partly achieve objectives) if 50-74 percent achieved; and Negligible otherwise. To determine the overall rating for each Phase, we assigned scores to outcome indicator ratings and calculated the mean score. 25 Joint Assistance Strategy (JAS) Report No. 52297-SL, March 4 2010 and “Strengthening the Development Partnership and Financing for Achieving the MDGs: An Africa Action Plan”, September 2005 26 FHCI addressed demand-side barriers to care use while RCHP-2 addressed supply-side barriers. Page 23 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) Table 4: Scoring System for Efficacy Ratings 4-Point Scale Percent Achievement Score High ≥ 100 1 Substantial 75-99 0.75 Modest 50-74 0.5 Negligible <50 0.25 PDO 1: To increase utilization of a package of essential health services by pregnant and lactating women and children under the age of five. 22. Rating: Substantial in Phases A and C; Modest in Phase B. 23. Outcome Indicators in Phase A: In Phase A, the Project recorded significant increases in the use of some essential services in the target population (see Annex 1-C). The percent achievement of target values for children sleeping under LLINs, total number of children immunized, births delivered in health facilities, and pregnant women attending at least 2 antenatal care visits were 87 percent, 55.38 percent, 164.74 percent, and 139.71 percent respectively. The target number of direct project beneficiaries and the number of pregnant women receiving antenatal care during a visit to a health provider were 118.06 percent and 127.82 percent achieved respectively. The target percentage of female beneficiaries was also 100 percent achieved. However, the value for exclusive breastfeeding in the first 6 months was only 14.77 percent achieved at this point. 27 The mean score in Phase A was 6.5/8, which equals 0.8125. Thus, the ICR team rates the efficacy of PDO 1 in Phase A as Substantial. 24. Outcome Indicators in Phase B: In Phase B, following the August 2014 restructuring for the EVD crisis, the Project exceeded 100 percent achievement for coverage of facility births (112.49 percent), proportion of pregnant women attending at least 2 antenatal visits (111.18 percent), and direct project beneficiaries (111.75 percent). Coverage for other indicators of essential use were below 100 percent by a substantial margin: children sleeping under LLINs (46.08 percent), number of children immunized (56.77 percent), number of pregnant women receiving any antenatal care (37.70 percent), and exclusive breastfeeding in the first 6 months (77.88 percent). The target percentage of female beneficiaries was almost fully achieved (98.73 percent). The mean score in Phase B was 5.5/8, which equals 0.6875. Thus, the ICR team rates the efficacy of PDO 1 in Phase B as Modest. 25. Outcome Indicators in Phase C: In phase 3, following the March 2015 restructuring, the retained Project indicators all exceeded 100 percent achievement of their revised target values: number of children immunized (107.61 percent), births delivered in facilities (136.21 percent), proportion of pregnant women attending at least 2 antenatal visits (110.86 percent), number of pregnant women receiving antenatal care (116.13 percent), and number of direct project beneficiaries (149.90 percent). 27 Exclusive breastfeeding is more accurately classified as an indicator of use of essential child health services and is considered an outcome indicator in this ICR. Page 24 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) The target percentage of female beneficiaries was also 100 percent achieved. Among indicators of health care use among mothers and children that were dropped via restructuring, achievements were more modest: children sleeping under LLINs (35.06 percent) and exclusive breastfeeding in the first 6 months (11.50 percent). The mean score in Phase C was 6.5/8, which equals 0.8125. Thus, the ICR team rates the efficacy of PDO 1 in phase C as Substantial. 26. Other Indicators along Theory of Change Relevant to PDO 1: Intermediate results indicators suggest that improvements in service delivery capacity, supported by RCHP-2 Component 1 and 2, may have contributed to the observed increases in service use. At the end of the Project, 87 PHUs (from a baseline of zero) could provide basic emergency obstetric care (133.85 percent achieved but indicator was dropped) 28; 459 facilities were renovated, equipped and/or constructed (330.6 percent achieved); 80 percent of households possessed an LLIN (81.27 percent achieved); and 1140000 LLINs were purchased and/or distributed (71.25 percent achieved). The ICR team did not find available data related to health care use by lactating mothers, such as postpartum care. 27. Thus, RCHP-2 appears to have contributed to increases in use of essential health services among pregnant and lactating mothers and children under five. While there were reductions in utilization in Phase B, during the EVD crises, the Project appears to have significantly enabled improvements in service delivery capacity and health care use in the target population. PDO 2: To support the emergency response needed to contain and control the Ebola crisis. 28. Rating: Substantial in Phases B and C. 29. PDO 2 was introduced in Phase B, following the August 2014 restructuring. The National Ebola Response Center (NERC), constituted in October 2014 and chaired by President Koroma coordinated the response at the national level, while the District Ebola Response Centers served as command-and- control hubs in the districts. The key activities involved in the response were surveillance, contact tracing, managing Ebola treatment centers, running isolation centers, home decontamination, safe burials, and social mobilization. 30. The Project re-allocated funds towards the United Nations (UN) Mission for Ebola Emergency Response (UNMEER) which represented UN agencies in NERC. The Multi-Partner Trust Fund paid salaries of core staff in the NERC and supported Ebola response surges. Modifications in disbursement and procurement mechanisms allowed UNDP to redirect Project PBF of health worker stipends towards hazard pay at a time when non-payment via the MoHS had triggered strikes. In one district, 28 Integrated Project Management Unit. Final Report: Impact Evaluation of RCHP-2. Ministry of Finance and Economic Development. 2017. Page 25 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) there is peer-reviewed evidence suggesting that service use levels were affected less by Ebola transmission in part due to hazard pay. 29 31. Interviewees in a study commissioned by the Royal Institute of International Affairs reported that World Bank (through RCHP-2 and other projects) and the AFDB were particularly responsive to the crisis from December 2014, and facilitated unlocking of surge funding for NERC quickly. These funds were used to contract international medical workers to provide essential services to Ebola patients. The PDO indicator measuring support for the response via contracting international medical workers was 100 percent achieved. 32. Surge funding was diverted to hazard pay and the recruited international medical workers considered to be essential support to control the EVD crisis. Thus, the ICR team rates efficacy related to PDO 2 in Phases B and C as Substantial. Justification of Overall Efficacy Rating 33. Rating: The ICR rates efficacy as Substantial in Phase A (single PDO), Modest in Phase B (because only one of the two PDOs was rated Substantial) and Substantial in Phase C (for both PDOs). 34. Justification: The Project recorded significant increases in the use of essential health services among pregnant and lactating mothers and under-five children. C. EFFICIENCY 35. Cost-effectiveness: The Project focused on scaling up a set of interventions that have been proven to be cost-effective across multiple contexts. In addition, an ex-post cost-effectiveness analysis was conducted for this ICR, accounting for the overlap between FHCI and Component 1 of RCHP-2 (which equals 69.5 percent of actual Project expenditure). The estimated total incremental expenditure of both interventions was US$ 266 million and estimated life years saved were 1,624,748. This amounts to $164 per life year saved (US$ 266 million/1,624,748 life years saved) which is considered highly cost-effective given commonly-used thresholds. Description of the analysis can be found in Annex 4. Thus, in terms of the PDO of increased utilization of health care among mothers and children, Component 1 appears to be cost- effective. 36. Implementation efficiency: As RCHP-2 relied on existing institutions for Project implementation (MoHS, IPAU, LGFD, LCs, and DMHTs), the Project avoided the costs of creating new units initially. However, there were other causes of implementation inefficiency. The EVD Crisis led to 29 Caulker et al. Life goes on: The resilience of maternal primary care during the Ebola outbreak in rural Sierra Leone. Public Health Action. 2017; 7 (Suppl. 1). Page 26 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) a reduction in human resources due to disease and death. It was imperative that available health workers were redeployed from some project functions to focus on surveillance and management as part of the Crisis Response. The short-term economic impact of Ebola in 2014 (in terms of spending on illness, lower productivity among the infected and their caregivers, and aversion behavior due to fear of contracting the disease) has been estimated at $163 million. 30 This provides insight into the scale of inefficiency introduced into the Sierra Leonean health system and wider economy. The frequent turnover of fiduciary staff in the MoHS and procurement delays within both the MoHS and IPAU also increased the time cost of the Project – the procurement of an ambulance boat took several years for example. There were also recurrent Project delays in the processing of performance-based financial incentives due to the difficulty in developing and endorsing subsidiary agreements between MoFED and LCs; local banking system transaction costs; national and local elections in 2012; and non-functionality of the health management information system (HMIS) that was to provide information for determining facility payments until the 3rd quarter of 2013. RCHP-2 also recorded a cost overrun of input-based grants to LCs of 12.9 percent (US$ 769,248.16)31. The Project Closing Date was extended to accommodate delays and enable completion of Project activities. Assessment of Efficiency and Rating 37. Rating: Modest for all 3 Phases. 38. Justification: The Project scaled cost-effective interventions in component one and estimates comparing marginal benefits and costs indicate that grant support and LLINs were a cost-effective complement to FHCI in improving essential service use and averting deaths. Notwithstanding, the Project experienced implementation delays, particularly due to the Ebola crises, and incurred cost overruns in one sub-component. 30 Edit V. Velenyi (2016) Health Care Spending and Economic Growth. World Scientific Handbook of Global Health Economics and Public Policy (pp. 1-154). 31 Integrated Project Management Unit. Final Report: Impact Evaluation of RCHP-2. Ministry of Finance and Economic Development. 2017. Page 27 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) D. JUSTIFICATION OF OVERALL OUTCOME RATING 39. Rating: Moderately Satisfactory 40. Justification: The objective of the RCHP-2 remains highly relevant to the GoSL and to the JAS of the World Bank and African Development Bank. 32 The Project appears to have contributed to increases in the use of essential maternal and child health services: by Phase C, most (7 out of 9) outcome indicators were at least 100 percent achieved; In Phase A, 5 out of 8 outcome indicators were 100 percent achieved; in Phase B, 5 out of 9 outcome indicators were about 100 percent achieved. RCHP-2 focused on scaling cost-effective interventions and Component 1 increased health service use in a cost-effective manner. However, there were implementation delays overall and cost overruns associated with Component 1. The disbursements in the ICR datasheet are system- generated and only include the Project Recipient-Executed Trust Funds. For the purposes of split rating, however, we use total disbursements, which include Trust Funds and the IDA Grant. 33 Table 5: Split Analysis Table. # Phase A Phase B Phase C Baseline – August, 2014 September, 2014 – March, 2015 April, 2015 – Endline 1. Relevance of objective High 2. Efficacy Substantial Modest Substantial a. PDO 1: Increase use of essential MCH services Substantial Modest Substantial b. PDO 2: Support to contain EVD crisis - Substantial Substantial 3. Efficiency Modest 4. Outcome ratings Moderately Satisfactory Moderately Unsatisfactory Moderately Satisfactory 5. Numerical outcome ratings34 4 3 4 6. Share of disbursement 0.71 0.08 0.21 Weighted value of outcome ratings: numerical outcome ratings * share of 7. disbursement 2.84 0.24 0.84 32 Joint Assistance Strategy (JAS) Report No. 52297-SL, March 4 2010 and “Strengthening the Development Partnership and Financing for Achieving the MDGs: An Africa Action Plan”, September 2005. 33 Total disbursements are as follows: Phase 1 (US$ 29,935,090.60), Phase 2 (US$ 3,484,473.95), and Phase 3 (US$ 8,939,705.92). As the IDA Grant was denominated in XDR and converted, these figures are approximations. 34 Highly Unsatisfactory (1); Unsatisfactory (2); Moderately Unsatisfactory (3); Moderately Satisfactory (4); Satisfactory (5); Highly Satisfactory (6) Page 28 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) Numerical ratings: sum of weighted value 8. of outcome ratings =round (2.84+0.24+0.84) = round (3.92) = 4 9. Overall outcome rating Moderately Satisfactory E. OTHER OUTCOMES AND IMPACTS Gender 41. RCHP-2 targeted pregnant and lactating mothers, explicitly incorporating a gender focus in the PDO, Project Components, and RF. The Project assessed the female proportion of Project beneficiaries as a PDO indicator and the target of 78 percent was recorded as 100 percent achieved in all 3 Phases of the Project. Institutional Strengthening 42. RCHP-2 invested in improving capacity for LC grant administration (Component 2); strengthening routine health information systems including the District Health Information System and national health surveys; and improving supervision of PHUs. Specific achievements related to institutional strengthening include the development of the first annual health sector report in 2015 and a subsequent report in 2016; financing for the Demographic and Health Survey in 2013; training of health personnel and LC staff in PBF and M&E (the target for health personnel training was 231.16 percent achieved35); training of undergraduate and postgraduate students in pharmacology and therapeutics with subsequent recruitment as teaching staff in local institutions (Annex 5); and the setup of IHPAU in the MoHS. The Project also contributed to building technical capacity for the introduction of a social health insurance scheme, through funding the supply of expertise and contributing to the review of policy documents. A national technical working group was convened to specify the roadmap adopting and implementing social health insurance and in 2017, a scheme was established under the National Social Security and Insurance Trust (NASSIT), with monthly contributions of Le15 per month to cover 35 diseases, and exempting specific groups from contributions (children below 12 years, adults above 65 years, inmates in correctional centers, and the extreme poor). 35 However, unique attendees were not counted, such that several individuals may have benefitted from multiple training experiences. Page 29 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 43. Background Analysis. The Project design benefitted from analysis of the health systems context in Sierra Leone, through nationally-representative reproductive and child health surveys, health financing and supply chain studies by Oxford Policy Management (OPM) and United Nations Children’s Fund (UNICEF), institutional assessments by the MoHS, and a nationwide consultation for the Poverty Reduction Strategy. Project preparation also involved consultations with Government counterparts and development partners in multiple pre-appraisal missions. RCHP-2 also incorporated lessons from RCHP Phase 1, the Bank’s experience in Sierra Leone and in health sector-specific projects in similar contexts, including specifying the procurement and programmatic responsibilities of the MoHS and MoFED in the Operations Manual; provision for training on procurement to LCs to reduce delays; financing technical assistance for Project implementation in the MoHS and MoFED; and providing block grants (rather than earmarked funds) to LCs so that flexible funding is available for changing local health needs. The Project appraisal analyzed Sierra Leonean experience in implementing decentralization of primary health care services since 2004, supported by the World Bank Group. 44. Adequacy of Participatory Processes. There was high-level Government commitment to the Project at the time of preparation, as the President had prioritized reducing maternal and infant mortality in Sierra Leone, and welcomed technical and financial support from the World Bank. Project Preparation involved consultations with a broad range of stakeholders including MoHS, MoFED, UNICEF, DHMTs, PHUs, and LCs, to inform the design of the Project Components and Implementation Arrangements. At the local level (that is DHMTs, PHUs, and LCs), ownership of the Project, particularly the PBF sub-component, was strong. 45. Identification of Risks and Mitigation. The overall risk rating was substantial. However, four risks were not identified at Appraisal and affected Project implementation, including the frequent rotation of civil servants involved in Project oversight within the MoHS led to delays in implementation; the large number of facilities involved in PBF with accounts located in multiple banks overwhelmed local financial institutions involved in payment disbursements; DHMTs lacked the time and financial resources to undertake regular PBF results verifications in a large number of facilities; and there was unclear division of responsibility among the multiple actors involved in handling medical and other waste in Sierra Leone. In addition, while the Appraisal recognized that financial resources for PBF may be insufficient for the anticipated growth in demand during the Project, there was little discussion regarding the risk of financial sustainability beyond the Project. Page 30 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) B. KEY FACTORS DURING IMPLEMENTATION 46. RCHP-2 was approved on 12th August, 2010 and became effective two months after, on 1 st October, 2010. The factors that affected implementation over the seven years of Project life have been grouped into factors within the control of the government and/or implementing entity; factors subject to World Bank control; and factors outside the control of any of the above entities. The Project mid-term review, which culminated in modifications to the RF and expanding the scope of LC grants, was merged with discussions on Additional Financing in August 2013. The PDO was rated Highly Satisfactory, Satisfactory, or Moderately Satisfactory until May 2017 when the PDO was rated Moderately Unsatisfactory, primarily due to pending updates to the Project indicators from December 2014 levels, which prevented objective assessment of Project progress per the results framework. 36 47. Factors within the control of the government or implementing entity a) Project coordination: The implementation arrangements used existing structures in the Ministry of Health and Sanitation and the Ministry of Finance, with clear roles and responsibilities with respect to the Project Components. By drawing on the relative expertise of each Ministry, delays in commencing Project activities and the cost of building initial implementation capacity was avoided. However, there were weaknesses in overall Project oversight due to coordination challenges between the MoHS and MoFED, particularly in monitoring grant administration at the LC level. b) Human resources and organizational capacity: IPAU and LGFD had the key skilled personnel and organizational structure to facilitate the nationwide implementation of Component 1, involving disbursement and monitoring of complex input and output-based grants to 19 LCs. However, the MoHS experienced significant capacity gaps that negatively influenced Component 2 of the Project including a delay in appointing focal points for health education and medical waste management; gaps in coordination of medical training in pediatrics and obstetrics; initial bottlenecks in the implementation of M&E plans for the Project including updating of DHIS-2 and HMIS; insufficient coordination with other parties involved in MWM (Environmental Health Officers and facility staff); and frequent transfer of civil servants with poor transition plans for new staff. The Project, through a restructuring, contributed to the establishment IHPAU to strengthen technical capacity in the MoHS. On the advice of the World Bank team, the IHPAU has recently been re-established and staffed through an open recruitment process. c) Fiduciary management: The IPAU and MoHS teams were largely able to maintain adequate procurement and financial management mechanisms. While the IPAU MoFED team 36 Implementation Status & Results Report, Report No.: ISR27901. Page 31 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) experienced delays of several years with the procurement of ambulance speedboats for Component 1, the team was otherwise able to appropriately support fiduciary management of Project activities. Due to staff turnover in the MoHS, there were delays in procurement and financial management compliance in the MoHS that required support from the IPAU MoFED team and the World Bank staff. d) PBF Implementation: The Project experienced challenges in implementing PBF nationally. Delays in the transfer of funds to LCs and facilities occurred due to difficulty in developing and endorsing subsidiary agreements between MoFED and LCs; local banking system transaction costs; national and local elections in 2012; and non-functionality of the health management information system (HMIS) that was to provide information for determining facility payments until the 3rd quarter of 2013. There was also difficulty in understanding of PBF verification roles among frontline service providers, gaps in data for verification37; and delays in the submission of verified data from DHMTs to the MoHS due to insufficient funding to support verification processes. To address these issues, the IPAU, MoHS, and World Bank team instituted coaching of facility, DHMT, and local council staff on PBF implementation; included mobile messaging to keep LCs updated on the status of funds transfers; and adjusted the PBF operational manual to account for the additional DHMT costs of verification in hard-to-reach areas. The Project also hired technical staff, with skills in health financing, monitoring, and evaluation, to build capacity for managing PBF verification in the MoHS. 48. Factors subject to World Bank control a) Turnover of and coordination of handover between Bank TTLs: As discussed in detail in the section on “Quality of Supervision”, the Project TTLs identified key implementation issues in internal reporting and specified agreed actions on these issues, including restructuring where necessary. There was a relatively high turnover of TTLs (5 TTLs over the 7-year period), and unclear procedures for handover, which contributed to discontinuities in supervision outside a joint handover mission. Delays in updates to Project indicators and inconsistencies in Project indicator definitions were addressed via a review of the Results Framework in close collaboration with the client. 49. Factors outside the control of government, implementing entity, and World Bank a) The EVD crisis: The Ebola crisis revealed health system deficiencies in Sierra Leone that in turn affected behavioral responses in the target population and reduced health care seeking. 38 These health system deficiencies were the low density of human resource for 37 External verification of PBF implementation by Cordaid found discrepancies between HMIS data and facility registers, missing facility registers and PBF data forms, and poor budgeting for verification procedures. 38 Edit V. Velenyi (2016) Health Care Spending and Economic Growth. World Scientific Handbook of Global Health Economics and Public Policy: pp. 1-154. Page 32 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) health (particularly culturally-sensitive community-based health provision), low core public health capacity for surveillance in the community, continued infrastructural challenges in health facilities to service provision (including lack of constant electricity and water supply), and weak supply chains for essential medicines and equipment. An assessment by WHO revealed that Sierra Leone had not attained the minimum 2005 International Health Regulations core capacity requirements by 2012. The Integrated Disease Surveillance and Response Guidelines had also not been implemented in Sierra Leone. With the onset of the crisis, the existing and sparse resources in public health system were diverted to the management of the Ebola Epidemic, such that there were barriers to access of other essential care. In Sierra Leone, 4 percent of the facilities closed, and malaria treatment for children and immunization coverage dropped by 39 percent and 21 percent respectively. The EVD crisis worsened existing shortages of skilled health providers as there were 864 reported health worker deaths. 39 The loss of human resources affected implementation and monitoring of RCHP-2. Fear of contracting the disease and misinformation also contributed to a decline of health service use, exacerbated by the absence of strong community-based health education and service provision. Following project restructuring in August 2014, the RCHP-2 re-allocated funds from Component 2 to support the overall World Bank response to the EVD crisis in partnership with key stakeholders under NERC. 40 When the EVD crisis resolved, training programs had to be conducted for newly recruited staff, introducing further delay in increasing health service coverage. On request from the GoSL, the project was again restructured by the World Bank team, extending the Closing Date to allow for implementation of activities delayed by the EVD crisis. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 50. The PDO of RCHP-2 identified the Project direct beneficiaries as pregnant and lactating women and children under five. The Project RF at design had outcome indicators that captured the use of health services among pregnant women and children. However, there were no clear indicators of utilization among lactating women such as postnatal care attendance and postpartum family planning services. Key indicators of intermediate outcomes related to PDO 1 were however identified at design, including: health facility rehabilitation, provision of basic 39 Edit V. Velenyi (2016) Health Care Spending and Economic Growth. World Scientific Handbook of Global Health Economics and Public Policy: pp. 1-154. 40 Component 3 involved pooling funds with other development partners to implement a comprehensive strategy managed by NERC, outside the Project. World Bank funds were also pooled under a new Project, P152359: Ebola Emergency Response Project. Page 33 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) emergency obstetric care in PHUs, ownership of LLINs by households, and training of health personnel among others. There were no intermediate outcomes in the RF linked to PDO 2 and one intermediate outcome linked to PDO 1 (infants exclusively breastfed for 6 months) is more appropriately classified as an outcome indicator. 41 51. The Project did not clearly specify how indicators would be measured, predisposing to ambiguity and inconsistent measurement. For example, the specific indicators included in direct project beneficiaries varied over the Project life. 42 Also, the number of children immunized and number of health workers trained were counted either annually or cumulatively at different times during the Project due to lack of clarity on whether the indicator was to be measured cumulatively. Target values for indicators in the RF were not specified in addition to Year 4 values in the PAD. There were also duplications in indicators. For example, there were two indicators of antenatal care attendance. While the RF stated sources of, frequency, and responsibility for data collection, the Framework assumed availability of data via routine monitoring systems (DHIS-2) every month. However, HMIS data in the DHIS-2 was not available until half-way through the Project in 2013 and omits all data before April 2011. Thus, the monitoring plan could have been better tailored to the context. The Bank team worked with the MoHS M&E team and Directors in iteratively revising the entire RF, to define each indicator consistently and determine achievements of target values across the Project life. M&E Implementation 52. RCHP-2 supported the conduct of national health surveys including the Demographic and Health Survey and the updating of the national health management information system (NHMIS) by the MoHS. Despite a sub-component dedicated to M&E, Project indicators were inconsistently monitored across the life of the Project 43 For example, an updated indicator asserted that 9 quarterly meetings had been held between LGFD and MoHS at a time when the Project life spanned only 2 quarters. 44 Also, the indicator for Direct Project Beneficiaries, which is cumulative and should increase or remain static, fell from 4,276,533 in December 2014 to 2,997,900 in June 2017 reflecting inconsistent definition and measurement as discussed above. 45 41 Lassi, Zohra S et al. “Essential Interventions for Child Health.” Reproductive Health 11. Suppl. 1 (2014): S4. PMC. Web. 31 Oct. 2017. 42 For example, the target value had been adjusted to accommodate service recipients during the Ebola crisis, some of which were not women or children. However, the definition did not always account for this, including in the final ISR. (Restructuring Paper, Report No.: RES15875). 43 Implementation Status & Results Report, Report No.: ISR27901. 44 Implementation Status & Results Report, Report No.: ISR3863. 45 Specifically, this indicator may have erroneously counted recipients of LLINs as distinct from users of other maternal and child health services. In addition, the specific maternal and child health services included varied over the Project life. Page 34 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) 53. In 2017, some Project indicators had not been updated since December 2014, in part due to the EVD crisis. However, initial delays in updating the RF were due to malfunctioning of the national server for the District Health Information System (DHIS) for which there was low capability within Sierra Leone to fix. During the Project, 2 PDO-level and 10 intermediate indicators were dropped to reduce duplication. However, some of these changes made it more difficult to track the influence of the Project on the PDO along the results chain. For example, the indicators measuring the percentage of health workers captured in health information systems, capacity to provide emergency obstetric care, and PHU reporting of MCH indicators were not duplicative but were dropped. 54. Despite the gaps in the RF, the M&E was largely rated Moderately Satisfactory prior to 2017. However, corrective actions on M&E implementation were undertaken. The Project facilitated trainings for MoFED and MoHS staff in M&E. The Project also hired consultants to strengthen M&E systems in the MoHS through coaching and to ensure regular updates of the DHIS. A series of virtual and in-person meetings with the MoHS, IPAU, and LGFD, were held to harmonize definitions and sources of data for Project indicators. 46 While there are still gaps in the RF, such as the lack of accounting for beneficiaries of Ebola services outside mothers and children in the direct project beneficiaries’ indicator and the absence of a measure of coverage changes among lactating mothers, the updated figures are consistent in their definition and measurement. Given the prior inconsistencies in the Project M&E, the harmonized updated values used in the ICR deviate from those reported in the Project Implementation Status & Results Reports (ISRs) and Aide-Memoires. M&E Utilization 55. The PBF component was intended to facilitate a direct link between data on service provision to improvements. During interviews, the IPAU and MoHS team noted that the facility management teams did not initially use the information from internal verification of PBF data to improve their facilities and service delivery. However, the introduction of the innovative Facility Improvement Teams (FIT) framework by the Director of Policy, Planning, and Information (MoHS) to guide facility use of grant funds established a checklist for improvement of services for obstetric care and strengthened the links between facility data and local innovation. Justification of Overall Rating of Quality of M&E 56. Overall Rating: Modest 46 A document with the updated data, Methodology for RCHP-2 M&E, June 28, 2017, can be found in the Operations Portal. Page 35 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) 57. Justification: The RF enabled measurement of PDO achievement using the outcome indicators, but should have included an indicator of health care utilization among lactating mothers since this was an explicit part of the PDO. The RF also included key indicators of intermediate outcomes along the results chain at appraisal, relating to PDO 1. The Project supported the conduct of an important national health survey and improvements in the NHMIS. The key design challenges were lack of clear indicator definitions and specified target values. During implementation, important measures of intermediate outcomes were dropped, while two duplicative indicators of antenatal care were retained in the RF. The delays in updating the RF were in part due to the Ebola crisis, but there was also gaps in complying with monitoring requirements at other times, creating difficulty in assessing progress in achieving the PDO consistently. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 58. Environmental Compliance: The Project was supposed to meet the requirements of an updated Environmental and Social Management Plan (ESMP) and Medical Waste Management Plan (MWMP). These requirements included development of a national policy on MWM, training district medical officers and environmental health officers (EHOs), training of MWM technicians and other health care staff, and distribution of guidelines on MWM nationally. An environmental safeguards focal person was not designated by the MoH until 2016 leading to delays in adopting the requirements. The focal person conducted trainings for DHMTs in 16 districts but was unable to cover other districts due to financial and time constraints. Per the Safeguards mission, the Project did not address other pertinent issues in the waste management plan including building the capacity of key personnel (e.g. waste handlers and transporters) and the provision and supervision of the use of personal protective equipment and receptacles for waste handling, storage, transportation and disposal. 59. Social Compliance. A GRM was established in 2013 and received complaints from beneficiaries, primarily related to non-receipt of PBF payments by facility staff. IPAU reported that all concerns were appropriately addressed. 47 60. Financial Management (FM) Compliance. No significant issue was noted by the ICR team. Supervision indicated that the IPAU and MoHS teams maintained adequate FM arrangements necessary to ensure that Bank Credit and Grant proceeds were used for intended purposes in an economical and efficient manner. 61. Procurement Compliance. With substantial support from the World Bank procurement teams, the MoFED/IPAU and MoHS procurement units largely followed the World Bank’s procurement procedures. Weak procurement units’ capacity and implementation arrangements necessitating coordination between MoHS and MoFED fiduciary management and technical 47 Noted in correspondence with the IPAU and LGFD team on the Grievance Redress Mechanism within the Operations Portal. Page 36 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) teams complicated procurement planning, tendering, and evaluation processes and contract management which caused delays in the implementation of the project. For example, the procurement of an ambulance boat for Bonthe District took several years to complete due to difficulty in ascertaining the appropriate technical specifications as well as training on its operation and maintenance. 48 C. BANK PERFORMANCE Quality at Entry 62. Rating: Moderately Satisfactory 63. The PDO was aligned with development priorities articulated by the GoSL in its 2008-2012 Poverty Reduction Strategy Paper, which highlighted the need to scale up a minimum package of essential maternal and child health services. The Project contributed to the Human Development pillar of the current JAS that focuses in part on investments to support decentralized delivery in health, as well as dedicated support to reproductive and child health, to address child and maternal mortality. The Components of the Project were grounded in an analysis of demand and supply-side factors affecting the use of maternal and child health care in Sierra Leone, including shortage of skilled staff, the cost of care, and low management capacity at all levels of the health system. The Project design also reflected on lessons learned from past projects including RCHP Phase 1 through consultations with stakeholders in the health sector. However, it can be argued that given the complexity of the design, and the indication that some LCs had low implementation capacity in Phase 1, this Project should have been piloted, iteratively designed, and scaled later, rather than implementing nationally. 64. Implementation arrangements for the Project components were clearly stipulated in the PAD which also identified capacity gaps in key agencies involved in implementation and factored these into the Project design. For example, UNICEF was contracted to procure medical supplies given the weak procurement capacity previously documented in the MoHS. An updated Environmental and Social Management Plan identified the key safeguard issue to be MWM. Thus, MWM was incorporated as an activity in Component 2 of the Project to encourage compliance. Similarly, there were assessments of capacity for financial management and procurement, identification of attendant risks, and stipulation of corrective actions to ensure fiduciary compliance. A key group that was not as thoroughly examined were the DHMT teams that proved to be central to successful implementation of PBF and for which adjustments in the operational manual had to be made later to facilitate their engagement. Also, constraints in the ability of the MoHS to provide adequate oversight to the Project were not 48 Aide-Memoire, July 11-20, 2016. Page 37 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) addressed adequately at design, leading to procurement delays and the later setup of IHPAU. Finally, the financial sustainability of the PBF beyond the project life was not sufficiently addressed. 65. The assessment of expected stream of benefits and costs was based on a Marginal Budgeting for Bottlenecks (MBB) exercise undertaken in 2006. During restructuring, there was a qualitative analysis of potential sources of inefficiency due to the EVD crisis, including behavioral aversion to care use, delays in transportation of essential supplies, and loss of human resource. 49 The initial analysis omitted four risks that subsequently affected the Project and are discussed under the section on “Key Factors During Preparation” above. The PDO identified the specific target beneficiaries as pregnant and lactating women and under-five children. The RF at appraisal clarified indicator baselines, responsibilities for data collection, and captured service delivery aspects relevant to the PDO, although there was no outcome indicator for essential service use among lactating mothers. There was also a Project sub-component which was dedicated to M&E. However, some indicators were duplicative and others non-specific. The definitions of some indicators were unclear. For example, the “Direct Project Beneficiaries” indicator was measured differently across the life of the Project. While other aspects of Project Quality at Entry were satisfactory, due to the limitations in risk assessment, the RF, and institutional assessments, the ICR team rates Quality at Entry as Moderately Satisfactory on balance. Quality of Supervision 66. Rating: Moderately Satisfactory 67. Supervision of the RCHP-2 was led by five TTLs over the Project length. In interviews with the country counterparts, it was noted that Project continuity was affected negatively during transitions. Aide memoires and internal reporting identified other key implementation issues that posed a risk to the PDO, such as the human resource constraints in the MoHS and delays in procurement. Aide memoires also included agreed actions or next steps on each implementation issues, the party responsible for the action, and due dates. Progress made on attaining indicator targets was used to assess implementation of the Project components and determine the intensity of implementation support for different Project activities. Solutions to the identified issues were identified, including Project restructuring where necessary, the setup of IHPAU, and provision of just-in-time support for fiduciary management by Bank staff to address capacity gaps in the MoHS. Much effort was placed on restructuring the Project to respond to the Ebola crisis, capacity constraints in the implementing agencies, and to extend the Project life, to achieve the PDO. The proactive restructuring to respond to the Ebola crisis enabled funds to be quickly re-allocated to the pooled funds managed by NERC in August 2014, acknowledged by local stakeholders to have enabled the initial response to the crisis. In most cases, the ISR ratings were consistent with Project performance, except for M&E which was rated 49 Restructuring Paper, Report No.: RES15875. Page 38 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) Moderately Satisfactory for most of the Project despite challenges in the design and updating of the Project indicators. 68. Supervision of fiduciary aspects of the Project was strong, including procurement and FM missions with accompanying reports. Reviews noted delays in procurement within IPAU and MoHS and suggested appropriate mitigation measures such as technical assistance for the specifications of the ambulance boat. Supervision of financial management focused on improving FM capacity and performance. Reports identified key implementation issues, next steps, and responsible parties for carrying out each action. There were gaps in addressing delays in compliance with environmental safeguards, including pertinent issues in the waste management plan. The Bank team noted that the MoHS team did not submit progress reports for the implementation of the safeguards instruments to the Bank safeguards team or of implementation of the Grievance Redress Mechanism, making it difficult to provide meaningful support in a timely manner. 69. Implementation support of RCHP-2 was hampered by delays and inconsistencies in updating the Project indicators. Thus, changes noted in indicator actual values might not have reflected progress or lack of progress in implementation of Project activities, constraining understanding of achievement of the PDO. These M&E gaps may have been addressed by the presence of an M&E specialist on the task team. While other aspects of Quality of Supervision were satisfactory, and commendable given the concurrent Ebola crisis, the delays in addressing the gaps in the RF hampered implementation support and the ICR team rates Quality of Supervision as Moderately Satisfactory. Justification of Overall Rating of Bank Performance 70. Overall Rating: Moderately Satisfactory 71. Justification: Overall, the Project was appropriately designed and implementation was actively supported. However, there were gaps in risk and institutional assessments at Appraisal. In addition, the limitations in the RF design affected implementation support and were not proactively addressed until the end of the Project. More support could have been provided to the MoHS to improve environmental safeguard compliance. D. RISK TO DEVELOPMENT OUTCOME 72. The main risks to sustaining and improving on Project gains are as follows: a) From a financial perspective, the ICR workshop highlighted the importance of World Bank financing of PBF for maintaining health worker stipends when there were delays in paying civil servants by the Government. With Project closure and discontinuation of PBF, it is unclear that these financial incentives can be maintained. As these incentives Page 39 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) were key to motivating health workers, it is also unclear that service delivery changes will be sustained. b) From an institutional perspective, national capacity has been built through the Project for disease surveillance and response via NERC, program management in MoHS and IPAU, M&E, PHU monitoring in DHMT and LCs, and teaching in schools of pharmacy. Investments are likely to be sustained since the trained staff have been retained in the health system. However, funding from the Project was used to support M&E and health financing consultants that may not be retained at Project closure. c) From a social perspective, there is strong support for the results of this Project among beneficiaries, DHMTs, PHUs, Hospitals and within Government, particularly with respect to PBF and the contributions towards the EVD crisis Response. d) Per environmental factors, compliance with safeguards requirements may be constrained by the lack of dedicated financing and a national MWM plan. The appointment of a focal person in the MoHS suggests that renewed attention may be given to this issue that needs to be matched with financing for implementation. LESSONS AND RECOMMENDATIONS 73. As a complement to the FHCI, the RCHP-2 appears to have contributed to increasing the use of essential maternal and child health services in Sierra Leone and to supporting efforts to control the EVD crisis. Below are recommendations that have emerged from examining factors that influenced preparation and implementation of this Project: a) Ensuring continuity in World Bank supervision of complex projects is key to implementation success. To this end, high turnover of TTLs and the lack of clear guidance and structure to ensure transition between one TTL and the next may have contributed to inefficiencies in project implementation. Thus, the subject of project handovers should be considered as part of the Health, Nutrition, and Population Global Practice’s Agile agenda. b) The effects of the EVD crisis on the Sierra Leonean health system illustrate the need for strong community-based systems for surveillance, health education, and service delivery. In this regard, it is key to improve system resilience to future crises by meeting recommendations of the 2005 WHO International Health Regulations, including further strengthening of core public health capacities for surveillance and response; strengthening the health work force, particularly community-based service delivery functions; and improving the coordination and reliability of supply chains for essential medicines and equipment. Page 40 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) c) Future operations should identify and involve key groups that are critical to implementation success during initial discussions on Project design. During the RCHP-2 ICR workshop, ownership of Project activities that involved DHMTs, PHUs, and facility staff from the design phase at inception was high. However, lack of their engagement in MWM, despite the importance of these stakeholders for implementation, may have influenced the low buy-in and adoption of these activities. d) Projects must ensure that implementation arrangements and project activities appropriately address the constraints introduced by capacity gaps and HR procedures within counterpart institutions. To prevent lags in implementation due to rotation of civil servants, future Projects must include procedures for transition planning and onboarding/coaching of new staff. Prior to taking on interventions with complex arrangements as with PBF, there is a need for coaching of all parties involved, to clarify roles and procedures, perhaps in the setting of a pilot, to avoid implementation delays. In addition, financial provision should be made for internal verification in PBF, which presents a significant cost to local agencies. e) M&E arrangements must include target values, clear definitions of indicators, and specific realistic sources of data for updates. There is a need to consider these and other factors during the design of the Project and to engage M&E specialists in the design of the RF. Linking the indicators to a clearly thought-out theory of change would have prevented arbitrary dropping of indicators that are necessary for tracking changes along the results chain, such as measures of essential drug supply for facilities in the RCHP-2 and institutional changes, or encouraged substitution of indicators with more appropriate ones. In the absence of routine health information systems that collect high quality data for M&E, Projects should specify contextually-relevant procedures for tracking Project progress across Components. f) While technical leadership of Project Components may involve multiple ministries and agencies in Sierra Leone, it is advisable to retain fiduciary management functions of future projects in a single agency with adequate capacity. The IPAU team in MoFED demonstrated clear understanding and efficient implementation of World Bank fiduciary management procedures and only experienced delays in procurement with a complicated ambulance boat acquisition. Frequent staff rotations in MoHS and reduced capacity otherwise led to several procurement delays that required coaching from MoFED and the World Bank team. Thus, it is recommended that future Projects retain fiduciary management functions within IPAU, include procedures for transfer of capacity to MoHS staff, and primarily focus the technical expertise of the MoHS and other relevant Government Ministries, Departments, and Agencies in implementation of Project activities. Page 41 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) g) The phased implementation of large-scale complex health interventions provides the opportunity to learn-by-doing and troubleshoot Project design problems during the pilot phase. Particularly, in the case of Sierra Leone, there appears to have been a mismatch between the complexity of the project design and the capacity needed to ensure efficient implementation. The ICR of RCHP-Phase 1 had indicated that there were implementation challenges in some districts, which could have informed the selection of pilot locations. Implementation of PBF required several stakeholders to work together seamlessly, with a clear demarcation of roles. Coordinating verification and payments placed a huge demand in terms of skill, coordination, time, and cost on these stakeholders, who could have built their capacity gradually over pilots with related operations research, iterative design and testing. Phased implementation of future complex health interventions provides the opportunity to clarify Project roles, identify and troubleshoot bottlenecks prior to scaling, and to account for all the costs of Project implementation. Financial sustainability studies projecting the costs of complex health interventions and alternative financing sources following project closure are also . necessary during project design. Page 42 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Objective 1: To increase utilization of a package of essential health services by pregnant and lactating women and children under the age of five Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Direct project beneficiaries Number 0.00 1357715.00 2000000.00 2997900.00 31-Dec-2009 31-Oct-2013 31-Oct-2016 30-Jun-2017 Female beneficiaries Percentage 0.00 78.00 78.00 79.40 31-Dec-2009 31-Oct-2013 31-Oct-2016 30-Jun-2017 Comments (achievements against targets): As split rating methodology is applied to assess achievement of project outcomes, see Annex 1.C for discussion of achievement against targets in the 3 Project Phases. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Pregnant women receiving Number 0.00 225907.00 246000.00 285680.00 antenatal care during a visit to a health provider (number) 31-Dec-2009 31-Oct-2013 31-Oct-2016 30-Dec-2016 Page 43 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) Comments (achievements against targets): As split rating methodology is applied to assess achievement of project outcomes, see Annex 1.C for discussion of achievement against targets in the 3 Project Phases. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Women who attend at least 2 Percentage 54.60 75.00 85.00 88.30 antenatal care (ANC) visits 31-Dec-2009 31-Oct-2013 31-Oct-2016 30-Jun-2017 Comments (achievements against targets): As split rating methodology is applied to assess achievement of project outcomes, see Annex 1.C for discussion of achievement against targets in the 3 Project Phases. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Births delivered in a health Percentage 30.30 55.00 63.00 74.80 facility 31-Dec-2009 31-Oct-2013 31-Oct-2016 30-Jun-2017 Comments (achievements against targets): As split rating methodology is applied to assess achievement of project outcomes, see Annex 1.C for discussion of achievement against targets in the 3 Project Phases. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Children immunized (number) Number 0.00 379844.00 216000.00 232437.00 31-Dec-2009 31-Oct-2013 31-Oct-2016 30-Jun-2017 Page 44 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) Comments (achievements against targets): As split rating methodology is applied to assess achievement of project outcomes, see Annex 1.C for discussion of achievement against targets in the 3 Project Phases. Objective/Outcome: Objective 2: To support the emergency response needed to contain and control the Ebola crisis Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Direct project beneficiaries Number 0.00 1357715.00 2000000.00 2997900.00 31-Dec-2009 31-Oct-2013 31-Oct-2016 30-Jun-2017 Female beneficiaries Percentage 0.00 78.00 78.00 79.40 31-Dec-2009 31-Oct-2013 31-Oct-2016 30-Jun-2017 Comments (achievements against targets): As split rating methodology is applied to assess achievement of project outcomes, see Annex 1.C for discussion of achievement against targets in the 3 Project Phases. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Doctors and other medical Number 0.00 130.00 130.00 130.00 workers who are contracted to (a) support the Ebola control 04-Aug-2014 31-Oct-2016 31-Oct-2016 30-Jun-2017 efforts; and/or (b) provide back-up medical services Comments (achievements against targets): As split rating methodology is applied to assess achievement of project outcomes, see Annex 1.C for discussion Page 45 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) of achievement against targets in the 3 Project Phases. A.2 Intermediate Results Indicators Component: Component 1: Strengthening Service Delivery Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Health facilities constructed, Number 0.00 150.00 150.00 459.00 renovated, and/or equipped (number) 31-Dec-2009 31-Oct-2013 31-Oct-2016 30-Jun-2017 Comments (achievements against targets): As split rating methodology is applied to assess achievement of project outcomes, see Annex 1.C for discussion of achievement against targets in the 3 Project Phases. Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Long-lasting insecticide-treated Number 0.00 1140000.00 1600000.00 1140000.00 malaria nets purchased and/or distributed (number) 31-Dec-2009 31-Oct-2013 31-Oct-2016 30-Jun-2017 Comments (achievements against targets): As split rating methodology is applied to assess achievement of project outcomes, see Annex 1.C for discussion of achievement against targets in the 3 Project Phases. Component: Component 2: Capacity Building Indicator Name Unit of Baseline Original Target Formally Revised Actual Achieved at Page 46 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) Measure Target Completion Health personnel receiving Number 0.00 1440.00 3770.00 5314.00 training (number) 31-Dec-2009 31-Oct-2013 31-Oct-2016 30-Jun-2017 Comments (achievements against targets): As split rating methodology is applied to assess achievement of project outcomes, see Annex 1.C for discussion of achievement against targets in the 3 Project Phases. Component: Component 3: Support for Emergency Ebola Response Page 47 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) B. KEY OUTPUTS BY COMPONENT This table only reports on indicators that were not dropped from the RF during the Project life. Objective 1: To increase utilization of a package of essential health services by pregnant and lactating women and children under the age of five 1. Number of children immunized 2. Percentage of births delivered in health facilities 3. Percentage of pregnant women attending at least 2 antenatal visits Outcome Indicators 4. Number of pregnant women receiving antenatal care during a visit to a health provider 5. Number of direct project beneficiaries 6. Percentage of female project beneficiaries 1. Number of health facilities, constructed, renovated, and/or equipped Intermediate Results 2. Number of long-lasting insecticide-treated malaria nets purchased and/or distributed Indicators 3. Number of health personnel receiving training 1. Component 1 (Strengthening Service Delivery): • Training of local and allied health personnel • Renovation of health facilities • Provision of basic equipment Key Outputs by • Financial and technical support for DHMT internal verification of service delivery outputs Component • Funding for operating costs of reproductive and child health care delivery (linked to the • Distribution of 3,807,000 LLINs achievement of the • Support for design and internal verification of PBF Objective/Outcome 1) 2. Component 2 (Capacity Building) • Recruitment of 5 external lecturers and 2 local lecturers to support training • 23 students trained at Master of Science level in para-clinical and pharmaceutical services • Training of lecturers who now work as graduate trainees and Lecturer-2 Page 48 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) • Refurbishing staff offices and classrooms, including supply of teaching aides and projectors • Generators provided to augment the National grid for College of Medicine and Allied Health Sciences and School of Community Health Sciences • ICT Centers set up at College of Medicine and Allied Health Sciences and School of Community Health Sciences • Libraries at College of Medicine and Allied Health Sciences and School of Community Health Sciences resourced with current editions of medical, pharmaceutical, and nursing textbooks • Commencement of National Social Health Insurance Scheme • Annual health sector reports produced in 2015 and 2016 Objective 2: To support the emergency response needed to contain and control the Ebola crisis 1. Number of doctors and other medical workers who are contracted to (a) support the Ebola control efforts; and/or Outcome Indicators b) provide back-up medical services Intermediate Results Indicators Key Outputs by 1. Component 3 (Support for Emergency Ebola Response): Financed recruitment of 130 medical workers to support Component Ebola Response (linked to the achievement of the Objective/Outcome 2) Page 49 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) C. ORGANIZATION OF THE ASSESSMENT OF THE PDO To clearly show the analysis behind the split ratings, a revised RF is also shown below. The indicators were not tracked monthly, thus the actual values in Phases A, B, and C correspond to annual values for 2014, 2015, and 2016/7 respectively. In each Phase, we use the target value at that phase to calculate achievement. # Indicator Baseline Phase A Phase B Phase C Comment (Baseline - August, 2014) (September, 2014 - March, 2015) (April, 2015 - Endline) Target value Actual value % Achieved Target value Actual value % Achieved Target value Actual value % Achieved (a) (b) (c) − (b) (c) − (b) (c) − =( ∗ =( ∗ =( ∗ − − − 100) 100) 100) PDO Indicators 1. Children 59.60 75.00 73.00 87.00 80.00 69.00 46.08 75.00 65.00 35.06 Indicator dropped under-five in March 2015. who slept the Gaps in DHIS. previous Source: Borrower’s night under impact evaluation. an Phase B value insecticide- obtained by treated interpolating 2014 mosquito net and 2016. (%) 2. Children 58.60 78.00 Not Not 90.00 Not available Not 90.00 Not Not Indicator dropped receiving available available available available available in March 2015. Penta-3 DHIS-2 does not before 12 indicate if months of vaccination age (%) received under 12 months of age. 3. Children 0.00 379,844.00 210,355.00 55.38 385,000.00 218,570.00 56.77 216,000.00 232,437.00 107.61 Indicator was not immunized tracked (number) cumulatively and target value did not appear to be cumulative. 4. Births 30.30 55.00 70.99 164.74 70.00 74.96 112.49 63.00 74.84 136.21 Page 50 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) delivered in health facilities (%) 5. Pregnant 54.60 75.00 83.10 139.71 85.00 88.40 111.18 85.00 88.30 110.86 Definition of women indicator revised attending at twice (4 versus 2 least 2 visits) during antenatal project. visits (%) 6. Pregnant 0.00 225,907.00 266,704.00 118.06 745,000.00 280,869.00 37.70 246,000.00 285,680.00 116.13 In the revised RF, women project tracked this receiving as an annual rather antenatal than cumulative care during a indicator. visit to a health provider (number) 7. Direct project 0.00 1,357,715.00 1,735,465.00 127.82 2,000,000.00 2,234,904.00 111.75 2,000,000.00 2,997,912.00 149.90 Indicator does not beneficiaries track unique (number) beneficiaries. Uses tracers for women and children. 8. Female 0.00 78.00 78.00 100.00 79.00 78.00 98.73 78.00 79.37 101.76 beneficiaries (%) 9. Doctors and 0.00 Not Not Not 130.00 130.00 100.00 130.00 130.00 100.00 other applicable applicable applicable medical workers who are contracted to (a) support the Ebola control efforts; and/or b) provide back- up medical Page 51 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) services (number) Intermediate Results Indicators 10. Health 16.00 150.00 266.00 186.57 150.00 287.00 202.24 150.00 459.00 330.60 This is a facilities, cumulative constructed, indicator. Unique renovated, facilities counted. and/or equipped (number) 11. PHUs 0.00 60.00 87.00 145.00 65.00 87.00 133.85 65.00 87.00 133.85 Indicator dropped providing in March, 2015. basic Source: Borrower’s emergency impact evaluation. obstetric care (number) 12. Households 36.60 80.00 87.00 116.13 90.00 83.50 87.83 90.00 80.00 81.27 Indicator dropped with a LLIN in March, 2015. (%) Phase 2 value obtained by interpolating 2014 and 2016. Source: Borrower’s impact evaluation. 13. Long-lasting 0.00 1,140,000.00 1,140,000.00 100.00 1,600,000.00 1,140,000.00 71.25 1,600,000.00 1,140,000.00 71.25 Source: Borrower’s insecticide- impact evaluation. treated malaria nets purchased and/or distributed (number) 14. Clinics having 32.00 60.00 Not Not 70.00 Not available Not 70.00 Not Not Indicator dropped all 10 available available available available available in March, 2015. essential Updated data not drugs available. available 15. Graduates 0.00 40.00 0.00 0.00 40.00 0.00 0.00 40.00 0.00 0.00 Indicator dropped having in August, 2013, Page 52 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) received a due to low six-month demand for training in activity. pediatrics and obstetrics (%) 16. Health 0.00 1,440.00 2,404.00 166.94 3,770.00 3,635.00 96.42 3,770.00 5,314.00 140.95 Source: Borrower’s personnel impact evaluation. receiving Does not count training unique attendees. (number) Unclear if 2011- 2014 is cumulative. 17. Infants 37.40 55.00 40.00 14.77 60.00 55.00 77.88 60.00 40.00 11.50 Indicator dropped exclusively in March, breastfed for 2015.Source: 6 months (%) Borrower’s impact evaluation. 18. Hospitals 0.00 18.00 Not Not 18.00 Not available Not 18.00 Not Not Indicator dropped with a available available available available available in March, 2015. functioning health information system (number) 19. PHUs jointly 10.00 90.00 Not Not 97.00 Not available Not 97.00 Not Not Indicator dropped supervised by available available available available available in March, 2015. DHMT and local council staff at least once in three months using the national approved checklists (%) 20. Quarterly 0.00 4.00 Not Not 4.00 Not available Not 4.00 Not Not Indicator dropped meetings available available available available available in March, 2015. held between Page 53 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) LGFD and MoHS (number) 21. PHUs for 80.00 90.00 Not Not 95.00 Not available Not 95.00 Not Not Indicator dropped which RCH available available available available available in March, 2015. reports can be produced for the previous month (%) 22. Health staff 0.00 25.00 Not Not 30.00 Not available Not 30.00 Not Not Indicator dropped trained in available available available available available in March, 2015. medical waste management (%) 23. Annual No Yes No Not Yes Yes Achieved Yes Yes Achieved Indicator dropped health sector achieved in March, 2015. report Annual health prepared reports produced (yes/no) in 2015 and 2016. 24. Health 0.00 100.00 Not Not 100.00 Not available Not 100.00 Not Not Indicator dropped workers available available available available available in March, 2015. captured in human resource information system (HRIS) (%) Page 54 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Supervision/ICR Irina Aleksandra Nikolic Task Team Leader Richard Olowo Procurement Specialist Sydney Augustus Olorunfe Godwin Financial Management Specialist Eileen Brainne Sullivan Team Member Aissatou Chipkaou Team Member Maria E. Gracheva Team Member John W. Fraser Stewart Social Safeguards Specialist Alexandra C. Bezeredi Social Safeguards Specialist Salieu Jalloh Team Member Raihona Atakhodjayeva Team Member Shiyong Wang Team Member Francisca Ayodeji Akala Team Member Daniel Rikichi Kajang Team Member Sariette Jene M. C. Jippe Team Member Beatrix Allah-Mensah Social Safeguards Specialist Allan Dunstant Odulami Cole Team Member Adanna Deborah Ugochi Chukwuma Team Member Alix Mary Louise Bonargent Team Member Innocent Kamugisha Team Member Anita Bimunka Takura Tingbani Environmental Safeguards Specialist Page 55 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) A. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY08 8.541 49,609.85 FY09 17.820 87,446.62 FY10 26.415 143,308.08 Total 52.78 280,364.55 Supervision/ICR FY11 3.729 34,522.99 FY12 5.235 117,819.00 FY13 2.886 40,175.50 FY14 20.422 63,646.21 FY15 15.818 85,204.03 FY16 13.338 65,929.39 FY17 25.715 92,084.48 FY18 10.728 52,367.91 Total 97.87 551,749.51 Page 56 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) 50 Closing (US$M) 51 (US$M) Strengthening of Service 30.498 30.099 98.69 Delivery Capacity Building 7.290 7.290 100.00 Support to Ebola Crisis 5.902 5.900 99.96 Response Total 43.69 43.29 99.08 50 These amounts reflect costs approved initially plus additional financing and re-allocations approved. 51 Integrated Project Management Unit. Final Report: Impact Evaluation of RCHP-2. Ministry of Finance and Economic Development. 2017. Page 57 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) ANNEX 4. EFFICIENCY ANALYSIS Summary 1. This annex summarizes an ex-post economic evaluation of Component 1 of the RCHP-2, Service Delivery Strengthening, which accounts for 69.5 percent of the actual Project expenditure, and has sustainability implications arising from the input and performance- based grants. As FHCI was implemented over the same period, targeting similar services and populations, this evaluation incorporates costs of FHCI. The evaluation follows a different approach from the efficiency analysis at Project Appraisal which used a Marginal Budgeting for Bottlenecks approach to estimate expected benefits; justified intervention selection by cost-effectiveness estimates in the literature; and identified additional financial resources required per child in Sierra Leone. We estimate the number of life years saved given increases in use of the PEHS compared to a without- Project scenario equivalent to maintaining pre-Project (2009) service use levels. The cost per life year saved of Component 1 of the RCHP-2 is $164, which indicates that this Component was cost-effective. Methodology 2. Given the overlap between costs and service coverage of FHCI and RCHP-2 Component 1, we are unable to clearly separately evaluate cost-effectiveness of these two national interventions. We obtained incremental FHCI costs from an evaluation of the FHCI in 2016. The average yearly increase in FHCI marginal costs between 2010 and 2013 was 22 percent. 52 We assumed a similar annual increase in incremental costs between 2013 and 2017. We consider the sum of cost of Component 1 plus the marginal service delivery costs of the FHCI from 2010-2017 to be the total marginal cost of these changes. 3. Component 1 via LC grants and distribution of LLINs, primarily focused on supporting service provision related to facility-based deliveries; ANC attendance; use of LLINs by children under five; child immunizations; and consultations for children under five years of age. The FHCI in addition covered skilled birth attendance; postnatal care attendance; and family planning. We estimate lives saved due to coverage of these interventions being higher than they would have been if they remained at 2009 values. To estimate marginal benefit related to these interventions, we use the Lives Saved Tool, which uses country-specific coverage data combined with data on causes of death and secondary 52 Witter et al. The Sierra Leone Free Health Care Initiative (FHCI): Process and Effectiveness Review. 27 April, 2016. See Table 11, we omit PBF estimates for the World Bank and other donor expenditure outside direct service delivery. Page 58 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) evidence on the effectiveness of interventions to model the impact of increased coverage on maternal and child mortality. 53 This allows us to model how increases in service coverage for the above indicators translates into reduced mortality among mothers and children, leveraging systematic review evidence on the links between service coverage and mortality rates. In the absence of a control group, given nationwide scope of the Project, we assume that in the absence of the Project (and FHCI), service coverage would have remained at 2009 levels. The data inputs into the Lives Saved Tool were derived from the updated RF and DHIS-2. 4. Where these sources did not have data, projections based on the 2008 and 2013 DHS in Sierra Leone were used to estimate maternal and child health intervention coverage. From the estimated deaths averted from service coverage, we derive number of life years saved using life expectancy and average age at death estimates54. Our estimates of cost-effectiveness are arrived at by dividing marginal costs (in US$) by marginal effects (in life years saved). Results 5. The estimated total incremental costs of FHCI from 2010 to 2017 was US$ 236 Million. The actual cost of Component 1 was US$30.099 Million. Thus, the estimated total marginal costs of achieving service delivery output increases would be US$ 266 Million. The total estimated number of deaths averted using the Lives Saved Tool was 62,403 with a total estimated Life Years Saved of 1,624,748. This estimate is plausible given the prior estimates of lives saved through the FHCI (which acknowledged overlap with the World Bank PBF and other donor programs) between 2010 and 2013 of 561,50055. Table 6: Estimated Deaths Averted Newborn 1-59 months Maternal 2010 1357 2,679 486 2011 820 5,058 87 2012 2243 6,284 603 2013 2610 7,666 638 2014 2657 6,012 649 2015 2905 6,026 703 2016 2798 -248 697 2017 2905 6,483 691 Total 18295 39960 4,554 53 LiST, Avenir Health, 2017. 54 United Nations, Department of Economic and Social Affairs, Population Division (2017). World Population Prospects: The 2017 Revision, Key Findings and Advance Tables. ESA/P/WP/248. 55 Witter et al. The Sierra Leone Free Health Care Initiative (FHCI): Process and Effectiveness Review. 27 April, 2016. See Table 11, we omit PBF estimates for the World Bank and other donor expenditure outside direct service delivery. Page 59 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) 6. Thus, the estimated cost per life year saved is US$ 164. A widely-used (but flawed56) threshold for cost-effectiveness in countries without pre-specified thresholds, promoted by the World Health Organization, considers an intervention to be highly cost-effective if it costs less than the annual GDP per capita per additional life year. Applied to the case of Sierra Leone [GDP/Capita (2016) = US$ 496.05], the combination of FHCI and Component 1 was potentially highly cost-effective. 56 Marseille et al. Thresholds for the cost-effectiveness of interventions: alternative approaches. Bulletin of the World Health Organization. 2015; 93:118-124. Page 60 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS A. Summary of Feedback from Stakeholder Workshop, Interviews, and Site Visits This tables below summarize feedback obtained through a stakeholder workshop attended by 86 participants, a series of in-depth individual and group interviews with stakeholders in the RCHP-2, and site visits to 2 health facilities. GROUP 1: Ministry of Finance and Economic Development Project Achievements Challenges Lessons Learnt Component/Aspect Component 1: Service • Input-based • Payment • Local councils can Delivery grants were misallocation and play a leading Strengthening used for health delays (due to role in health worker training, verification delays management if facility in DHMT and there is strong renovation, MoH, incorrect collaboration equipment facility bank between the purchasing, and details, as well as DHMT and MoHS support for bank processes) • Local councils internal reduced PHU and frontline verification of motivation to service providers performance by engage in PBF can appropriately DHMTs • Initial poor use additional • Performance- understanding of resources to based funds PBF including data scale local service provide monitoring, delivery activities autonomy to verification, and • Regular and health facilities payment timely PBF for small mechanisms payments are projects – • Non-SMART necessary to improving work indicators of keep frontline environment, performance service providers equipment and difficult to track engaged supplies of • Low capacity for • There is a need essential drugs performance to be more • Grants management at efficient in compensate for the facility level verification income that • Poor coordination process for PBF used to come of facility • Coaching of from out-of- management facility managers Page 61 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) pocket between DHMT and DHMTs is payments in the and FMT essential to facilities (now • High frequency improving the abolished under and low volume effectiveness of FHCI) input-based PBF schemes • Grants provide grants create • Dedicated staff cash flow to contractual for PBF in the communities as problems for MoHS are the resources facility necessary to may be spent in rehabilitation improve local businesses • Low capacity in efficiency of • Grants increase LCs to develop verification and facility annual work plans reduce payment resilience, when delays saved and used for extraordinary expenses Component 2: • 23 students • High turnover of • Unified fiduciary Capacity Building trained in staff within MoHS management relevant sub- negatively would improve specialties at influenced Project Master of compliance with management and Science level fiduciary ratings • Training of requirements – • Institutionalize facility leading to delays medical waste management in procurement management – committees on • Separation of clear structure PBF monitoring fiduciary team for and reporting in functions for accountability, 2015 and 2017 Component 2 and dedicated staff at • Plan for training 1 led to delays central and on medical due to costs of district level, waste coordination surveillance management • Delays in system, and developed appointing focal financing • National surveys person for waste were management undertaken: The prevented MWM 2013 implementation Demographic • Weak capacity in and Health the MoHS, Page 62 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) Survey, the 2011 DHMTs, and Health Facility facility to Survey, and operationalize external MWM plan verifications of • Poor working the PBF in 2013 relationship and 2016 between facility staff and environmental health officers in DHMTs • DHMT did not prioritize medical waste management in expenditure plans Component 3: • There was a • EVD crisis • Coordination of Emergency Response rapid, concerted changed priorities multiple external to the Ebola Crisis response to the to focus on and internal crisis involving sensitization of players in the multiple the community, health sector is stakeholders and ring-fenced funds possible a national were diverted/ • There is a need response center less focus on to increase facility capacity to improvement respond to emergencies such that other service delivery is not affected GROUP 2: Ministry of Health and Sanitation Project Achievements Challenges Lessons Learnt Component/Aspect Component 1: Service • Introduction of • Weak supply • Insisting spending Delivery Facility chains for plans for PBF are Strengthening Improvement essential signed off by Teams (FIT) medicines, other staff in the framework that excluding anti- facility prior to focused LC malarial drugs, approval to Page 63 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) grants on and gaps were ensure inclusion attaining safe not closed by LC and transparency facility status for grants • Important to obstetric care • Lack of strengthen supply transparency by chains as facility in-charges complementary in decision- to motivating making for the staff for service use of PBF funds delivery using PBF for facility improvements Component 2: • Introduction of • Unclear • Supervision, Capacity Building FITs who processes for coaching, developed transfer of onboarding of framework for capacity from new staff enhancing IHPAU to MoHS • Clarity on roles of capacity for fiduciary staff IHPAU and means obstetric care • Indicators that of engaging other • PHU facility required data- MoHS staff management sharing with • Need to integrate committees set other agencies data collection up e.g. Malaria systems across • Medical waste program in multiple partners management MoHS, were plan was difficult to track developed to • Duplication and improve facility fragmentation of safety following monitoring and increased service evaluation demand systems across • Later in the donors and non- project, a focal profit person was organizations appointed to • Information from deliver training health on MWM in information DHMTs systems was not • IHPAU instituted used for policy to build capacity and projects in MoHS for Page 64 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) project management • Training of all monitoring and evaluation officers in the DHMT and MoH on verification of PBF results • Hospital information systems now submit aggregate data on key indicators GROUP 3: District Councils Project Achievements Challenges Lessons Learnt Component/Aspect Component 1: Service • Facility grants • Delays in • Prior to PBF, Delivery improved the verification and important to Strengthening status and payment of PBF ensure motivation of grants leads to verification PHUs conflicts between procedures are • Grants Provided facilities and clarified among complementary contractors for all parties funding for LC rehabilitation • Grant development • There is low absorptive budgets absorptive capacity of capacity for both facilities must input and be considered in performance- PBFs based grants in some facilities, leaving funds unspent despite needs Component 2: • Trainings were • Most of the funds • Important to Capacity Building held in M/E, PBF were used for reduce administrative administrative Page 65 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) implementation, (computers, costs of and MWM other equipment, programs staffing) rather than programmatic expenses and may not contribute to achievements GROUP 4: District Health Management Teams Project Achievements Challenges Lessons Learnt Component/Aspect Component 1: Service • PBF provided • Verification of PBF • There should be Delivery increased was burdensome clarity on the Strengthening attention to and initially there role of MoH service delivery were inadequate versus DHMTs in outputs rather financial providing than just inputs resources for technical • PBF provided DHMTs to oversight for PBF flexible funds undertake this • Extensive that were task consultations available in EVD • There was very with service crisis to meet little oversight for providers and health needs the programmatic policy makers • Funds increased aspects of should precede ownership of implementation selection of PBF local health of PBF from MoH, indicators service e.g. for internal • Demand-side improvement by verification of financing for facility performance community • PBF created engagement may incentives to be a useful focus on the complement to indicators being supply-side PBF verified, while initiatives other important • To sustain PBF, indicators lagged the central government Page 66 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) e.g. indicators of should be service quality encouraged to • The irregular re-start the “cash disbursement for PHU” pattern led to initiative but scale down of reward results planned activities and defaults in contracts • Non-involvement of other relevant ministries (including Ministry of Works) led to loss of economies of scale in facility rehabilitation • The lack of demand-side financing to engage communities was a constraint to improving service use • Cases of mis- procurement because items e.g. boat ambulance were too complex for a local council or DHMT to manage Component 2: • LCs funded • Lack of clarity on • Clarify roles of Capacity Building improvements in the roles and multiple agencies waste responsibilities of responsible for management in environmental waste Bo and Kenema health officers in management, in DHMTs, Infection a national plan prevention and Page 67 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) control unit in • Environmental MoHS, and local health officers councils in MWM should be • There was little trained in MWM awareness at the • MWM should be start of the integrated into Project within LCs the annual work of the need to plans of LCs, implement a cannot be MWM plan managed centrally GROUP 5: PHU and Hospital Staff Project Achievements Challenges Lessons Learnt Component/Aspect Component 1: Service • Funds enabled • Some constraints • Important to Delivery facilities to to achieving undertake Strengthening protect staff service targets complementary from EVD via were outside activities that training and facility control improve facility protective gear e.g. the services, in • As fund flows maldistribution of addition to PBF, from the central health workers such as supply of government are between urban health workers delayed, LC and rural areas • Ensuring not to grants stopped • Too many overburden staff gaps in funding indicators were with M/E operational costs tracked under the requirements of and staff PBF scheme PBF incentives may • The rewards for • Clarity on criteria partly cover outputs under for choosing salaries PBF were not pattern of roll-out • PBF was adjusted to of PBF associated with reflect exchange improvements in rate fluctuations the behavior of • Confusion on the staff including lack of extension punctuality, of PBF to non- Page 68 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) innovation, and PHU facilities in efficiency the country GROUP 5: Beneficiaries (Mothers and Trainees) Project Achievements Challenges Lessons Learnt Component/Aspect Component 1: Service • There were • Poor links • PBF and Input Delivery improvements in between grants reduce Strengthening facilities e.g. hospitals and the need for medical PHUs: One informal out-of- equipment, woman lost her pocket fees benches, supplies daughter because • Facilities are • Staff were better of a later referral able to use motivated to following delivery grants to provider services complications improve service • Insecticide- delivery treated nets • There is a need protected to strengthen households from links between malaria PHUs and • Care is now free higher-level at the point of maternal and service, there are child health care fewer informal and formal fees Component 2: • Visiting • Program length • Sub-specialty Capacity Building professors from too short training locally Nigeria filled gaps increases in medical and retention of allied health trainees in local science training institutions • The first set of • Capacity for post-graduate service provision fellows in and training can pharmacy sub- be exchanged specialties were between African trained under the institutions program Page 69 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) • Graduates were mentored by professors in teaching methods and retained as lecturers • Students could undertake laboratory research in external facilities Page 70 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) B. Summary of Borrowers Implementation Completion Report Background 1. Prior to the establishment of decentralization in 2004, Sierra Leone had some of the poorest health indicators in the world, with a life expectancy of 47 years, an infant mortality rate of 92 per 1000 live births, an under-five mortality rate of 156 per 1,000 live births, and a maternal mortality ratio of 1,165 per 100 000 births. 2. The Government launched the Reproductive and Child Health Strategic Plan in February 2008, and Phase 1 of the Reproductive and Child Health Project (RCHP I) was designed to complement the RCHSP with the primary objective of helping the GoSL address the immediate constraints to reducing long-term maternal and under-five mortality. RCHP I enabled the identification of weaker performing LCs that were targeted to receive intense supervision in RCHP II. 3. The President of Sierra Leone also launched the Reproductive and Child Health Strategic Plan (RCHSP) in 2008 aiming to reduce maternal, under-five, and infant mortality rates by 30 percent of the 2005 values by 2010; and the Free Health Care policy for pregnant women, lactating mothers and children under five on 27th April 2010, the same year as RCHP 2 became effective. 4. The Project Development Objective as stated was: “to increase utilization of a package of essential health services by pregnant and lactating women and children under the age of five and support the emergency response needed to contain and control the Ebola crisis”. This was done by providing additional funds to Councils for frontline service provision, and by improving the quality of services through better-trained staff, greater availability of equipment and drugs, and better infrastructure. . Project Components and Achievements 5. Component 1: Strengthening Service Delivery a) Sub-Component 1.1: Local Council Grants The 19 local councils benefited from funds under this scheme to the tune of US$6.9 million for the period 2011-2017. Of this, about 34.8% (US$2.4 million) was expended on the fight against Ebola Virus Disease. A pre-requisite for disbursement of RCHP 2 funds was submission of approved annual work plans and budgets, procurement plans, revised development plans and signed subsidiary and service provider agreements. Funds under this sub-component were used to support the effective functioning of District Health Management Teams (DHMTs) and health Page 71 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) facilities via equipment purchase and maintenance, facility rehabilitation, supervision of performance-based financing, etc. b) Sub-Component 1.2: Performance-Based Financing About 1,200 peripheral health units (“PHUs”) in the country and two hospitals (Ola During Children’s Hospital and Princess Christian Maternity Hospital) in Freetown benefited from funds under the PBF scheme to the tune of about US$16.62 million for the period 2011-2017. The scheme, focused on six key reproductive and child health care indicators that have proved to be critical for the reduction of child and maternal mortality. They include: i) family planning; ii) antenatal consultations; iii) safe deliveries; iv) postnatal consultations; v) full vaccination of children under one; and vi) outpatient consultations for children under five years. In addition, there were quality scores applied to rewards based on i) attendance at work; ii) timeliness of submission of HMIS Report; iii) functioning health management committee; iv) display of updated information; v) appropriate record keeping; v) cleaning of facility; vi) good drugs management; vii) no stock-out of essential drugs; and ix) appropriate medical waste management procedures in place and being observed. There was an overall improvement in not only the status of the facilities but also in the attitude of in-charges and other health personnel in the delivery of health care. This led to a significant increase in the number of patients visiting the PHUs. c) Sub-Component 1.3: Procurement and Distribution of Bed Nets A total of 1.16 million bed nets were procured and distributed and the immediate post malaria campaign assessment showed a high coverage level of 98.7 percent. d) Sub-Component 1.4: Local Council Grant Administration This sub component supported the implementation of the PBF program and management of the entire component. Joint quarterly monitoring of RCHP 2 expenditure and activity implementation in LCs (DHMTs and health facilities) was coordinated by the LGFD and reports on the utilization of the grants were prepared by the councils which were subsequently reviewed by IPAU in collaboration with LGFD. Technical and mentoring support missions were also facilitated by the LGFD/IPAU to develop RCHP 2 annual work plans. The support aligned activities with the Government Agenda for Prosperity (A4P), coded activities in line with the Government’s coding of the A4P and other suitable activities to be consistent with the PDO. 6. Component 2: Capacity Building There was capacity building in M&E at all levels of the health system (facility, chiefdom, local council, and national), the LCs district information management systems, and the Ministry’s Health Information System were also strengthened. It supported key institutional organization and management for the sector as well as health personnel training. The ministry through Page 72 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) support of the project conducted a comprehensive assessment of facilities using the Facility Improvement Team (FIT). The FIT framework was the objectively verifiable roadmap to achieving safe health facilities. FIT has helped improved coordination among partners. Project interventions around strengthening of facilities, providing obstetric, maternal and neonatal care were guided by the FIT framework. The project supported training to strengthen the DHIS and supported the introduction of a Hospital Information System. It financed independent data quality audits, and served as quality-control for data used in PBF, including surveys of a sample of health facilities to review patient registers, and assess the reliability of reported statistics and record-keeping practices. A total of 5 external Lecturers and 2 Local Lecturers were recruited for COHMAS under RCHP II to support the training of the health officers in sexual, reproductive ad child health care. Four (4) of these external Lecturers were also recruited for the School of Community Health Sciences, Njala University. Overall, a total of 23 students were trained at M.Sc. level in the basic para-clinical and pharmaceutical sciences. The post graduates have been recruited by the University of Sierra Leone and they are now lecturing under graduate trainees. Teaching Assistants who have been part of RCHP II funded training have been promoted to Lecturer 2 status. The project improved organization, management, coordination and monitoring of health sector donor funds through the establishment of the Integrated Health Project Administration Unit (IHPAU) for fund management and disbursement. The establishment and integration of the IHPAU into the operating structures of the MOHS has positively contributed to the consolidation, management and adequate oversight over all externally financed projects within the MOHS. 7. Component 3: Support to the fight against the Ebola Virus Disease Outbreak To respond to the Ebola outbreak, the project was restructured and made available $6M for the first funding to support the fight against the EVD which was initially earmarked to support the response efforts, finance hazard pay for Ebola-response workers and some procurement of supplies for the response. The project provided support for 130 Doctors and other medical workers contracted to support the Ebola control effort and provide backup medical services Major Implementation Issues Encountered 8. There were weaknesses in overall project oversight due to coordination challenges between the MoHS and MoFED, particularly in monitoring the implementation of the performance based financing for which the two agencies were responsible. 9. MoHS experienced significant capacity gaps that negatively affected Component 2 of the Project including delay in appointing focal points for health education and medical Page 73 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) waste management. Additionally, the unexpected departure of the core MOHS team in the Directorate of Policy, Planning, and information severely disrupted the smooth implementation of component 2. It was difficult to replace their core competencies and the project had to rely on Overseas Development Institute fellows to fill the capacity gaps. 10. The creation of two separate fiduciary units in MoHS and MoFED created a lot of challenges in overall project implementation and reporting. While the MoFED team is experienced and familiar with bank processes and procedures, there were delays in procurement and financial management compliance in the MoHS. This problem was compounded by the frequent turnover of fiduciary staff at the MOHS. The fiduciary staff handling component 2 were mainstream civil servants who are subjected to frequent transfers; and some of them do not have the minimum requirement to manage World Bank projects at that level. The project faced several challenges in procuring an ambulance boat for the riverine areas. This was due to the nature of the procurement and preparation of technical specification. 11. The EVD crisis worsened existing shortages of skilled health providers as there were 864 reported health worker deaths. The EVD crisis also had a human cost and the loss of human resources affected implementation and monitoring of RCHP-2. Fear of contracting the disease and misinformation also contributed to a decline of health service use. At the peak of the EVD funds meant for the implementation of non-EVD related activities were all diverted to the fight the virus. 12. Some of the indicators in the results framework were poorly defined and the means of data collection was unclear. This affected monitoring and proper supervision, determining the outcome of some projects activities, and updating the PDO indicators. 13. Late submission of financial returns on previous disbursed funds by local councils impacted negatively on the timely disbursement of grants to council and implementation of project activities in a timely manner. Late verification on utilized PBF funds by the DHMTs, which is a requirement for subsequent disbursements of funds under the program in part explains delays encountered in the disbursement of funds to health facilities. 14. The project experienced Exchange loss between the SDR and the US Dollar which impacted negatively on available funds for implementation to all stakeholders. Page 74 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) Project Sustainability 15. The project development objective aligned with the Government’s policy, as articulated in the PRSP III (Agenda for prosperity) for the health sector and directly supports the Free Healthcare Initiative (FHCI). In the short to medium term it is likely that some of the positive outcomes of the project will be sustained as government continues to implements the FHCI. 16. Whilst government will continue to provide funding to the sector, these funds are unlikely to match the level provided under RCHP-2. This is particularly true for the PBF scheme which involved significant quarterly transfers to facilities. However, given the level of support from partners to the sector, effective coordination and harmonization of financial support could provide additional resources for activities conducted under RCHP-2. 17. Investments in medical training under the project are likely to be sustained through support provided by development partners in other projects. Implementation of services at the district level is well established. It is likely that capacity building and management reforms undertaken through support from other development partners will help in sustaining gains made under the project. Lessons Learnt 18. Exclusion of relevant stakeholders at the onset of sector projects may result to non- compliance or low buying-in of these stakeholders in the implementation. This may explain the low engagement of DHMTs and PHUs on Medical Waste Management under RCHP-2. 19. Investment in capacity building for all stakeholders involved in projects with complex arrangements like the PBF is essential if all parties are to effectively deliver on their mandates. This should occur at the onset of the project and must include coaching and mentoring as well as clarifying roles and responsibilities. 20. It was difficult to have piloted the PBF scheme given that it was introduce to complement the nationwide FCHI. Otherwise, piloting implementation of such scheme with complex implementation arrangements could have help in providing lessons learnt to inform scale up. 21. The involvement of a M&E Specialist at the onset of discussions on the design of the results framework is important to address potential challenges relating to tracking or measurement of indicators, reporting period on the indicators etc. For example, few Page 75 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) indicators were arbitrarily dropped as they were not considered necessary for tracking changes along the results chain. This is particularly important when there is no system to routinely collect high quality data. Page 76 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) 1. Caulker et al. Life goes on: The resilience of maternal primary care during the Ebola outbreak in rural Sierra Leone. Public Health Action. 2017; 7 (Suppl. 1) 2. Edit V. Velenyi (2016) Health Care Spending and Economic Growth. World Scientific Handbook of Global Health Economics and Public Policy (pp. 1-154). 3. Integrated Project Management Unit. Final Report: Impact Evaluation of RCHP-2. Ministry of Finance and Economic Development. 2017. 4. Kerber et al (2007). Continuum of care for maternal, newborn, and child health: From slogan to service delivery. Lancet. 370: 1358-69. 5. Lassi, Zohra S et al. “Essential Interventions for Child Health.” Reproductive Health 11. Suppl. 1 (2014): S4. 6. LiST, Avenir Health, 2017. 7. National Health Sector Strategic Plan (NHSSP) 2010-2015 8. Phyllis Benonia Ocran 2009. Essential Medicines and Consumables Quantification Exercise, Supply Chain Assessment, May 2009. Freetown, Sierra Leone: UNICEF/ MoHS. 9. Project Documents: Aide Memoires, Implementation Status & Results Reports, Methodology for RCHP-2 M&E, Restructuring Papers, Project Appraisal Document, and Project Papers. 10. Ross et al. Sierra Leone’s Response to the Ebola Outbreak: Management Strategies and Key Responder Experiences. Chatham House. March 2017. 11. Second Poverty Reduction Strategy (PRSP-II) 2008-2012 12. Seray Consulting Associates. Abridged Final Report: External Verification of the PBF Scheme. Submitted to the Integrated Project Administration Unit, Ministry of Finance and Economic Development. 2016. 13. Statistics Sierra Leone (SSL). Core Welfare Indicators Questionnaire, 2007. 14. United Nations Children’s Fund (2015). Levels and Trends in Child Mortality: Report 2015. Estimates Developed by the UN Inter-Agency Group for Child Mortality Estimation 15. Witter et al. The Sierra Leone Free Health Care Initiative (FHCI): Process and Effectiveness Review. 27 April, 2016. Page 77 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) 16. World Bank and African Development Bank. Joint Assistance Strategy (JAS) Report No. 52297-SL, March 4 2010 17. World Bank. Strengthening the Development Partnership and Financing for Achieving the MDGs: An Africa Action Plan, September 2005 18. World Health Organization (2015). Trends in Maternal Mortality: 1990 to 2015. Estimates by WHO, UNICEF, UNFPA, World Bank Group and the United Nations Population Division. Page 78 of 75 The World Bank Reproductive and Child Health Project - Phase 2 ( P110535 ) ANNEX 7. SIERRA LEONE COUNTRY MAP Page 79 of 75