Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00004994 IMPLEMENTATION COMPLETION AND RESULTS REPORT Credit Number 5229-LS TF 14147 ON A CREDIT IN THE AMOUNT OF SDR7.8 MILLION (US$12 MILLION EQUIVALENT) TO THE KINGDOM OF LESOTHO FOR A HEALTH SECTOR PERFORMANCE ENHANCEMENT PROJECT February 14, 2020 Health, Nutrition & Population Global Practice Africa Region CURRENCY EQUIVALENTS (Exchange Rate Effective {Sep 26, 2019}) Currency Unit = = US$1 US$ 1.38 = SDR 1 FISCAL YEAR July 1 - June 30 Regional Vice President: Hafez M. H. Ghanem Country Director: Marie Francoise Marie-Nelly Regional Director: Dena Ringold Practice Manager: Magnus Lindelow Task Team Leader(s): Christine Lao Pena, Omer Ramses Zang Sidjou John Stephen Osika, Amer Mikail Dastgir; Katelyn Jison ICR Main Contributor(s): Yoo ABBREVIATIONS AND ACRONYMS AIDS Acquired Human Immunodeficiency Syndrome CBA Cost-Benefit Analysis CHAI Clinton Health Access Initiative CHAL Christian Health Association of Lesotho CLNHCWMP Consolidated Lesotho National Health Care Waste Management Plan CPA Complementary Package of Activities CPF Country Partnership Framework DHIO District Health Information Officer DHMT District Health Management Team DHS Demographic and Health Surveys DPs Development Partners EGPAF Elizabeth Glasier Pediatric AIDS Foundation EmONC Emergency Obstetric and Neonatal Care ESAMI Eastern and Southern African Management Institute GOL Government of Lesotho GPOBA Global Partnership for Output-Based Aid HCWM Health Care Waste Management HDI Human Development Index HIV Human Immunodeficiency Virus HMIS Health Management Information Systems HNP Health, Nutrition and Population HRAA Human resources Alliance for Africa HRH Human Resources for Health HRITF Health Results Innovation Trust Fund ICT Information and Communication Technology IDA International Development Association IFC International Finance Corporation IRI Intermediate Results Indicator IRR Internal Rate of Return ISR Implementation Status and Results LeBoHa Lesotho Boston Health Alliance LENASO Lesotho Network of AIDS Services Organizations LRCS Lesotho Red Cross Society M&E Monitoring and Evaluation M2M Mothers to Mothers MCA Millennium Challenge Account MCC Millennium Challenge Corporation MCH Maternal and Child Health MDGs Millennium Development Goals MICS Multiple Indicator Cluster Survey MMR Maternal Mortality Ratio MNH Maternal and Newborn Health MODP Ministry of Development Planning MOF Ministry of Finance MOH Ministry of Health MPA Minimum Package of Activities NDSP National Strategic Development Plan NSRHSC National Sexual and Reproductive Health Steering Committee PBF Performance-Based Financing PDO Project Development Objective PIH Partners in Health PPP Public Private Partnership PPTA Performance Purchasing Technical Assistance RF Results Framework SACU Southern African Customs Union TA Technical Assistance TB Tuberculosis UNFPA United Nations Population Fund UNICEF United Nations Children's Fund VHWs Village Health Workers WFP World Food Programme WHO World Health Organization The World Bank Lesotho Health Sector Performance Enhancement (P114859) TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 7 A. CONTEXT AT APPRAISAL .........................................................................................................7 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) ..................................... 14 II. OUTCOME .................................................................................................................... 18 A. RELEVANCE OF PDOs ............................................................................................................ 18 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 19 C. EFFICIENCY ........................................................................................................................... 26 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 28 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 28 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 30 A. KEY FACTORS DURING PREPARATION ................................................................................... 30 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 31 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 32 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 32 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 33 C. BANK PERFORMANCE ........................................................................................................... 34 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 35 V. LESSONS AND RECOMMENDATIONS ............................................................................. 36 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 38 ANNEX 2. PDO AND INTERMEDIATE RESULTS INDICATORS AT APPRAISAL, 2016 AND 2018 RESTRUCTURINGS ............................................................................................................... 51 ANNEX 3. PDO UTILIZATION INDICATORS’ ACHIEVEMENTS COMPARING ‘WITH’ AND ‘WITHOUT’ PHASE 3 OF PROJECT IMPLEMENTATION ............................................................................. 57 ANNEX 4. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 58 ANNEX 5. PROJECT COST BY COMPONENT .......................................................................... 60 ANNEX 6. EFFICIENCY ANALYSIS ........................................................................................... 61 ANNEX 7. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 66 The World Bank Lesotho Health Sector Performance Enhancement (P114859) The World Bank Lesotho Health Sector Performance Enhancement (P114859) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P114859 Lesotho Health Sector Performance Enhancement Country Financing Instrument Lesotho Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency Ministry of Finance Ministry of Health, Ministry of Finance Project Development Objective (PDO) Original PDO The overall project development objective is to improve the utilization and quality of maternal and newborn health (MNH)servicesinselected districts in Lesotho. Revised PDO The overall project development objective is to: (i) increase utilization and improve the quality of primary health services inselected districts in Lesotho with a particular focus on maternal and child health, TB and HIV; (ii) improve contract management ofselect PPPs; and (iii) in the event of an Eligible Crisis or Emergency, to provide immediate and effective response to saidEligible Crisis or Emergency. Page 1 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 12,000,000 12,000,000 10,821,783 IDA-52290 4,000,000 4,000,000 3,906,234 TF-14147 Total 16,000,000 16,000,000 14,728,017 Non-World Bank Financing 0 0 0 Borrower/Recipient 4,000,000 0 2,551,107 Total 4,000,000 0 2,551,107 Total Project Cost 20,000,000 16,000,000 17,279,124 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 11-Apr-2013 14-Feb-2014 13-Mar-2017 30-Jun-2017 30-Jun-2019 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 15-Nov-2016 2.96 Change in Implementing Agency Change in Project Development Objectives Change in Results Framework Change in Components and Cost Change in Loan Closing Date(s) Reallocation between Disbursement Categories Change in Legal Covenants Change in Institutional Arrangements Change in Procurement Change in Implementation Schedule Other Change(s) 05-May-2018 8.91 Change in Results Framework Change in Components and Cost Reallocation between Disbursement Categories Page 2 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Moderately Satisfactory Modest RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 30-Nov-2013 Satisfactory Satisfactory .56 02 25-Jun-2014 Moderately Satisfactory Moderately Unsatisfactory .80 Moderately 03 18-Dec-2014 Moderately Unsatisfactory 1.54 Unsatisfactory Moderately 04 12-Jun-2015 Moderately Unsatisfactory 1.92 Unsatisfactory Moderately 05 23-Dec-2015 Moderately Unsatisfactory 3.04 Unsatisfactory Moderately 06 30-Jun-2016 Moderately Unsatisfactory 3.80 Unsatisfactory Moderately 07 29-Dec-2016 Moderately Unsatisfactory 4.19 Unsatisfactory 08 21-Jul-2017 Moderately Satisfactory Moderately Satisfactory 6.65 09 13-Feb-2018 Moderately Satisfactory Moderately Satisfactory 8.85 10 21-Sep-2018 Moderately Satisfactory Moderately Satisfactory 12.06 11 03-Apr-2019 Moderately Satisfactory Moderately Satisfactory 14.90 Moderately 12 28-Jun-2019 Moderately Satisfactory 15.84 Unsatisfactory SECTORS AND THEMES Sectors Major Sector/Sector (%) Health 100 Public Administration - Health 30 Health 70 Page 3 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Human Development and Gender 0 Disease Control 5 HIV/AIDS 5 Health Systems and Policies 95 Health System Strengthening 35 Reproductive and Maternal Health 40 Child Health 20 ADM STAFF Role At Approval At ICR Regional Vice President: Makhtar Diop Hafez M. H. Ghanem Country Director: Asad Alam Marie Francoise Marie-Nelly Director: Ritva S. Reinikka Dena Ringold Practice Manager: Olusoji O. Adeyi Magnus Lindelow Christine Lao Pena, Omer Task Team Leader(s): Kanako Yamashita-Allen Ramses Zang Sidjou ICR Contributing Author: John Stephen Osika Page 4 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) EXECUTIVE SUMMARY This project was prepared at a time when performance-based financing (PBF) was a relatively new concept in Lesotho. During project appraisal, the PBF experience in the country was limited to the experience of ‘Partners in Health’ in selected districts. That experience had little involvement of central level Ministry of Health (MOH) and Christian Health Association of Lesotho (CHAL), who are the main providers of health services in the country. There was, therefore, limited understanding by country stakeholders, during project preparation, of how PBF works and its benefits. During project preparation, the task team made efforts to prepare government officials for the new project, including carrying out two feasibility studies in 2010, arranging for eight staff from central MOH and district levels of the health system to participate in an African Regional Result Based Financing workshop, organized by the World Bank in Zambia (May 29- June 1, 2012). MOH Directors were also supported to undertake a PBF study tour to Rwanda and Zimbabwe between August 12 and August 18, 2012. The project team, based on lessons from other countries, designed a phased implementation of the project, i.e. starting as a pilot in two districts (the Government selected Leribe and Quthing as the pilot districts) during the first year of implementation (phase 1), then scaling up to phase 2 with four additional districts, before moving to phase 3 with the final three districts of the project for a total of nine districts (Maseru district was initially excluded from the project although it was incorporated during the last phase of project implementation). The project was phased to ensure that lessons are learned in the initial districts that could then be applied during the scale-up to other districts. Project preparation time was relatively long, but necessary to allow for sufficient preparation and buy-in by key stakeholders. During the initial stages of project implementation, the project encountered numerous implementation challenges including delays in constituting a fully-staffed PBF team at the MOH and in contracting and operationalizing the performance purchasing technical assistance (PPTA) firm to provide technical assistance in building in-country capacity, and capacity challenges in the government’s financial management system that was to be used by the project. The World Bank supervision team provided regular project supervision enabling the team to identify the need for and implement two project restructurings (in November 2016 and May 2018, respectively) and produce a total of 12 Project Implementation Status Reports (ISRs). The two restructurings contributed to subsequently better project performance, particularly in terms of quality, leading to project expansion to all the originally planned 9 districts, in addition to the district of Maseru, thus covering all of Lesotho’s districts. A key element in the restructuring of November 2016 was the introduction of a component to support capacity building for contract management for public private partnerships (PPP). This was a highly relevant component as Lesotho was implementing a PPP jointly supported by the World Bank Group through the International Finance Corporation (IFC) for Queen Mamohato Memorial Hospital (QMMH) in Lesotho. This PPP was at that time considered a flagship PPP in the Africa region and, therefore, important for the region and the health system in Lesotho in particular. The project was able to demonstrate the benefits of the PBF approach to service delivery in Lesotho, to the extent that, by the end of the project, stakeholders1 interviewed by the ICR team expressed unanimous support for the continuation of the PBF approach in Lesotho. Stakeholders highlighted the contribution of the project to increasing utilization of services and quality of services with the application of facility quality checklists. Facility 1Interviewed stakeholders include Central and District level officials, District Health Management Teams, and health facility staff. Page 5 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) staff and other stakeholders noted the equipment, staffing, training, coaching and mentoring provided by the project which supported utilization and quality of services. In terms of results framework indicators, at project end-line, the project exceeded its outcome targets for full immunization of children under 1 year and for treating people with HIV; partially achieved its outcome targets for women using modern contraceptive method and for detecting and treating underweight children under 5 years; and showed no increase in new Tuberculosis (TB) patient treatments, mainly due to continued decline in the TB incidence – this is a positive outcome. In improving the quality of primary health care services, the project demonstrated continued progress and surpassed its outcome target measured by the Quality of Care Score composite indicator. After the closure of the project, the MOH continued to implement the PBF approach by paying for performance from counterpart funding that had been mobilized for the additional financing (AF) of the project, even though the AF did not happen due to WB portfolio consolidation, with planned activities to be incorporated in the follow-on health sector project (Nutrition and Health System Strengthening Project/NHSSP P170278). The Government’s continued financing of the PBF approach indicates the government’s commitment to implement PBF in the country beyond the project’s life-span. The government’s strong buy-in can contribute toward improving quality of care in district hospitals and help create systemic efficiencies by alleviating unnecessary demand for services provided by the PPP referral hospital. The upcoming NHSSP is designed to finance the provision of quality and bonus grants to eligible health facilities by using the government system, based on the lessons learned under this project. This project has created awareness on ways to improve value for money in the QMMH PPP contract and the upcoming project will support PPP contract renegotiation. This project also highlighted a few health system issues such as a weak supply chain and referral system as well as village health worker program implementation constraints. The upcoming project will contribute to strengthening these areas. The government, through the leadership of the Ministry of Development Planning, has expressed interest in implementing the PBF approach in other sectors as well. This further demonstrates the contribution that this project has made to understanding PBF and its benefits in Lesotho and the political will to implement and sustain the approach. Based on its assessment, the ICR team has rated the overall outcome of the project as moderately satisfactory. This rating differs from the rating of the final Implementation Status Report (ISR) as the ICR team had access to additional data which were not available at the time of the final ISR. The ICR team used project data up to June 2019, while the final ISR used project data up to only December 2018. The ICR team also used data from the Multiple Indicator Cluster Survey (MICS) which were not available when the last ISR was prepared. Page 6 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Country Context 1. Lesotho is a landlocked country surrounded by South Africa, covering an area of 30,355 square kilometers. The population during project appraisal was estimated at 1.9 million and had a growth rate of less than 1 percent per annum. Nearly a quarter of the population was residing in Maseru district, the country’s capital. Lesotho had in recent years witnessed improvements in its key macroeconomic indicators. Lesotho was a lower middle- income country with per capita gross national income of US$1,210 and annual GDP growth of 3.7 percent in 2011. Inflation reached almost 7 percent at the end of 2011. Inflation was induced by high international commodity prices and agricultural supply shortages as a result of floods that hit Lesotho in 2010-2011. Lesotho suffered a significant drop in revenues from the Southern African Customs Union (SACU) in fiscal year 2010/11. In FY2011/12, higher government spending placed additional pressure on fiscal and external balances, increasing the fiscal deficit from 5 percent of GDP from 2010/2011 to about 10.3 percent of GDP. In FY2012/13 the fiscal balance was projected to reach 5.7 percent of GDP given the doubling of SACU revenues and under- execution of capital expenditures. 2. Lesotho continued to have one of the highest levels of poverty and inequality with 57 percent of the population living below the national poverty line and a Gini coefficient of about 0.63 (based upon 2003/04 survey data). It was ranked 160 out of 187 in the 2011 United Nations Human Development Index (HDI). The Government of Lesotho (GOL) had identified ‘Improve health, combat HIV and AIDS and reduce vulnerability’ as one of the six key pillars of the 2012/13 to 2016/17 National Strategic Development Plan (NSDP). The Strategic Plan was the implementation strategy of the country’s Vision 2020 that envisage s that by 2020, Lesotho shall be a stable democracy, a united and prosperous nation, that is at peace with itself and its neighbors. The GOL had significantly increased its allocations to the health sector from US$147.80 million in fiscal year 2009/10 to US$186.70 million in fiscal year 2011/12. Comparable expenditure data at the time showed that Lesotho spent on average US$33.20 per capita over the period 2004/5 to 2009/10. The Ministry of Finance (MOF), Ministry of Development Planning (MODP) and the Ministry of Health (MOH) expressed commitment to bring efficiency and results to health sector public spending. 3. The Country Assistance Strategy (CAS) for Lesotho (2010-2014) identified human development and service delivery as one of the three main areas of strategic engagement between the World Bank and the government of Lesotho. Within this strategic area, the CAS supported the government’s efforts to reduce the incidence of HIV/AIDS, increase access to hospital services and improve the quality of service provision. 4. Lesotho has 10 administrative districts, with Maseru that encompasses the capital city being the most populous district and the most prosperous. On the other hand, Quthing is among the most socio-economically challenged districts in the country. Each district is led by a district administrator. Sectoral and Institutional Context 5. Divergence between economic growth and human development in Lesotho was evident in the country’s poor health outcomes. Lesotho was off track to meet the Millennium Development Goals (MDGs) 4 (reducing child mortality) and 5 (improving maternal health). The Demographic and Health Survey (DHS) of 2009 reported Maternal Mortality Ratio (MMR) to be very high at 1,155 per 100,000 live births. WHO/UNICEF/UNFPA/World Page 7 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Bank report of 2012 indicated that the average annual percentage decline in MMR between 1990 and 2010 was only 0.9 percent, which was less than the 5.5 percent or more needed to be “on track” towards achieving MDG 5. Since MDG 5 is considered a proxy indicator for overall health system functioning, the lack of progress in this indicator was of serious national concern. Under-five mortality rate was also of concern as it was estimated to have only slightly decreased from 89 deaths per 1000 live births in 1990 to 86 deaths per 1000 live births in 2011. 6. Complications during pregnancy and delivery were the primary cause of maternal morbidity and mortality in Lesotho. The MOH Annual Joint Review of 2011/2012 indicated that the most frequent cause of female admissions at health facilities was abortion complications at 16 percent, followed by HIV/AIDS at 10 percent. The 2009 DHS indicated that deliveries attended by skilled providers (doctors/nurses/midwives) increased from 55 percent in 2004 to 61 percent in 2009, but wide disparities based on income still existed. In the wealthiest quintile, 90 percent of women delivered with assistance of skilled health personnel compared to only 35 percent of the women in the poorest quintile. 7. Two main providers dominated Lesotho’s health system: the MOH and the Christian Health Association of Lesotho (CHAL). The health system consisted of four-tiers: (i) tertiary and specialized hospitals; (ii) district hospitals; (iii) filter clinics and health centers; and (iv) village health posts. Lesotho had 10 administrative districts. In 2009, there were 216 health facilities across the country including one national referral hospital, two specialized hospitals, 19 hospitals, 190 health centers and four filter clinics. Among the 216 health facilities, 97 were operated by MOH, 81 were operated by CHAL, 34 were privately owned, and 4 were operated by the Lesotho Red Cross Society (LRCS). MOH routinely provides financial support for service provision implemented in CHAL health facilities. 8. A number of system-wide problems in the health sector contributed to Lesotho’s worsening outcomes. Among them were low utilization of health facilities, lack of equipment, a poor referral system between health centers and hospitals, and inadequate numbers of healthcare workers. The country had one of the worst ratios of health workers to population in sub-Saharan Africa with just over one health professional per 1,000 population. There were nine primary facilities and just one hospital per 100,000 people, with Quthing and Mohale’s Hoek districts having the lowest ratios of primary facilities to population. 9. Apart from nursing schools, no formal medical education system existed in the country. Most Basotho attended medical school outside the country, with few returning to practice in their home country. Irish Aid was working with MOH in the recruitment of nurses while the Millennium Challenge Account Lesotho (MCA-Lesotho) was working with the MOH in the retention of nurses through provision of staff houses across the country. The minimum nursing staff complement for a health center was one nursing officer (nurse clinician or nurse with advanced midwifery), one nursing sister (registered nurse with midwifery), and one nursing assistant. At the time of project appraisal, the vacancies for nursing officers, nursing sisters, and nursing assistants yet to be filled in health centers across the country were 33, 59 and 46 respectively. This reflected the country’s challenges with human resources for health (HRH) and the need to focus on maximizing the productivity and performance of existing healthcare workers through incentive-based compensation schemes. 10. Geographic and financial challenges to access were also prevalent in the country. About 40 percent of the population lived in remote rural villages, often needing to walk several hours through rough mountain paths to the nearest health facility. The 2009 Lesotho DHS reported that 73 percent of women cited at least one problem in accessing health care – respondents cited unavailability of drugs (59 percent), treatment costs (33 percent), transportation costs (32 percent) and long distances to facilities (31 percent) as problems for accessing health services. Page 8 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) 11. While Village Health Workers (VHWs) played a crucial role in improving the health of the Basotho, they were under-utilized in primary health care outreach and referral. They were previously unpaid volunteers. In 2008, GOL announced that VHWs would be paid a monthly flat rate of LSL 300 or approximately US$35. This compensation was not performance-based and had not yielded the intended results. 12. At project appraisal, Lesotho had some experience with output-based financing models for improved service delivery. Such experience included: (i) establishing a Public Private Partnership (PPP) hospital and filter clinics with a subsidy provided by the Global Partnership on Output-based Aid (GPOBA), (ii) introducing performance indicators in service contracts with CHAL and Lesotho Red Cross Society (LRCS), and (iii) small PBF projects supported by Partners in Health (PIH) to test the feasibility in the Lesotho rural community context. PIH projects include training and performance-based financing to maternal health workers to promote maternal and child health seeking behavior in the communities they serve. The outcomes thus far had been very encouraging. The Bank’s International Development Association (IDA) and the International Finance Corporation (IFC) supported a PPP initiative for the establishment of filter clinics (in April/May 2010) and the replacement of the old national referral hospital (Queen Elizabeth II) with the Queen ’Mamohato Memorial Hospital (QMMH) in October 2011. Disbursements were based on the achievement of performance targets; actual performance consistently exceeded its key targets for in-patient admission, outpatient visits and client satisfaction. Building on the positive experience with the hospital PPP project, IFC supported the GOL to negotiate another PPP to strengthen the Information and Communication Technology (ICT) and Health Management Information Systems (HMIS) and to improve health care waste management (HCWM) at the health center level. The government planned to use the lessons learned from this project to catalyze its own investments as well as those from other development partners (DPs) to support PBF more broadly across the health sector. 13. HIV/AIDS and Tuberculosis (an opportunistic infection associated with HIV/AIDS) were of public health concern in Lesotho. The World Bank supported HIV/AIDS work through the HIV/AIDS Technical Assistance Project (2009-2015). Many other development partners also prioritized HIV/AIDS and Tuberculosis (TB) at both national and sub-national levels. Quality of health services was an area supported by relatively fewer development partners and, therefore, had a relatively greater gap in investment by development partners. Rationale for World Bank’s Involvement 14. The project was a continuation of the Bank’s involvement in the health sector in Lesotho. The Bank supported the Government-led health sector reform through Health Sector Reform Project Phase I (2000 – 2005) and II (2005-2009) through an Adaptable Program Loan. The main reform areas were: health financing, human resources, district health services, decentralization, pharmaceuticals, monitoring and evaluation, infrastructure and partnerships. The Bank also played a catalytic role in supporting Lesotho’s response to the HIV and AIDS epidemic. There was an ongoing HIV and AIDS Technical Assistance Project (2009-2015) which was building the capacity of the government and civil society to address implementation gaps of the National HIV and AIDS Strategic Plan. 15. Two feasibility studies carried out in 2010 with resources from the World Bank-administered Health Results Innovation Trust Fund (HRITF) guided project design. The PBF feasibility studies (February and August 2010) identified multiple critical supply side challenges, including: (i) variable productivity by health workers, (ii) lack of qualified health personnel in many health facilities as a result of uncompetitive salaries particularly in remote areas, (iii) lack of autonomy for health facility staff, (iv) low quality of care, and (v) lack of drive for results in facilities. They also identified options for suitable PBF pilot design and potential implementation arrangements. Eight staff from central and district levels of the health system participated in an African regional Page 9 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) RBF workshop, organized by the Bank in Zambia (May 29- June 1, 2012). MOH Directors also undertook a PBF study tour to Rwanda and Zimbabwe between August 12-18, 2012. Development Partners’ Engagement 16. At the time of project appraisal, there were several partners that were supporting or had previously supported Lesotho in the area of maternal and newborn health. UNICEF, UNFPA, WHO, Elizabeth Glasier Pediatric AIDS Foundation (EGPAF), Millennium Challenge Corporation (MCC), John Hopkins Program for International Education in Gynecology and Obstetrics (Jhpiego), Clinton Health Access Initiative (CHAI), ICAP, Lesotho Network of AIDS Services Organizations (LENASO), and Human resources Alliance for Africa (HRAA) were partners that were supporting the country on a predominantly country-wide scale. The support was mostly to maternal and newborn health as it related to HIV/AIDS. Other partners who worked to support maternal and newborn health in a few districts included Lesotho Boston Health Alliance (LeBoHa), World Food Program (WFP), SolidarMed, and Mothers to Mothers (M2M). Theory of Change (Results Chain) 17. At both project appraisal and the 2016 restructuring (when the PDO was revised – see details below), the project did not explicitly outline its theory of change as this was not a Bank requirement at that time. Therefore, the ICR team had to extrapolate the theory of change at appraisal and at the 2016 restructuring from the project description and illustrated them in 18. 19. 20. Figure 1 and Figure 2 below. The theory of change’s assumptions in both cases, were that (a) the MOH and CHAL would ensure adequate staffing at health facilities to deliver health services and (b) MOH leadership in financial management and procurement would ensure timely fiduciary support for the project. Page 10 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Figure 1: Theory of Change at Appraisal Page 11 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) • Activity #1: Provision of PBF for village health workers, health centers and hospitals • Activity #2: Provision of PBF to district health management teams (DHMTs) • Activity #3: Implementation and supervision support • Activity #4: Capacity building for MOH and CHAL at central and district levels • Activity #5: Provision of Performance Purchasing Technical Assistance (PPTA) to MOH PBF unit and facilities Activities • Activity #6: Training of village health workers and health professionals including on EmONC and supply chain • Activity #7: Capacity building to improve monitoring and evaluation at central and district levels • #1 PBF provided to health workers, health centers and hospitals • #2 PBF provided to district health management teams • #3 Improved implementation and supervision of services • #4 Improved capacity of MOH and CHAL at central and district levels Outputs • #5 Improved performance purchasing capacity of MOH PBF unit and facilities • #6 Knowledge and skills of village health workers and health professionals improved, including in EmONC and supply chain • #7 Better knowledge and skills in monitoring and evaluation at central and district levels • Improved maternal and newborn service delivery at community, primary and secondary levels through PBF Intermediate • Trained health professionals and village health workers with improved monitoring and outcome evaluation capacity (IOs) • Improved utilization of maternal and newborn health services in selected districts Expected • Improved quality of maternal and newborn health services in selected districts project outcomes Long-term Reduction in mortality and morbidity outcome Project Development Objectives (PDOs) 21. The project development objective (PDO) was to improve the utilization and quality of maternal and newborn health (MNH) services in selected districts in Lesotho. Page 12 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Key Expected Outcomes and Outcome Indicators 22. Key expected outcomes and outcome indicators at project appraisal are indicated in Table 1 below. Table 1. Outcome and Outcome Indicators Outcome Outcome Indicator (* PDO level ⁰ Intermediate) PDO1: Pregnant women delivering in health facilities * PDO 2: Children 1 year old who received all basic vaccinations* PDO 3: Currently married women using modern contraceptive method * IR1: Pregnant women in a lowest wealth quintile delivering in health facilities⁰ IR2: Women with at least four antenatal care visits during pregnancy⁰ IR3: Births attended by skilled health personnel⁰ IR4: Mothers who received postnatal care within two days of childbirth⁰ IR5: Pregnant women receiving antenatal care from a health provider ⁰ Improve Utilization of maternal and newborn IR6: Children receiving pentavalent vaccine (diphtheria, tetanus, whooping health (MNH) services in selected districts in cough, hepatitis B and Haemophilus influenza type b) ⁰ Lesotho IR8: People receiving tuberculosis treatment in accordance to the WHO- recommended “Directly Observed treatment Short Course” (DOTS) ⁰ IR9: Pregnant women living with HIV who received ARV prophylaxis or complete course of ARV to reduce the risk of MTCT⁰ IR10: Children under 5 years whose weight and height are monitored regularly ⁰ IR11: Number of health facilities with PBF contract⁰ IR14: Health personnel receiving pre-service nurse anesthetists training⁰ IR15: Nurses receiving training on the MOH adopted drug supply management manual⁰ PDO 4: Average health facility quality of care score* IR7: Tuberculosis treatment success rate⁰ IR12: Health facilities reporting stock-out of tracer medicines and medical supplies at the time of the health facility quality of care assessment⁰ IR13: Health personnel receiving training in Advanced Midwifery and Improve quality of maternal and newborn health Neonatology⁰ (MNH)services in selected districts in Lesotho IR16: Hospital and DHMT pharmacists receiving ESAMI training courses IR17: Personnel receiving training in procurement and financial management⁰ IR18: Village health workers trained⁰ IR19: Monitoring and Evaluation officers and District Health Information Officers receiving formal M&E training⁰ Page 13 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Components 23. The project had the following two major components during project appraisal: 24. Component 1: Improving Maternal and Newborn Health (MNH) Service Delivery at Community, Primary and Secondary levels through PBF (US$13.7 million). This component was jointly financed by IDA (US$9.7 million) and the Health Results Innovation Trust Fund (US$4 million). The component objective was to improve MNH service delivery at health facility and community level through two sub-components, as detailed below. 25. Sub-component 1A: Delivery of MNH Services through PBF. This sub-component was to support the provision of quality MNH services as well as selected services in the Essential Services Package in communities, health centers and hospitals by providing PBF to VHWs, health centers, and hospitals. Health centers and VHWs were to be considered as one unit for financing in their respective catchment areas in order to strengthen their collaboration. Furthermore, PBF for VHWs was to be linked to the overall performance of the health centers to which they were mapped. The incentivized services to be delivered by health centers were called the Minimum Package of Activities (MPA) and those delivered by hospitals were called Complementary Package of Activities (CPA). Additionally, this sub-component was to provide PBF to District Health Management Teams (DHMTs) - which were to become part of the District Councils with the decentralization of health services. Based on supervision of health facilities using a quality checklist, DHMTs were to provide feedback to health facility staff, and submit quarterly overall reports to the District Council Secretary. PBF was to be introduced in three phases, beginning with two districts (phase 1), to be followed by phase 2 with four additional districts, and then ending with phase 3 with the final three project districts for a total of nine districts (Maseru district was not initially planned to be included, although it was included toward the end of the project). 26. Sub-component 1B: PBF Implementation and Supervision Support. This sub-component was to provide critical support for: (i) PBF implementation and supervision; (ii) capacity building of the MOH and CHAL at central and district levels, district and community councils; and, (iii) best practice documentation and sharing. The MOH had established a central PBF Unit to handle the day-to- day management of the MNH PBF Project. The PBF unit consisted of five full time MOH staff. Given that MOH and CHAL had limited experience with PBF, this component was to build both strategic and operational capacity at respective levels. During project appraisal the PBF experience was limited to the experience of ‘Partners for Health’ in selected districts. That experience had little involvement of central level MOH and CHAL. The project was to competitively recruit a performance purchasing technical assistance (PPTA) firm to provide technical assistance and build in-country capacity. The PPTA’s key functions were to: (i) provide technical and implementation support to the MOH PBF unit and other PBF implementing entities on managing performance-based contracts for the delivery of incentivized services; and (ii) verify delivery of the quantity and quality of services, prepare the invoices for performance-based financing, and assist health facilities with preparing their PBF business plans. The role of the PPTA was to gradually reduce as the implementing entities and facilities gained greater experience with implementation of PBF. 27. Component 2: Training of health professionals and VHWs and improving Monitoring and Evaluation (M&E) capacity (US$2.3 million). This component was to be solely financed by IDA and have two sub-components as shown below. Page 14 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) 28. Sub-component 2A: Training health professionals and Village Health Workers. This sub-component was to support an ongoing MOH program for training doctors, nurse anesthetists and midwives to achieve an acceptable standard of competency in the delivery of MNH services including Emergency Obstetric and Neonatal Care (EmONC). It included a 5-day training of health center nurses on the MOH adopted drug supply management manual. This would allow the health centers to improve their forecasting and order preparation and potentially reduce stock-outs of drugs and medical supplies at the health center level. Additionally, 18 hospital and DHMT pharmacists, one NDSO staff, and one MOH Pharmacy Directorate staff were to participate in the Eastern and Southern African Management Institute (ESAMI) training courses on: (i) overview of supply chain management and (ii) quantification of health commodities. Refresher training was also be provided to MOH financial management and procurement staff. Sub-component activities also included support for part-time training for 15-20 nurse midwives at a university in South Africa for Advanced University Diploma in Advanced Midwifery and Neonatology; the then ongoing MOH effort to provide pre- service training of nurse anesthetists; an EmONC assessment to inform the need for on-the-job training for nurse midwives and medical doctors providing obstetric services in districts; and the then ongoing VHW training on basic services such as family planning and referrals as well as postnatal period care of mothers and children and promotion of exclusive breastfeeding. VHWs were also to be supported to conduct community head count and periodically update the village health registers for more accurate health facility catchment area data. 29. Sub-component 2B: Improving M&E capacity. This sub-component was to support the strengthening of the Health Management Information System (HMIS) in all districts and build the capacity of M&E personnel at the central and district levels. Activities under this sub-component included: (i) improving the quality of health data by reviewing, updating and harmonizing data collection tools for strengthening the HMIS; (ii) printing, training, dissemination, and utilization of the updated data collection tools, HMIS registers, forms and reports at all health facilities over the project duration; (iii) enrolling District Health Information Officers (DHIO) and central MOH staff in a short course on M&E of health programs (for two central and 10 district personnel) as well as a two-year part-time Master of Public Health (MPH) degree program with an M&E or Biostatistics concentration. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets 30. The PDO was revised during the level-1 restructuring in November 2016 and the overall project development objective became to: i) increase utilization and improve the quality of primary health services in selected districts in Lesotho with a particular focus on maternal and child health, TB and HIV; ii) improve contract management of select PPPs; and iii) in the event of an Eligible Crisis or Emergency, to provide an effective and immediate response to said Eligible Crisis or Emergency. Page 15 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Revised PDO Indicators 31. Table 2 shows the revised PDO indicators and targets. It includes PDO indicators and targets during appraisal (original PDO indicators), during the November 2016 restructuring, and during the May 2018 restructuring. The table showing both the revised PDO and IR indicators during these restructurings can be found in ANNEX 2 of this document. ANNEX 2 provides the revisions for the entire RF. Revised Theory of Change Figure 2: Theory of Change at the 2016 restructuring • Activity #1. Provision of PBF to district health management teams (DHMTs) • Activity #2: Implementation and supervision support • Activity #3: Capacity building for MOH and CHAL at central and district levels • Activity #4: Provision of Performance Purchasing Technical Assistance (PPTA) to MOH PBF unit and facilities Activities • Activity #5: Training of health professionals including on EmONC, HIV, TB and supply chain (TB & HIV - new) • Activity #6: Capacity building to improve monitoring and evaluation at central and district levelsActivity • Activity #7 : Contract management of select PPPs (new activity introduced) • Activity # 8: Emergency preparedness for future eligible crisis or emergency (new activity introduced) • #1 PBF provided to health workers, health centers and hospitals • #2 PBF provided to district health management teams • #3 Improved implementation and supervision of services • #4 Improved capacity of MOH and CHAL at central and district levels Outputs • #5 Improved performance purchasing capacity of MOH PBF unit and facilities • #6 Knowledge and skills of health professionals improved, including in EmONC and supply chain • #7 Better knowledge and skills in monitoring and evaluation at central and district levels • #8 Improved contract management for PPPs • #9 Structures to respond to future crisis or emergency • Improved maternal, newborn, HIV, TB service delivery at community, primary and secondary levels through PBF • Trained health professionals and village health workers with improved monitoring and evaluation capacity Intermediate • Strengthened capacity for managing PPPs outcome (IOs) • Better preparedness for future eligible crisis or emergency • Improved utilization of primary health services in selected districts with a particular focus on maternal and child health, TB and HIV • Improved quality of primary health services in selected districts with a particular focus on maternal and child Expected project health, TB and HIV outcomes • Efficient management of resources through PPPs • Strengthened capacity to respond to crisis or emergency Long-term Reduction in mortality and morbidity outcome Page 16 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Table 2: PDO Indicators2 PDO Indicators at Appraisal November 2016 Restructuring Comments May 2018 Restructuring Comments PDO 1. Pregnant women PDO 1. Pregnant women delivering in Revised for clarity of Dropped because percentage of institutional delivering in health facilities (%) PBF enrolled health facilities in target language deliveries improved during project Baseline: 56.2%; Target: 64% districts (%) implementation and was relatively high and Baseline: 53.2% decision was to include a more challenging to Target: 80% attain indicator, e.g. related to family planning PDO 2. Children 1 year old who PDO 2. Children under 1-year fully Revised for clarity of PDO 2. Number of children under- Revised from percentage to numerical indicator, received all basic vaccinations (%) immunized in PBF enrolled health language 1 fully immunized at PBF enrolled and consistent with HNP Corporate Results Baseline: 62.1%; Target: 67% facilities in the target districts (%) facilities Indicator (CRI) Baseline: 60.1%; Target: 72% Baseline: 22,834; Target: 31,440 PDO 3. Currently married women PDO 3. Currently married women using Revised for clarity of PDO 1. Number of women using New indicator introduced to cover all women to using modern contraceptive modern contraceptive method in target language modern contraceptive method in replace former indicator on only married method (%) districts (%) PBF enrolled health facilities in women, and revised from percentage to Baseline: 42.6%; Target: 48% Baseline: 40.4%; Target: 62% target districts numerical indicator to use indicator being Baseline: 70,956; Target: 117,900 tracked using HMIS PDO 3. Number of underweight New indicator to address malnutrition children under 5 years detected (underweight) and use an indicator being and treated in the target districts tracked using HMIS Baseline: 1319; Target: 2050 PDO 4. People receiving tuberculosis Upgraded from IR PDO 4. Number of patients started New indicator to measure initiation of TB treatment to repl treatment in accordance with the WHO- level due to revised on TB treatment in the target former TB DOTS indicator and to use an indicator tracked u recommended “Directly Observed PDO districts HMIS Treatment Strategy” (DOTS) (Number) Baseline: 3725; Target: 4220 Baseline: 4925; Target: 9500 PDO 5. Pregnant women living with HIV Upgraded from IR PDO 5. Number of people New indicator introduced to measure all HIV who received ARV prophylaxis or level due to revised currently on HIV treatment in the patients on treatment to replace former MTCT complete course of ARV to reduce the PDO target districts indicator risk of MTCT in target districts (Number) Baseline: 128,037; Target: 178,300 Baseline: 3910; Target: 13,000 PDO 4. Average health facility PDO 6. Average Health Facility Quality of Revised for clarity of No change quality of care score (%) Care Score in target districts (%) language Baseline: 43.8%; Target: 50% Baseline: 59.6%; Target: 78% PDO 7. MOF Central PPP Unit and MOH New indicator to reflect PDO 7. MOF Central PPP Unit and Language revised to include “‘and fully staffed’” PPP Contract Management office inclusion of PPP Contract MOH PPP Contract Management established Management Support in office established and fully staffed PDO Baseline: 0%; Target: 100% Baseline: 0%; Target: 100% 2 Annex 1 includes additional information including baseline and target years Page 17 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Revised Components 32. During level-1 restructuring of the project in November 2016, the project components were revised as follows. 33. Component 1 (revised allocation US$ 11.45 million) was renamed to: "Improving Health Service Delivery through PBF." The PBF program under Component 1 was revised to: (i) suspend implementation of the village health worker (VHW) PBF program, given the MOH's decision to substantially change the VHW organizational set-up, as well as the challenges associated with the modalities of rewarding incentive payments to VHWs3; (ii) adjust the PBF program at the District Hospital level to focus more on the quality of services, and provide individual bonuses to hospital staff; and (iii) revise the quantitative incentivized indicators at the health center level by revising existing and including additional HIV and TB indicators. This new title of this component reflected the broader support provided under the PBF program. 34. Component 2 (revised allocation US$ 3.73 million) was renamed to: "Capacity Building Support to the Ministry of Health." The scope of activities under Component 2 was expanded to provide additional capacity building support to: (i) strengthen MOH procurement capacity and streamline procedures; (ii) better align the MOH Annual Joint Review with health sector strategic objectives, with a greater focus on program impact; and (iii) improve the integration of the QMMH network into the rest of the health system. Medical equipment was to be procured to improve MNH outcomes following the findings of the February 2016 EmONC report. This expanded scope of activities was reflected in the revised title of the component, and allocations to the component. 35. New Component 3 (allocation US$ 815,000) was added and named: "Enhance PPP Management Capacity within the Government of Lesotho." Activities under this new component comprised the establishment of the MOF Central PPP unit and MOH PPP Contract Management Office, the recruitment of the full PPP management staff complement, and the provision of technical assistance (TA) to strengthen oversight over the QMMH network PPP and other existing health PPPs, including PPP management capacity-building. 36. New Component 4 (allocation US$ zero) was added and named: ‘Contingent Emergency Response’. This component was added in the event of the potential need to support activities related to mitigating the impact of the El Niño induced drought in Lesotho or other emergencies. Other Changes 37. The name of the project was changed during the level-1 restructuring in November 2016 from the original name of ‘Lesotho Maternal and Newborn Health PBF’ to ‘Health Sector Performance Enhancement Project’. The geographic scope of the project changed twice during project implementation. During the level-1 restructuring of the project in November 2016, the geographic scope of the project was reduced from nine districts to six districts, given low implementation capacities. During the level-2 restructuring in May 2018, the geographic scope of the project was increased from six districts to all 10 districts in the country as implementation capacities had considerably improved and the Government, (especially the MOH) expressed interest in institutionalizing the PBF approach based on its generally positive results. Modifications to the results framework occurred during each of the two restructurings above. During the November 2016 restructuring, modifications were made to the results framework to reflect the broadened PDO and 3During the preparation phase, the intention was for the PBF model to build on what Partners in Health was doing. However, the absence of a village health worker (VHW) policy has resulted in a large cadre of VHWs with unstandardized profiles and tasks that tended to be based on priorities of donors. In addition, the project planned to train existing VHWs in the PBF enrolled districts based on a 1-week training curriculum. The Ministry of Health, however, eventually decided to draft a VHW policy in late 2018 which includes introducing a 6-week training curriculum for VHWs. The VHW policy was approved on November 27, 2019. Page 18 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) improve measurability and consistency of data sources. During the May 2018 restructuring, modifications were made in the results framework to accommodate the geographic scale-up and alignment with regular data sources generated by the project and the newly introduced electronic (was previously paper-based) health management information system (HMIS). Previous indicators were heavily dependent on infrequently collected household surveys. The results framework indicators were also revised to be consistent in wording with the indicators in the national health management information system. Rationale for Changes and Their Implication on the Original Theory of Change 38. The level I restructuring (November 2016) was to: (i) Revise the Project Development Objective (PDO) to reflect broadened scope of project activities focused on primary health care and Public Private Partnership (PPP) contract management capacity building support to the Government of Lesotho (GOL); (ii) reflect the reduction of the project’s original geographic scope from nine districts to six; (iii) revise both of the original project components and add two components; (iv) modify the Results Framework (RF) to include new and revised PDO and intermediary indicators to reflect the broadened PDO and improve measurability and consistency of data sources; (v) reallocate the original credit and grant proceeds across the project components and disbursement categories; (vi) revise the project implementation arrangements and governance structures. In particular, the project included the MOF as an implementing agency responsible for new component 3 while the MOH remained as the implementing agency for components 1, 2 and new component 4; and (vii) extend for a period of 24 months the original project closing date from June 30, 2017, to June 30, 2019. 39. The level 2 Restructuring (May 2018) was to: (i) Increase the project’s geographic scope from six districts to all ten districts as part of geographic scale-up; (ii) revise the RF to accommodate the geographic scale-up and align indicators and targets with regular data sources; and (iii) reallocate credit and grant proceeds across the project components and disbursement categories to respond to the geographic scale-up. 40. The theory of change derived during the preparation of the Implementation Completion Report (based on the description of the project during appraisal) was a focused theory of change that targeted maternal and newborn health services. The changes during the project restructurings broadened the theory of change to include child health (beyond newborn health), HIV, TB and PPP contract management. II. OUTCOME 41. This evaluation does not use a split rating methodology given the project’s expanded scope and increased targets. A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 42. The relevance of the PDO is rated High. Both the PDO at the beginning of the project and the revised PDO during the 2016 restructuring remained highly relevant. Across all phases of implementation, the PDOs (at beginning and at 2016 restructuring) remained relevant to both Lesotho and the World Bank due to several factors. 43. First, the project’s components contributed to and were aligned with the national priorities as outlined in the National Strategic Development Plans (NDSP I and II) which aim to “improve the quality of health; reduce maternal and child mortality; combat and prevent the spread and new infections of HIV and AIDS; and, reduce social vulnerability, Page 19 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) especially for children and old people,” as well as the National Health Sector Strategic Plan which aims to ensure equity and access to good quality health services at all levels of care. The project design also reflects the high priority accorded to human capital in the NDSP II 2018-23, which has as a strategic key pillar, the strengthening of human capital: health, education and skills development. Second, the project also continued to be consistent with World Bank Group priorities for the country, as outlined in the current FY16-FY20 Country Partnership Framework (CPF), specifically strategic objective 4 calling for “Improving Health Outcomes” within Focus Area I entitled “Improving Efficiency and Effectiveness of the Public Sector." Through “introducing a performance-based approach in primary health centers and district hospitals to improve health outcomes, including the HIV/AIDS response” the project directly responds to strategic objective 4 of Focus Area I of the CPF. Moreover, through PDO Part C (contract management for PPPs) and its associated Component 3, which promotes Public Private Partnership (PPP) Management Capacity within GOL, the project again directly responds to strategic objective 4 of Focus Area I, which calls for solutions to overcome the challenges associated with the MOH’s allocation of “most total health care spending (79 percent) to purchasing the services of various public and private providers.” This was a highly relevant activity as Lesotho was implementing a PPP jointly supported by the WB Group through IFC for Queen Mamohato Memorial hospital in Lesotho (the hospital accounts for approximately 30 percent of MOH budget). This PPP was at that time a flagship PPP in the Africa region and, therefore, important for the Lesotho health system and the Africa region of the World Bank. Finally, the project was and continued to be in line with the WB Group’s twin goals to reduce poverty and promote shared prosperity. In addition, the project was aligned with the WB’s Health, Nutrition, and Population Strategy by supporting clients to promote the equitable and efficient provision of public services, including strengthening health systems. B. ACHIEVEMENT OF PDOs (EFFICACY) Assessment of Achievement of Each Objective/Outcome 44. The PDO of the project is assessed against each of its three parts: (A) increase utilization of primary health services in selected districts in Lesotho with a particular focus on maternal and child health, TB and HIV; (B) improve the quality of primary health services in selected districts in Lesotho with a particular focus on maternal and child health, TB and HIV; and (C) improve contract management of select PPPs. The PDO “in the event of an Eligible Crisis or Emergency, to provide an effective and immediate response to said Eligible Crisis or Emergency” is not assessed because no such event occurred during project implementation. Table 3 below the analysis summarizes the combination of PDO-level and Intermediate Results indicators (IRIs) from the project RF to support the achievement of each of the PDO’s three parts. In addition, in order to supplement the data from the project RF (particularly at the household level) in support of the achievement of the PDO, the project contributed to the financing of the 2018 Multiple Indicator Cluster Survey (MICS). These recently available MICS data are utilized in the ICR to substantiate achievements under the PDO. The global MICS program was developed by UNICEF in the 1990s as an international multi-purpose household survey to support countries to collect internationally comparable data on key indicators that measure the well-being of women and children. The MICS data were therefore relevant for this project as they include the most recent (2018) information on the health status of women and children in the districts covered by the project.4 The project contributed to the results reported in the MICS data as the project supported PBF related interventions for women and children in the project districts, as well as capacity building activities for management and service delivery at central and district levels. MICS data were not yet available at the time of the project closing date and were therefore not reflected in the final project ISR. This partly explains the lower rating of the project in the final ISR than in the ICR as the MICS data show more positive outcomes of the project. In addition, the final ISR also used results framework project data up to December 2018, while the ICR used data up to the project closing date (June 2019), which demonstrated better project achievements than at the time of the last ISR. 4 There were no Lesotho DHS data as recent as 2018 Page 20 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) 45. PDO Part A (increase utilization of primary health services in selected districts in Lesotho with a particular focus on maternal and child health, TB and HIV) is rated Substantial. Majority of indicators were achieved while the performance of antenatal and maternal indicators has notably improved. This PDO is measured by five PDO-level indicators (PDOs 1-5) and one IRI (IRI 1), which contribute towards improving utilization of key primary health services, particularly maternal and child health, TB and HIV services, and are thereby closely linked to the achievement of this PDO. Two out of the five PDO indicators linked to this PDO (PDO 2 and PDO 5), and IRI 1 substantially surpassed or achieved their targets. On the other hand, two of the remaining three PDO level indicators did not achieve their targets (PDO 1 and PDO 3). The third remaining PDO indicator, (PDO 4) – measuring persons initiating TB treatment – while not achieving its target, reflects the decreasing TB incidence in Lesotho and can therefore be viewed as a positive outcome. The performance of the RF indicators linked to PDO Part A is also substantiated by data from the 2018 MICS which demonstrates significant improvements (since DHS 2014) in the utilization of several key primary health indicators such as modern contraception usage among married women, full immunization, antenatal care, institutional deliveries, etc., both at the national and district levels, and particularly in those districts enrolled the longest in the PBF program. These are discussed below. (a) PDO indicator 1 - Number of women using modern contraceptive method in PBF enrolled health facilities in target districts. This indicator serves to measure utilization of a key reproductive health-related primary health service and was an incentivized indicator under the HSPEP PBF program. This PDO indicator’s achievement rate is 43. 9 percent, attaining 91, 568 at endline of its final project target of 117,900 as of June 2019, from a 2014 baseline of 70,956. This indicator struggled to achieve its target owing to shortages of supplies such as family planning (FP) commodities due to supply chain management and budget reduction issues that were not within the control of PBF enrolled health facilities. The WB team had discussed the issue with the MOH and development partners who were directly supporting the Government on FP commodities. The issue was expected to be resolved before project closing but it took longer to be addressed. However, based on the results of the 2018 MICS survey, the prevalence of married women aged 15- 49 using modern contraception increased nationally from 59.8 percent (DHS 2014) to 64.6 percent (MICS 2018). It is useful to note the performance of this indicator at the district level particularly in those districts exposed longest to the PBF program. In Quthing district, the first to be enrolled in the PBF in May 2014 and with the poorest health indicators, performance of this contraception indicator improved from 63.6 percent (DHS 2014) to 67 percent (MICS 2018). In Leribe, the second enrolled district in PBF, the indicator remained relatively stable from 63.4 percent to 63.6 percent. The Phase II districts achieved more significant increases between 2014 DHS data and 2018 MICS, from 53.4 percent to 65 percent in Mohale’s Hoek, 48.4 percent to 63.3percent in Mokhotlong, 58.2 percent to 65.5 percent in Mafeteng, and finally, 56.4 percent to 62.5 percent in Thaba-Tseka. (b) PDO indicator 2 - Number of children under-1 fully immunized at PBF enrolled facilities. This indicator serves to measure utilization of a key child health-related primary health service and was incentivized under the PBF program. Despite the supply chain management and budget reduction issues that resulted in shortages of vaccines, this PDO indicator surpassed its endline project target of 31,440 as of June 2019, achieving 35,607 at endline from a 2014 baseline of 22,834, an achievement rate of 148.4 percent. While the 2018 MICS survey (68.8%) did not observe significant improvements since DHS 2014 (68.3%) in terms of full immunization coverage (basic antigens5), notable strides were made at the district level. In Quthing and Leribe, the first two districts to be enrolled in the PBF, full immunization coverage improved from 60.1 percent to 81.5 percent between 2014 and 2018 in Quthing, a historically underperforming district, and between 69.3 percent (2014) to 78.2 percent (2018) in Leribe. In the Phase II district of Mokhotlong, immunization coverage improved from 47.5 percent (2014) to 64.1 percent (2018). (c) PDO indicator 3 - Number of underweight children under 5 years detected and treated in the target districts. This indicator serves to measure utilization of another key child health-related primary health service and was incentivized 5 Basic antigens include: BCG, Polio3, DTP3, Measles1/Measles-Rubella1 Page 21 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) under the PBF program. This PDO indicator partially achieved (66%) its final project target of 2,050 as of June 2019, achieving 1,8066 as of December 20187 from a 2016 baseline of 1,319. The lower than expected performance of this indicator is attributed to budget issues due to the reduction in funding for village health workers by a key development partner. At the household level, while the MICS 2018 reflected a slight increase in stunting since DHS 2014 (from 33.2% to 34.5%) and small reductions in underweight (10.3% to 10.5%) and wasting (from 2.8% to 2.1%), at the district level, some notable strides were made. For example, in Leribe, one of the two phase I districts enrolled in the PBF in January 2015, the prevalence of underweight, stunting, and wasting decreased between 2014 and 2018 from 8 percent to 5 percent, 31.3 percent to 29.8 percent, and 3.3 percent to 1.7 percent, respectively. Similarly, in the phase II district of Mokhotlong, prevalence of underweight, stunting, and wasting declined from 15.8 percent to 10.1 percent, 47.7 percent to 43.3 percent, and from 3.6 percent to 2.2 percent respectively between 2014 and 2018. (d) PDO indicator 4 - Number of patients started on TB treatment in the target districts. This indicator serves to measure utilization of a key TB-related primary health service and was incentivized under the PBF program. This PDO indicator did not achieve its final project target of 4,220 as of June 2019, achieving 3,428 at endline, down from a 2014 baseline of 3,725. While this indicator did not meet its target, a downward trend in the number of patients initiated on TB treatment may be viewed as a positive outcome, reflecting reduced TB incidence from 852 new cases per 100,000 in 2015 to 611 new cases in 2018.8 In other words, the decline in TB incidence has affected the volume of patients initiating treatment, suggesting that the most relevant indicator would have been to monitor the incidence itself. The decline in TB incidence will be reflected in the Lesotho National TB Strategy that is currently being revised to adjust its targets in terms of new cases and their treatment. The results of this indicator are therefore consistent with the Lesotho National TB Strategy’s objective of reduced incidence of TB cases. (e) PDO indicator 5 - Number of people currently on HIV treatment in the target districts. This indicator serves to measure utilization of a key HIV-related primary health service and was incentivized under the PBF program. This PDO indicator surpassed its final project target of 178,3009 as of June 2019, achieving 213,233 at endline from a 2016 baseline of 128,037, an achievement rate of 169.5 percent. (f) Number of health facilities with PBF contracts. This IR indicator achieved its final project target of 171 as of June 2019, with all 171 facilities signing PBF contracts at project endline, an achievement rate of 100 percent. By signing PBF contracts, health facilities have been able to utilize PBF to undertake measures to enhance utilization of key primary health services, thereby contributing to this part of the PDO. (g) Antenatal care. Maternal health indicators related to provision of antenatal care to expectant mothers formed part of the original project results framework and were subsequently dropped during the two restructurings, but nevertheless remained incentivized as part of the PBF program. Overall, utilization of antenatal care services remains high in Lesotho, with 94.7 percent of women with a live birth in the last two years having at least one ANC visit (DHS 2014), which improved to 96.4 percent as of MICS 2018. Similarly, women with at least four ANC visits improved from 74.4 percent in 2014 to 76.6 percent in 2018. Most notably, the percentage of women undertaking their first ANC visit in the first trimester of pregnancy rose from 41.2 percent in 2014 to 57.1 percent in 2018. As a key incentivized indicator at the facility level in the districts benefiting from PBF, these improvements in the utilization of ANC services at the household level support the achievement of PDO Part A. 6 The target for PDO 3 was miscalculated in the final project ISR as 2,900, when the correct figure is 2,050 when all four batches are aggregated, which is corrected in this ICR. Similarly, the actual achievement was miscalculated as 1,968 in the final ISR but is in fact 1,806. 7 Project endline data for this indicator was not available at the time this ICR was drafted. 8 World Health Organization. 2018. Lesotho TB Country Profile. 9 This target was miscalculated in the final project ISR as 216,300, when in fact the correct figure is 178,300 when all four batches are aggregated, which is corrected in this ICR. Page 22 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) (h) Institutional Deliveries and deliveries attended by skilled personnel. The maternal health indicator related to pregnant women delivering in PBF enrolled facilities was originally part of the project RF and was subsequently dropped during the 2018 restructuring, although it nevertheless remained a key incentivized indicator under the PBF. Coverage of institutional deliveries and deliveries attended by a skilled attendant significantly improved overall in Lesotho, from 76.5 percent and 77.9 percent in 2014 to 89.4 percent and 86.6 percent respectively according to MICS 2018. Meanwhile at the district level strong performances were also observed in districts where GOL and CHAL facilities were enrolled for the longest in the PBF program. In Quthing, a historically underperforming district and first to enroll in PBF, coverage of institutional deliveries improved from 71.9 percent to 87.8 percent between 2014 and 2018, and deliveries attended by a skilled attendant increased from 72.8 percent to 89.1 percent. In the remote phase II district of Mokhotlong, strong improvements were also observed, with coverage of institutional deliveries improving from 60.8 percent to 88.6 percent between 2014 and 2018, and births attended by a skilled attendant also increasing from 62.8 percent to 87.8 percent over the same period. Similar positive trends in the coverage of institutional deliveries were observed in the remaining Phase II districts of Thaba-Tseka (68% to 80.9%), Mohale’s Hoek (74% to 91.2%), and Mafeteng (75.3% to 89.9%) between 2014 and 2018. As a key incentivized indicator at the facility level in the districts benefiting from PBF, these improvements in the utilization of facility-based delivery services at the household level support the achievement of PDO Part A. 46. In assessing performance of PDO Part A it is important to also consider that the final phase (phase 3) of project implementation was introduced in mid-2018, which was about a year before the project closing date in June 2019, in anticipation of AF and project implementation extension. Comparison of achievements of PDO utilization indicators before phase 3 of project implementation and after the implementation of phase 3 of the project, shows that the project’s PDO achievements in all five PDO utilization indicators were higher when phase 3 of the project was not included in the analysis. For PDO1 indicator (Number of women using modern contraceptive method in PBF enrolled health facilities in target districts), the achievement was 77.3 percent for the first two phases of the project alone, compared with 43.9 percent when phase 3 of the project is included). For PDO2 indicator (Number of children under-1 fully immunized at PBF enrolled facilities in target districts), achievement of the first two phases was 197.5 percent compared with 148.4 percent when phase 3 is included. A similar pattern is observed in project achievements for the remaining three PDO indicators (PDO3 indicator: 76.9 percent for first two phases compared with 66.6 percent when phase 3 is included; PDO4 indicator: -10.10 percent for first two phases compared with -60.0 percent when phase 3 is included; PDO 5 indicator: 268.7 percent for first two phases compared with 169.5 percent when phase 3 is included). ANNEX 3 of this report has the table that presents the above comparisons. The above analysis suggests that the overall achievements of the PDOs of the project would have been higher if phase 3 of the project had not been incorporated in anticipation of the AF that ultimately did not take place. Targets for phase 3 of the project had considered the inclusion of the four remaining districts which were generally considered to be relatively better performing districts in anticipation of a project extension with the anticipated AF which had been already discussed with the Government. 47. PDO Part B (improve the quality of primary health services in selected districts in Lesotho with a focus on maternal and child health, TB and HIV) is rated Substantial because the majority of indicators supporting this PDO were achieved and especially given the very high achievement of the one key PDO indicator measuring the quality of services (PDO6). This PDO 6 (Average Health Facility Quality of Care Score in target districts) is a composite score for different dimensions of quality of these essentially primary level services. This quality of care score is obtained using a health facility quality checklist covering quality domains of staff attendance, record-keeping and timeliness of reports, adherence to protocols and guidelines for child survival, environmental health, general consultations, reproductive health, essential drugs management, tracer drugs, maternal health, STI, HIV, tuberculosis, and community-based services. The high achievement of this PDO indicator therefore reflects high achievement of different dimensions of quality that are embedded within the indicator. Furthermore, throughout project implementation, the health facilities’ quality scores steadily improved. Because the quality of care scores in all district hospitals improved significantly, the MOH and the Page 23 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) University of Pretoria revised the Quality Checklists at the district hospital level in June 2018 to ensure that the high scores truly reflect the quality of care provided to patients. The revised checklists put greater emphasis on the evaluation of health workers’ clinical skills such as essential steps in management of obstetrics emergencies. Quality scores based on the revised quality checklists decreased initially but eventually improved again which was consistent with the positive trend recorded before the revision of the checklists. In ANNEX 7 (Figure and Figure 2), these trends are graphically presented. Also, beyond indicators covered in the RF, data from the PPP referral hospital confirmed that among maternal deaths recorded from referrals in PBF-enrolled district hospitals relative to non-PBF-enrolled hospitals, 63 percent and 88 percent respectively were caused directly by a maternity-related condition indicating a better-quality handling of these cases in PBF-enrolled hospitals. (a) Average Health Facility Quality of Care Score in target districts. This key PDO indicator serves to measure the overall quality of primary health services provided at the participating facilities in the target districts of the PBF program, and is therefore directly linked to the achievement of PDO Part B. This PDO indicator surpassed its final project target of 78 percent as of June 2019, achieving 81 percent at endline from a 2015 baseline of 59.6 percent. This is an achievement rate of 116.3 percent. Starting from a very low-quality rate baseline, improvement of this indicator is attributable to the range of on-the-job clinical trainings and coaching provided to enhance the quality of service provision, and the financial autonomy granted to facilities under the PBF program to expeditiously procure necessary inputs and temporary HR support for continuous service provision and quality improvement. (b) Number of health personnel in the target districts that received training focused on clinical services. This IR indicator surpassed its target of 465 as of June 2019, achieving 826 trained at project endline from a 2014 baseline of 0 an achievement rate of 177.6 percent. This clinical training indicator was particularly successful owing to the mentoring program conducted in all public hospitals of the country, and thereby strongly supports the achievement of PDO Part B. (c) Number of health personnel in the target districts that received non-clinical health systems-related training. This IR indicator did not achieve its target of 6500 as of June 2019, achieving 2266 trained at project endline from a 2014 baseline of 0 - an achievement rate of 34.9 percent. While substantial progress was made with respect to this indicator with 2266 trained, the training of non-clinical staff did not reach its target due to the delayed adoption of the Village Health Workers (VHW) policy which involved significant revisions to the training curriculum of these personnel, thereby preventing them from benefiting from planned training activities under the project. In view of these circumstances involving the delayed adoption of the VHW policy, the target of this indicator should have been revised downwards during another restructuring to better reflect the realistic expectations of achievements under this non- clinical training indicator and this part of the PDO. (d) Community-based satisfaction score for PBF enrolled facilities in the target districts. This IR indicator did not achieve its target score of 89 percent as of June 2019, achieving 78 percent as of December 201810, from a 2014 baseline score of 75 percent - an achievement rate of 21.4 percent. Community-based satisfaction scores had steadily improved to as high as 87.8 percent in June 2018, very close to its end target of 89 percent. However, it declined to 78 percent as of December 2018 due to the reported shortage in health commodities owing to budget and stock management issues at central level, and the reduction in village worker case management support following reduced funding from a key development partner, and issues beyond the control of PBF enrolled facilities. Also, short exposure of the four new Phase III districts to the PBF scheme did not allow enough time to influence perceived quality as measured at the community-level, therefore decreasing the average community-based satisfaction score. (e) Number of District Steering Committee meetings in target districts providing feedback and grievance redress mechanisms based on assessments to facilities and involve community representatives. This IR indicator surpassed 10 Project endline data for this indicator was not available at the time this ICR was drafted. Page 24 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) its target of 18 meetings as of June 2019, with 42 meetings held at project endline from a 2014 baseline of two meetings - an achievement rate of 250 percent. (f) Number of Quality checklists revised and streamlined. The project’s successful revision, i.e. scaling-up of the quality checklist (utilized during quality assessments) on all four targeted occasions over the implementation period meant that this IRI indicator achieved its target of four revisions as of the June 2019 project endline date from a 2014 baseline of 0, an achievement rate of 100 percent. 48. PDO Part C (Improve contract management of select PPPs) is rated Modest. This PDO is measured by one PDO- level indicator (PDO 7) and one intermediate results indicator (IRI 5). Lesotho was implementing a PPP jointly supported by the WB Group through IFC for Queen Mamohato Memorial hospital in Lesotho (the hospital accounts for approximately 30 percent of MOH budget). This PPP was at that time a flagship PPP in the Africa region and, therefore, important for the Lesotho health system and the WB’s Africa region. PDO indicator 7 did not achieve its target while IRI 5 linked to PDO part C achieved its target. The PDO and IRI indicators contribute toward building PPP contract management capacity within the GOL and are thereby linked to the achievement of PDO part C. Beyond Results Framework indicators, the project supported PPP training and certification of many officials from Ministries of Health, Finance, Development Planning, and Public Works. Officials of these Ministries also gained skills on PPP negotiations. The project also contributed to developing PPP legal and regulatory frameworks in Lesotho. Currently, the government has initiated a referral hospital network PPP agreement renegotiation with the private operators to improve fiscal predictability and value for money. a) MOF Central PPP Unit and MOH PPP Contract Management office established and fully staffed. This PDO indicator did not achieve its target of filling the required four positions in the MOF PPP Unit and MOH PPP Contract Management Office by project closing in June 2019, with only one position, that of the Clinical Officer being filled as of endline, an achievement rate of 25 percent. This was due to delays in approving the positions by the MOF and MOH, and protracted procurement procedures, i.e. failure to attract viable candidates and delays in the procurement decision-making process because of political reasons, once the positions were approved. Consultants for two of the remaining three positions (Legal and Finance Officers) were eventually identified within the second quarter of 2019, but these consultants could not be hired due to closure of the HSPEP and the cancellation of the Additional Financing (AF) meant to finance the positions. The AF cancellation was a decision made by the Country Management Unit to consolidate the portfolio, i.e. activities planned under the AF are being incorporated in the new Nutrition and Health System Strengthening Project that is currently being prepared. Subsequently, the Public Sector Modernization Project based at the MOF - which was recently restructured - assumed responsibility for procurement of these positions. Both the Legal Officer and Finance Officer positions have already been filled. These developments, after the closing of the project, have improved the development objectives of the project (achievement rate of the PDO indicator is 75 percent if post closure developments are considered), beyond what was reflected at project closing. b) Number of Terms of References for key PPP positions in the Government drafted and approved. This IR indicator was achieved, with all four TORs being drafted and approved as of the project endline of June 2019, an achievement rate of 100 percent. Page 25 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Table 3. Achievement of PDOs and Intermediate Indicators Outcome Outcome Indicator Baseline Achieved End Original Revised Targets Revised Targets Achievement (* PDO level; ⁰ Intermediate) (A) Line June Targets restructuring: May 2018 (B-A/C-A) 2019 (B) (at approval) Nov 2016 restructuring: (C) PDO Part A: PDO 1: Number of women using modern contraceptive 70,956 91,568 48% 62% 117,900 Not Achieved Increase method in PBF enrolled health facilities in target districts*11 (2014) (43.9%) utilization of PDO 2: Number of children under-1 fully immunized at PBF 22,834 35,607 67% 72% 31,440 Surpassed primary health enrolled facilities*11 (2014) (148.42%) services in PDO 3: Number of underweight children under 5 years 1319 1806 N/A N/A 2050 Partially Achieved selected districts detected and treated in the target districts* (2016) (Dec 2018) (66.6%) in Lesotho with a PDO 4: Number of patients started on TB treatment in the 3725 3428 N/A N/A 4220 Not Achieved particular focus target districts* (2014)** (-60.0%)12 on maternal and PDO 5: Number of people currently on HIV treatment in the 128,037 213,233 N/A N/A 178,300 Surpassed child health, TB target districts* (2016) (169.5%) and HIV; IRI 1: Number of health facilities with PBF contracts⁰ 0 (2013) 171 107 75 171 Achieved (100%) PDO 6: Average Health Facility Quality of Care Score in target 59.6 81.0 50 78 78 Surpassed PDO Part B: districts (%)* (2015) (116.3%) Improve the IRI 2: Number of health personnel in the target districts that 0 826 N/A N/A 465 Surpassed quality of received training focused on clinical services⁰ (2014) (177.6%) primary health IRI 3: Number of health personnel in the target districts that 0 2266 N/A N/A 6500 Not Achieved services in received non-clinical health systems-related training⁰ (2014) (34.9%) selected districts IRI 4: Community-based satisfaction score for PBF enrolled 75 78 N/A N/A 89 Not Achieved in Lesotho with a facilities in the target districts (%)⁰ (2014) (Dec 2018) (21.4%) particular focus IRI 6: Number of District Steering Committee meetings in 2 42 N/A N/A 18 Surpassed on maternal and target districts providing feedback and grievance redress (2014) (250.0%) child health, TB mechanisms based on assessments to facilities and involve and HIV community representatives⁰ IRI 7: Number of Quality checklists revised and streamlined⁰ 0 (2014) 4 N/A N/A 4 Achieved (100.0%) PDO Part C: PDO 7: MOF Central PPP Unit and MOH PPP Contract 0 25 N/A 100 100 Not Achieved Improve contract Management office established and fully staffed (%)* (2016) (25.0%) management of IRI 5: Number of Terms of References for key PPP positions in 0 4 N/A N/A 4 Achieved (100.0%) select PPPs the Government drafted and approved⁰ (2016) **2014 data were used for the TB treatment indicator because it was the year that had relatively complete information at the time of restructuring. Note: The ICR team found and corrected aggregation errors among end targets in the RF for TB treatment, HIV/AIDS treatment, and number of underweight children. 11 During the May 2018 restructuring: (i) the definition of PDO 1 was refined from “currently married women” to “women;” and (ii ) both PDO 1 and PDO 2 were redefined from percentage to numerical indicators 12 TB incidence decreased, resulting in reduction in number of new patients that needed treatment Page 26 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Justification of Overall Efficacy Rating 49. The achievement of PDO Part A is rated on a four-point scale as Substantial, PDO Part B is rated Substantial, and PDO Part C is rated Modest. Thus, the overall efficacy of the PDO equates to a rating of Substantial. In increasing utilization (PDO Part A), the project exceeded its outcome targets for full immunization of children under 1 year and for treating people with HIV; partially achieved its outcome targets for women using modern contraceptive method and for detecting and treating underweight children under 5 years; and showed no increase in new TB patient treatments, mainly due to continued decline in the TB incidence from 2015 to 2018– this is a positive outcome. Furthermore, data from the 2018 MICS demonstrate significant improvements (since DHS 2014) in the utilization of several key primary health indicators including services supported by the project. In improving the quality of primary health care services (PDO Part B), the project demonstrated continued progress and surpassed its outcome target measured by the composite Quality of Care Score indicator. The project made modest progress towards improving the contract management of select PPPs through the PPP Contract Management office (PDO Part C), although this improved after the closing of the project. During the project’s life, no eligible crisis or emergency occurred requiring response from the project . The project has made substantial strides in terms of utilization and quality with some modest shortcomings. C. EFFICIENCY Economic Analysis (Allocative Efficiency) 50. The economic analysis for the project shows a solid economic rationale for the investment. Overall, the project contributed to reducing underweight prevalence, maternal mortality, under-1 mortality, and prevalence of TB and HIV/AIDS, and its benefits justify the costs. The project did so by improving staff motivation, allowing for less disruption of services through facilitated local procurement of services and commodities that are not centrally procured, and investing in outreach services to improve service coverage. Furthermore, the various clinical and non-clinical training provided have improved service delivery capacities across supported districts. The net economic benefits generated by the project’s inputs and outputs resulted in a positive net present value (NPV) of US$48.4M, an internal rate of return (IRR) of 14 percent, and a cost-benefit ratio of 3.1, with a conservative approach in estimating the benefits from the project. These numbers are, lower than the PAD as some of the project’s indicators did not meet their original targets and the original investment was not completely disbursed over the cycle. In addition, the PAD and ICR estimates are not directly comparable because the project underwent two restructurings during implementation leading to different indicators, targets, scope, components and resource allocations across components. The differences in the IRR at ICR stage compared to the expected economic rate of return (ERR) as outlined in the PAD are also due to different methodologies used in the analysis. The PAD EA includes an input-output approach with different assumptions made compared to the further EAs performed during project implementation and for the ICR. 51. The analysis suggests that with an investment of $14.9M, which is the total amount disbursed (IDA and Trust Fund) over 5 years, 3,300 children’s lives would have been saved, 239 fewer children under-5 would be stunted, and 2,192 women’s lives would be saved due to receiving project’s services (Table 4). The investment financed the PBF incentives that were later allocated to motivate staff and procure goods and services for quality improvement at the facility level, as well as clinical and non-clinical trainings at central and decentralized levels, and operating costs. Investing in this project would result in an estimated US$71M in economic benefits over the lifetime of beneficiaries. Page 27 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Table 4.Impact of the Project Year 0 Year 1 Year 2 Year 3 Year 4 Year 5 TOTAL Interventions'period 2014 2015 2016 2017 2018 2019 Assumed disbursement 397,611 963,593 1,594,277 2,584,204 4,495,829 4,954,372 14,989,886 ALL # maternal deaths averted 320 343 360 373 383 413 2,192 # child (<1) deaths averted 432 492 537 571 596 672 3,300 # children underweight averted (<5) 32 36 39 41 43 48 239 # deaths averted (TB,HIV) 167 194 215 230 242 276 1,325 Total number of deaths averted 919 1,029 1,112 1,174 1,221 1,361 6,817 # children underweight averted (<5) 32 36 39 41 43 48 239 Source: calculated based on service utilization data and their relation to mortality /mortality averted 52. Sensitivity analyses showed that the results of the Cost benefit analysis (CBA) were sensitive to changes in the modelling assumptions, but the main conclusions remain unchanged, i.e. that the investment was justified on economic grounds (Table 5). A higher discount rate (from 5 to 10 percent) would reduce the economic return of the project’s investment, yet it would remain economically sound. Likewise, factoring the double counting factor (please see assumption box in Annex 5 for more information) also yields 12 percent IRR with US$ 34M NPV, thus providing a solid economic rationale for the proposed investment. Table 5. Sensitivity Analysis BCR IRR NPV Base-case scenario (5% discount rate) 3.1 14% 48 M Sensitivity analysis Double counting (80%), same discount rate 2.5 12% 34 M Discount rate (10%) 1.5 14% 8M Double counting (80%), Discount rate (10%) 1.2 12% 3 M 53. The CBA is based on the PDO and its indicators while recognizing the greater complexity of the underlying causal chain of the interventions on outcomes and economic benefits. The project’s effectiveness/benefits were estimated separately for each project component and then transformed into monetary value in aggregated form. Operational Efficiency 54. The project experienced effectiveness and implementation delays. First, time between approval and effectiveness was about 14 months. The reason for delayed effectiveness was primarily due to delays in the recruitment of the Performance Purchasing Technical Assistance Firm (PPTA) that would support the MOH PBF Unit in its oversight function of the HSPEP and provide the necessary capacity transfer to ultimately enable the PBF Unit to fully take over project implementation. This resulted in a seven-month extension of the original effectiveness deadline from July 2013 to February 2014. The time between effectiveness and the first disbursement (May 2014), however, was three months, consistent with the average across the Africa region. Second, after project effectiveness, the pace of implementation was delayed for several months due to the project’s financial management and M&E arrangements, and the requisite technical assistance and capacity-building initiatives within the MOH PBF Unit which would serve as the project implementing unit, given the task team’s decision to utilize existing government institutions for project implementation Page 28 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) to the greatest extent possible. Financial management and funds flow capacity constraints were eventually resolved, with the MOH Project Accounting Unit (PAU) taking over FM responsibilities for the project, and with health facilities opening Health Center Committee administered bank accounts. The PBF Unit also recruited additional staff members, including a Senior PBF Officer and PBF Officer, in order to resolve staffing constraints. The Government’s decision to pilot PBF in Quthing - which had among the lowest socioeconomic indicators and the high turnover of senior government officials also contributed to implementation delays. 55. The project underwent a level I Restructuring (November 2016) and a level II restructuring (May 2018) during its approximately five and a half years of implementation since effectiveness. These restructurings were necessary to ensure that the project made the required course adjustments to achieve its overall objectives. These included adjusting the PBF program’s geographic scope, extending the project’s closing date to allow for enough time to implement the PBF in pilot, phase II and III districts following initial implementation bottlenecks, and responding to the GOL’s emerging need for PPP Management capacity building support in the health sector. 56. Despite some implementation delays, particularly regarding Component 3 and the PBF program, disbursements progressed, and ultimately almost all the funds were disbursed. Assessment of Efficiency and Rating 57. Considering the above efficiency considerations, the overall efficiency of the project is rated Modest. D. JUSTIFICATION OF OVERALL OUTCOME RATING 58. Based on High Relevance, Substantial Efficacy and Modest Efficiency, the overall outcome rating (according to the 2018 ICR Guidelines) is Moderately Satisfactory. The project’s relevance is considered High as the project development objectives remain well aligned with Lesotho’s national priorities and the WB Group’s strategic priorities. Efficacy is considered Substantial on balance noting the significant improvements in utilization and quality of high priority services, as well as some modest shortcomings. Efficiency is considered Modest based on implementation delays which are usually encountered when new approaches such as the PBF are introduced and institutional capacity building needs are high within a context of high turnover of senior government officials. Project shortcomings are being addressed as the Government continues the project’s performance-oriented approach using its own funds. In addition, the PBF approach will be fine-tuned with the support of the Nutrition and Health System Strengthening Project/NHSSP (P170278) that is under preparation. E. OTHER OUTCOMES AND IMPACTS (IF ANY) Gender 59. Women of child-bearing age, pregnant women, newborns and young children under five (including girls) were among the primary target beneficiaries. Although the project did not specifically focus on raising awareness of relevant gaps between males and females and on contributing to increase women’s assets, income, or employment opportunities, it did contribute to the improvement of reproductive, maternal, newborn and child health. The project targeted women for contraceptive services, ante-natal and maternity services and in so doing, promoted women’s health before pregnancy, during pregnancy, and during child-birth. Through the PBF provided by the project, health Page 29 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) facilities were able to purchase necessary maternity equipment and supplies in addition to hiring more midwives to fill in vacant needed positions for provision of quality health services to women. Institutional Strengthening 60. The project supported several institutional strengthening activities at national, district and facility levels. For example, at national level, the PBF-Unit at the MOH was supported by the project to manage day-to-day implementation, monitoring and evaluation, and management of PBF activities at the MOH. The Unit was staffed by mostly MOH personnel, ensuring the building of MOH institutional capacity in the area of PBF. The project also supported the National Sexual and Reproductive Health Steering Committee (NSRHSC) which provided national-level policy guidance and oversight of the project activities. The functioning of this committee was important as it had representation from the key directorates of the MOH, ensuring buy-in and coordination of the directorates in implementation of project activities. The project contracted a Performance Purchasing Technical Assistance (PPTA) organization with PBF implementation experience which built local capacity on strategic purchasing. Once local capacity was built particularly within the MOH, the firm was phased out. At district level, the project supported capacity building of the district steering committees which included the district health management teams, district councils, CHAL representatives and civil society representatives. These committees supported oversight and supervision of the delivery of PBF services at district levels. This capacity at the district level will continue to be useful for implementation of health services beyond the life of the project. Ultimately, the project has introduced a relatively new scheme in the health financing landscape in Lesotho: the PBF. The MOH has created a unit dedicated to this scheme. Training provided on PPP contract management was meant to improve the daily management of the PPP hospital in Maseru; while it could not fully achieve its potential, it certainly had some impact, for example, it contributed to improving the link between clinical outcomes and payments and the Government initiated a renegotiation process with the private operator to obtain more value for money and make the contract more affordable. Clinical-related trainings and mentoring programs have improved the capacity of nurses, doctors and midwives to deal with maternal, newborn and child health, and handle complex HIV/AIDS and TB cases. Mobilizing Private Sector Financing 61. Mobilizing private sector financing was not one of the areas of focus of the project. The project was therefore financed by IDA funding, Trust Fund financing and counterpart funding from the government. No private sector financing was involved in the financing of the project. Poverty Reduction and Shared Prosperity 62. The project was designed and implemented in such a way as to ensure that relatively poorer districts are not neglected during the implementation of the project. For example, during the pilot phase of the project (phase 1 at the beginning of the project when two districts were selected for piloting), Quthing district, which was one of the poorest districts and known for having the lowest health indicators among Lesotho’s districts, was selected to be part of the pilot together with the district of Leribe (which, while not among the poorest, had a large population). Village health workers (VHWs) who were supported by the project, were reported by key-informants, to have been particularly effective in mobilizing poor communities in remote areas which are not easily accessed by regular health workers. This contributed to the increase in access to health services by mothers, newborns and children from these poorer communities. Page 30 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Other Unintended Outcomes and Impacts 63. There were no reported unintended outcomes and impacts of the project. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 64. During the preparation of the project, the PDOs were realistic as they focused on a distinct target area for intervention (maternal and newborn health services) and, particularly on utilization and quality of these specific health services. These areas of focus needed support at the time of project preparation and the project was responding to a realistic and Government prioritized need. 65. The project was designed to start as a pilot in two districts (Leribe and Quthing) during the first year of the project (phase 1), then scale up to phase 2 with four additional districts, before moving to phase 3 with the final three districts of the project for a total of nine districts. The project was phased to ensure that lessons are learned in the initial districts that could then be applied during the scale-up to other districts. 66. At appraisal, the project had a results framework and arrangements for monitoring and evaluation which reflected the PDOs during project preparation. However, the MOH had not yet developed its own monitoring and evaluation plan/ framework which would be in accordance with their new national health policy. There were also challenges in M&E capacity at central and district levels, particularly regarding the human resources for M&E. 67. The project had clearly identified beneficiaries in the districts selected for the three phases of project implementation. Maternal and newborn health was a clearly identified area for the beneficiaries in those districts. 68. The project incorporated lessons learnt on PBF in the Africa region and other regions of the world including lessons from Rwanda (senior government officials went on a study-tour to Rwanda and Zimbabwe during project preparation), Cambodia and Afghanistan. The project was designed to start with a pilot in two districts, which was based on lessons learned in other countries that initiated PBF in pilot areas before scaling-up. Other lessons reflected during project preparation included ensuring the following: transparency and independent verification; individual health facilities were recipients of the PBF proceeds; clear processes for approving PBF proceeds; use of a manual of procedures; health workers and supervisors were trained on PBF, and measurement of results. 69. The overall project implementation risk at appraisal was rated as ‘High’. This reflected the anticipated challenges that the project team expected to be encountered during project implementation, particularly due to the PBF approach. The project team designed mitigation measures which included, among others, gradual scale-up, technical assistance and training, including PBF training courses targeting senior decision makers from the MOF and MOH. With these mitigation measures, the project team had reason to believe that the project was ready for implementation. Page 31 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) B. KEY FACTORS DURING IMPLEMENTATION Factors subject to the control of government and/or implementing entities 70. At the beginning of project implementation, commitment to PBF was not universal among government officials as this was a relatively less well-known instrument, particularly at the district and facility levels. In addition, both recruitment of the PPTA organization which was to support capacity-building in PBF and staffing of the MOH PBF unit took a long time, contributing to the slow start of project implementation. 71. The financial management of the project at country-level was planned to be led by an experienced team that understood government financial management procedures. However, during the beginning of the implementation of the project, there were human resources changes which led to weak capacity in financial management of the project at country level. This contributed to the initial slow implementation of the project. 72. Coaching and mentoring including case simulations/vignettes, which were provided by the project to complement regular trainings, contributed to the improvement of quality scores at health facilities. The contribution of these interventions was acknowledged by key stakeholders including facility staff, government staff and Bank project team members. 73. The procurement process for goods and services for the project had delays, particularly during the bid evaluation stage of the procurement process. Some of these delays were due to inadequate specifications for medical equipment which slowed down the procurement process. In other cases, the bid evaluation teams were not constituted on time or took longer than necessary to decide on the bids. These challenges led to delays in procurement or even inconclusive evaluation processes. 74. The project was implemented under a relatively complex political climate in the country, which frequently led to changes in leadership at the MOH at short notice. Transitions in MOH leadership contributed to delays in key decisions that affected the smooth running of the project. 75. Two legal covenants relating to the recruitment of a procurement manager and two procurement officers were dropped during the restructuring of 2016 as they did not reflect the setup of the MOH procurement unit. 76. After the project closed, the MOH continued to implement the PBF approach by paying for performance from counterpart funding that had been mobilized for the additional financing of the project, even though the additional financing did not happen, as planned activities were to be incorporated in the follow-on health sector project (Nutrition and Health Systems Strengthening Project/NHSSP). This is an indication of political will by the government to continue PBF in the country beyond the project’s life-span. The upcoming NHSSP is designed to finance the provision of quality and bonus grants to eligible health facilities, by using the government system and is based on the lessons learned under this project. The government, through the leadership of the Ministry of Development Planning, has expressed interest in implementing similar approaches in other sectors as well. The above indicates a political will in the country for sustainability of the benefits of the project. 77. The PBF scheme has attracted two other donors who contributed to funding specific indicators: UNICEF supported child and health nutrition and UNFPA financed family planning. The project also collaborated with these partners in a quality improvement program implemented by the University of Pretoria. Toward project closing, the dialogue on PBF institutionalization also included the US Government-sponsored Millennium Challenge Compact II preparation. Page 32 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Factors subject to the control of the World Bank 78. The World Bank supervision team provided regular supervision for the project, which enabled the team to identify the need for and implement two restructurings of the project and produce a total of 12 ISRs for the project. 79. Additional financing (AF) for the project had been planned and the project team was advanced in the preparation of the AF. However, Management decided to drop the AF to align with the WB Africa Region’s move toward portfolio consolidation and let the project close and instead move ahead with the preparation of a new nutrition and health systems strengthening project that would include activities that were planned under the AF. The last four districts would not have been included in the PBF program and adjusted project targets accordingly had the AF not been planned and preparation discussions initiated with Government. Factors outside the control of government and /or implementing entities 80. There was no conflict or instability or natural disasters during the implementation period of the project. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 81. At the project design stage, a theory of change was not explicitly articulated as during that time it was not required practice in the Bank to explicitly articulate a theory of change. During the 2018 restructuring, the theory of change was discussed by the project team, but it was not required to be included in the restructuring documents. Nevertheless, the results framework was clearly defined with key PDOs, PDO indicators, IRI indicators, baseline figures and targets, sources of data and responsibility for collection of the data. A few indicators were designed to be collected from surveys such as the DHS while others were to be collected from routine HMIS data. Institutional responsibility for collection of the indicators was spread between the relevant departs of the MOH. The results framework continued to be refined in subsequent restructurings to improve measurability and consistency of data sources. For these reasons, M&E design is rated Modest. M&E Implementation 82. The task team conducted extensive discussions with the MOH on M&E implementation, particularly during the revisions to the M&E framework during the two restructurings of the project. At first, indicators were dependent on national surveys especially the Demographic Health Survey which only took place every 5 to 6 years. As a result, indicators were changed to more regularly collected indicators and to be consistent with the newly implemented electronic format (web-based) health management information system in Lesotho. Baseline data, and data for monitoring the indicators in the results framework were routinely collected. M&E data were collected through several channels to monitor both PDO and intermediate indicators. These included routine data from the HMIS and data from DHS. There was a need during project implementation to rely more on routine data, which were reported more frequently, rather than relying on DHS data which could only be collected every 5 to 6 years. The project, in order to get other population data, contributed financially to the collection of the Multiple Indicator Cluster Survey (MICS) data, Page 33 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) which has been used in this ICR report. The PBF unit included a person who had responsibility for supporting M&E. Additionally, capacity building for M&E was provided by the World Bank team through supervision missions that included an M&E Specialist. The Senior Health Specialist based at the World Bank Office in the country (who was the co- Task Team Leader), took a hands-on approach to capacity building for M&E of the PBF unit on a routine basis. Capacity for M&E reporting was inconsistent at the beginning of project implementation, but it improved towards the end of project implementation. M&E implementation is rated Modest. M&E Utilization 83. Results provided by the M&E system were used during the entire implementation period to inform project management and decision making, particularly due to the data-intensive nature of PBF Results were compared to targets and the project’s targets were fine-tuned and adjusted during the two project restructurings. Policy making and advocacy at all levels of the project were informed by these results. Routine data that were collected by the M&E system contributed to the ratings and determination of payments of PBF funds to the implementing entities. While there is still room to improve M&E utilization capacity in the country such utilization of M&E data for PBF payments during project implementation was a notable development. M&E utilization is rated Substantial. Justification of Overall Rating of Quality of M&E 84. The overall quality of M&E is rated Modest. This is based on the Modest rating for M&E design, Modest rating for M&E implementation and Substantial rating for M&E Utilization. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE Environmental Compliance 85. The safeguard policy O/BP 4.01 pertaining to biomedical waste management was triggered. During project preparation, the project was classified as Category B - Partial Assessment as it triggered OP/BP 4.01 for environmental assessment. This was because of the anticipated increase in health care waste generated by the health facilities. No major works were financed by the project. However, health centers and hospitals could use PBF funds for small repairs of existing healthcare structures. In such cases, national and local laws were followed. To respond to OP/BP 4.01, the MOH implemented the Consolidated Lesotho National Health Care Waste Management Plan (CLNHCWMP) which was prepared and adopted in 2010 and updated in 2012 for the purposes of implementation of this project. Quarterly quality assessments were carried out for health centers and hospitals to monitor their compliance with national environmental and healthcare waste management regulations and guidelines. Based on these quarterly assessments, performance on environment health in health centers located in the initial six districts of the project improved between 2014 and 2018 during project implementation. A quality check-list for health centers helped the health centers to identify gaps and then use PBF funds to address those gaps. For example, health centers used PBF funds to purchase pedal waste bins, protective gear for personal protection when handling waste, and to engage cleaning companies. Social Compliance 86. The project did not trigger any applicable social safeguards issues. It was expected that the project would have positive social impacts as the project was supporting the improvement in utilization and quality of maternal and newborn health services in the selected districts of the country. Page 34 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Financial Management Compliance 87. At project appraisal, the World Bank conducted a Financial Management Assessment of the MOH and CHAL who were the planned implementing partners for the project. This assessment was triggered by the World Bank’s policy on Financial Management, OP/BP 10.02, and complied with the Financial Management Manual for World Bank-Financed Investment Operations that became effective on March 1, 2010 and Africa Region Financial Management (AFTME) Financial Management Assessment and Risk Rating. The conclusion of the assessment was that the financial management arrangements met the Bank’s minimum requirements under OP/BP10.02. The overall residual risk rating for MOH was Moderate. During project implementation, independent external audits of project financial statements, in accordance with the International Standards on Auditing, were carried out regularly. During project implementation, there was one reported qualified external audit of the project, while the rest of the external audits were unqualified. The single qualified audit was due to low financial management capacity at the country-level at the time which led to lack of reconciliation of accounts. The issue was resolved with the subsequent strengthening of financial management capacity at country- level. At the date of project closure, the project had a disbursement rate of 93.7 percent (combined IDA and Trust Fund). Procurement Compliance 88. At project appraisal, procurement to be financed under the project was to be carried out in accordance with the World Bank’s “Guidelines: Procurement under IBRD Loans and IDA Credits” dated January 2011, and “Guidelines: Selection and Employment of Consultants by World Bank Borrowers” dated January 2011, and the provisions stipulated in the Legal Agreement. During appraisal, the overall country context risk for procurement was rated ‘High’. The key project procurement issues were: (a) the need for MOH to fully staff the Procurement Unit; (b) limited capacity for new staff at MOH and existing staff at Leribe, Quthing and CHAL Secretariat to assure adherence to World Bank Procurement and Consultant Selection Guidelines; and (c) the potential risk of erroneously using Government of Lesotho or CHAL procurement procedures for Bank financed activities. Measures to mitigate the overall risks were: (a) MOH to fully staff the Procurement Unit; (b) training of key MOH staff on World Bank Procurement; (c) strengthening of procurement systems at participating District Councils, DHMTs, District Hospitals and CHAL Secretariat; (d) selected contracts to be subject to prior review; and (e) MOH preparation of a Procurement Manual with clear roles and responsibilities. During project implementation there were several delays in procurement of goods and services for the project. One of the key causes of delays in the procurement of goods was inadequate capacity to write quality product specifications, particularly for medical equipment to be procured by the project. Another common cause for procurement delays was the slow pace in forming bid evaluation teams and the slow functioning of these bid evaluation teams. C. BANK PERFORMANCE Quality at Entry 89. Quality at entry is rated Moderately Satisfactory. The World Bank team adequately assessed the risk for implementation of the project as ‘High’. The team also designed mitigation measures, which at the time seemed to consider the local situation and the level of capacity and experience that existed in the country. However, the mitigation measures proved insufficient during project implementation Experience from other countries in Africa and outside Africa was included in the design of the project. A study tour was conducted for senior government officials to Rwanda and Zimbabwe to see and learn how these countries were implementing PBF. A results framework for the project was designed which had clear PDOs at the beginning of the project, PDO indicators, and Intermediate results indicators. However, the results framework depended on indicators which were based on household surveys which were not Page 35 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) periodically collected, and which made regular project monitoring a challenge. In addition, project design (institutional arrangements) could have been further simplified based on existing institutional capacity especially the Government’s financial management systems and staffing. Quality of Supervision 90. Quality of supervision by the World Bank is rated Satisfactory. In addition to having a health specialist based in Lesotho to provide regular implementation support, supervision missions were carried out regularly, at the frequency of about once every six months. Fiduciary and other subject matter technical specialists were part of the supervision missions to ensure that relevant areas of supervision were adequately supported. The supervision team used information from the Implementation Status and Results Reports (ISRs)—a total of 12 ISRs were produced—to support the implementation of the project as demonstrated by the team’s conduct of two restructurings of the project. The team was proactive and modified the Results Framework whenever needed. Task Team leadership turnover was minimal with only two Task Team Leaders for the entire life of the project including the preparation of the project. Furthermore, the World Bank team was flexible in supporting the extension of the closing dates of the project to ensure that project activities were carried out as much as possible to achieve the PDOs. Justification of Overall Rating of Bank Performance 91. At entry, the World Bank team realistically identified the risks that the project would have with an overall rating of ‘High’. However, project design could have been further simplified by using regularly collected indicators in lieu of periodically undertaken surveys, and by taking into account the Government’s financial management capacity. During supervision, in addition to having a Senior Health Specialist based in Lesotho, the World Bank team carried out regular supervision missions, that included both fiduciary and subject matter technical specialists. The supervision teams used the information from the ISRs, to flexibly respond to the issues identified, including the implementation of two project restructurings and modification of the Results Framework indicators. The World Bank’s overall performance is therefore rated ‘Moderately Satisfactory’. D. RISK TO DEVELOPMENT OUTCOME Sustainability Risk 92. The project made significant effort and successfully turned PBF from an abstract concept at the beginning of the project to a popular and well accepted approach at the end of the project, whose benefits are appreciated by stakeholders. Key government officials at national, district and facility levels have expressed during key informant interviews, commitment to supporting future PBF-like interventions in the health sector and potentially in other sectors as well. During the third year of project implementation, both UNICEF and UNFPA signed an MOU with the MOH to support the PBF scheme, demonstrating support for the project’s activities. However, there is a risk that this commitment may not be sustained because PBF is not yet institutionalized in government accounting systems. Budget items for PBF- like interventions are not yet clearly defined in government budgets. This risk needs to be mitigated by continuous dialogue with senior government officials who can lead the institutionalization of PBF budgeting and implementation in the public sector, as well as additional support through the new nutrition and health project. Page 36 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Human Resources Risk 93. The project has been able to support the training of human resources at various levels of the health system, these staff now understand how PBF can be successfully implemented. There is, however, a risk that some key people who have been trained may leave the health sector or the country due to labor mobility in the Southern Africa region, that could weaken the human resources capacity in PBF in the country that has been built with the support of the project. V. LESSONS AND RECOMMENDATIONS 94. Prioritize staffing for early start-up of a PBF project. During the implementation of the project there were delays in getting the project up to speed due to low or no staffing in key areas of project implementation. The staff of the PPTA organization, the staff of the PBF-Unit of MOH and quality staff for financial management took a long time to be recruited and contributed to the slow pace of starting the project. Lesson: Early staffing of key positions is essential for quick start- up. Recommendation: Any future similar project should therefore prioritize staffing to ensure early project start-up. 95. Starting PBF projects with pilots continues to be good practice. The project started in two pilot districts (Leribe and Quthing) out of the total of 10 districts in the country. The experience of these pilot districts allowed for better implementation and scale-up in the subsequent districts. This lesson of starting PBF on a pilot scale is consistent with other previous experiences of implementing PBF projects and was further demonstrated in this project. Considering the challenges that the project overcame in these two districts during the pilot phase, it is unthinkable what the magnitude of those challenges would have been if the project had started in more districts. Lesson: Starting PBF projects with pilots continues to be a good practice. Recommendation: Any future PBF projects should be started on a small scale as pilots, before scaling up. 96. Build capacity for writing quality medical equipment specifications. One of the reasons for delay in procurement of needed medical equipment was the lack of capacity at MOH to write quality medical equipment specifications for the procurement process. Writing quality medical equipment specifications is an essential component of the procurement process for medical equipment. Lesson: There is need to address the gap in capacity for writing medical equipment specifications at MOH. Recommendation: Any future project that plans to procure medical equipment in the country should invest in building capacity for writing quality medical equipment specifications. 97. Institutionalize PBF in government budgeting to ensure sustainability. There is a risk that the benefits of PBF-like interventions in the health sector may not be sustained because PBF is not yet institutionalized in government budgeting and accounting systems. Budget items for PBF-like interventions are not yet defined in government budgets. However, the Government is continuing, based on the experience from this project, to implement a revised PBF scheme while working towards its institutionalization. The proposed new PBF approach (in the upcoming Lesotho nutrition and health systems strengthening project) is based on lessons learnt from the previous PBF model and builds on the capacity built within the MOH. The new model will focus on quality improvement and aim to maximize the use of in-built government systems for verification, M&E and payment. Lesson: The absence of PBF budget items in MOH and other sector budgets is a risk for sustainability of the benefits of PBF-like interventions. Recommendation: Institutionalize PBF in overall government budgeting to ensure sustainability of the benefits of PBF-like interventions. It would be important to look at existing overall government laws and processes that may need to be revised to facilitate institutionalization of PBF. This would need to be done concurrently with systematic sensitization, across all sectors, of government staff to ensure that turnover of staff does not erode institutional memory on PBF knowledge and practices. This report should also be widely shared among government sectors to enhance learning about PBF. Page 37 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) 98. Improve health system while introducing PBF. During the implementation of the project, essential health system components such as improvements in supply chain management and human resources for health, were critical to the success of PBF. Lesson: PBF works well when health system components are strengthened. Recommendation: Ensure . health system strengthening in order to facilitate successful implementation of PBF-like interventions. Page 38 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Increase utilization and improve the quality of primary health services (MCH, HIV, TB) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion PDO 1: Number of women Number 70956.00 117900.00 91568.00 using modern contraceptive method in PBF enrolled 31-Dec-2014 30-Jun-2019 30-Jun-2019 health facilities in target districts PDO 1a: Number of women Number 2648.00 2900.00 2776.00 using modern contraceptive method in PBF enrolled 31-Dec-2014 30-Jun-2019 30-Jun-2019 health facilities in target districts - first batch (Quthing) PDO 1b: Number of women Number 13757.00 18000.00 16413.00 using modern contraceptive method in PBF enrolled 30-Jan-2015 30-Jun-2019 30-Jun-2019 health facilities in target Page 39 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) districts - second batch (Leribe) PDO 1c: Number of women Number 21958.00 32000.00 30410.00 using modern contraceptive method in PBF enrolled 29-Jul-2016 30-Jun-2019 30-Jun-2019 health facilities in target districts - third batch (Mafeteng, Mohale's Hoek, Mokhotlong, Thaba-Tseka) PDO 1d: Number of women Number 50903.00 65000.00 41969.00 using modern contraceptive method in PBF enrolled 15-Mar-2018 30-Jun-2019 30-Jun-2019 health facilities in target districts - fourth batch (Maseru, Berea, Qacha's Nek, Butha-Buthe) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion People who have received Number 24153.00 34340.00 37575.00 essential health, nutrition, and population (HNP) 31-Dec-2014 30-Jun-2019 30-Jun-2019 services Page 40 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Number of children Number 22834.00 31440.00 35607.00 immunized 31-Dec-2014 30-Jun-2019 30-Jun-2019 PDO 2a: Number of children Number 1236.00 1500.00 1645.00 under-1 fully immunized at PBF enrolled facilities-first 31-Dec-2014 30-Jun-2019 30-Jun-2019 batch (Quthing) PDO 2b: Number of children Number 4391.00 4500.00 5292.00 under-1 fully immunized at PBF enrolled facilities- 30-Jan-2015 30-Jun-2019 30-Jun-2019 second batch (Leribe) PDO 2c: Number of children Number 5804.00 8440.00 10438.00 under-1 fully immunized at PBF enrolled facilities - third 31-Oct-2016 30-Jun-2019 30-Jun-2019 batch (Mafeteng, Mohale's Hoek, Mokhotlong, Thaba- Tseka) PDO 2d: Number of children Number 10923.00 17000.00 18232.00 under-1 fully immunized at PBF enrolled facilities - 15-Mar-2018 30-Jun-2019 30-Jun-2019 fourth batch (Maseru, Berea, Qacha's Nek, Butha- Buthe) Page 41 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Number of women and Number 1319.00 2900.00 1968.00 children who have received basic nutrition services 30-Dec-2016 30-Jun-2019 31-Dec-2018 PDO 3a: Number of Number 78.00 150.00 106.00 underweight children under 5 years detected and 30-Dec-2016 30-Jun-2019 31-Dec-2018 treated in the target districts-first batch (Quthing) PDO 3b: Number of Number 208.00 250.00 127.00 underweight children under 5 years detected and 30-Dec-2016 30-Jun-2019 31-Dec-2018 treated in the target districts- second batch (Leribe) PDO 3c: Number of Number 225.00 700.00 249.00 underweight children under 5 years detected and 30-Dec-2016 30-Jun-2019 31-Dec-2018 treated in the target districts - third batch (Mafeteng, Mohale's Hoek, Mokhotlong, Thaba-Tseka) PDO 3d: Number of Number 808.00 950.00 421.00 Page 42 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) underweight children under 15-Mar-2018 30-Jun-2019 31-Dec-2018 5 years detected and treated in the target districts- fourth batch (Maseru, Berea, Qacha's Nek, Butha-Buthe) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion PDO 4: Number of patients Number 3725.00 4220.00 3428.00 started on TB treatment in the target districts 31-Dec-2014 30-Jun-2019 30-Jun-2019 PDO 4a: Number of patients Number 150.00 200.00 198.00 started on TB treatment in the target districts-first 31-Dec-2014 30-Jun-2019 30-Jun-2019 batch (Quthing) PDO 4b: Number of patients Number 595.00 670.00 475.00 started on TB treatment in the target districts- second 30-Jan-2015 30-Jun-2019 30-Jun-2019 batch (Leribe) PDO 4c: Number of patients Number 888.00 1050.00 931.00 started on TB treatment in the target districts- third 29-Jul-2016 30-Jun-2019 30-Jun-2019 Page 43 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) batch (Mafeteng, Mohale's Hoek, Mokhotlong, Thaba- Tseka) PDO 4d: Number of patients Number 2119.00 2300.00 1824.00 started on TB treatment in the target districts- fourth 15-Mar-2018 30-Jun-2019 30-Jun-2019 batch (Maseru, Berea, Qacha's Nek, Butha-Buthe) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion PDO 5: Number of people Number 128037.00 216300.00 213233.00 currently on HIV treatment in the target districts 30-Dec-2016 30-Jun-2019 31-Dec-2018 PDO 5a: Number of people Number 5354.00 7500.00 6943.00 currently on HIV treatment in the target districts-first 30-Dec-2016 30-Jun-2019 30-Jun-2019 batch (Quthing) PDO 5b: Number of people Number 24690.00 38500.00 35824.00 currently on HIV treatment in the target districts - 30-Dec-2016 30-Jun-2019 30-Jun-2019 second batch (Leribe) Page 44 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) PDO 5c: Number of people Number 13797.00 17300.00 53369.00 currently on HIV treatment in the target districts- third 30-Dec-2016 30-Jun-2019 30-Jun-2019 batch (Mafeteng, Mohale's Hoek, Mokhotlong, Thaba- Tseka) PDO 5d: Number of people Number 108496.00 115000.00 117097.00 currently on HIV treatment in the target districts - 15-Mar-2018 30-Jun-2019 30-Jun-2019 fourth batch (Maseru, Berea, Qacha's Nek, Butha- Buthe) Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion PDO 6: Average Health Percentage 59.60 50.00 78.00 81.00 Facility Quality of Care Score in target districts 31-Dec-2015 30-Jun-2017 30-Jun-2019 07-Jun-2019 Comments (achievements against targets): Objective/Outcome: Improve contract management of select PPPs Page 45 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion PDO 7: MOF Central PPP Unit Percentage 0.00 100.00 25.00 and MOH PPP Contract Management office 31-Oct-2016 30-Jun-2019 07-Jun-2019 established and fully staffed Comments (achievements against targets): A.2 Intermediate Results Indicators Component: Component 1: Improving Health Service Delivery through Performance-Based Financing Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI 1: Number of health Number 0.00 107.00 171.00 171.00 facilities with PBF contracts 30-Jan-2013 30-Jun-2017 30-Jun-2019 30-Jun-2019 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI 4: Community-based Percentage 75.00 89.00 78.00 Page 46 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) satisfaction score for PBF 31-Dec-2014 30-Jun-2019 31-Dec-2018 enrolled facilities in the target districts Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI 7: Number of Quality Number 0.00 4.00 4.00 checklists revised and streamlined (cumulative) 31-Dec-2014 30-Jun-2019 29-Mar-2019 Comments (achievements against targets): Component: Component 2: Capacity Building Support to the Ministry of Health Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI 2: Number of health Number 0.00 465.00 826.00 personnel in the target districts that received 31-Dec-2014 30-Jun-2019 29-Mar-2019 training focused on clinical services Comments (achievements against targets): Page 47 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI 3: Number of health Number 0.00 6500.00 2266.00 personnel in the target districts that received non- 31-Dec-2014 30-Jun-2019 31-Dec-2018 clinical health systems- related training Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI 6: Number of District Number 2.00 18.00 42.00 Steering Committee meetings in target districts 31-Dec-2014 30-Jun-2019 29-Mar-2019 providing feedback and grievance redress mechanims based on assessments to facilities and involve community representatives Comments (achievements against targets): Page 48 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Component: Component 3: Enhance PPP Management Capacity within the Government of Lesotho Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IRI 5: Number of Terms of Number 0.00 4.00 4.00 References for key PPP positions in the Government 31-Oct-2016 30-Jun-2019 31-Dec-2018 drafted and approved Comments (achievements against targets): Page 49 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1. Increase utilization of primary health services in selected districts in Lesotho with a particular focus on maternal and child health, TB and HIV 1. Number of women using modern contraceptive method in PBF enrolled health facilities in target districts 2. Number of children under-1 fully immunized at PBF enrolled facilities Outcome Indicators 3. Number of underweight children under 5 years detected and treated in the target districts 4. Number of patients started on TB treatment in the target districts 5. Number of people currently on HIV treatment in the target districts Intermediate Results Indicators 1. Number of health facilities with PBF contracts 1. PBF services provided by health workers, health centers and hospitals (C1) Key Outputs by Component 2. PBF services provided by district health management teams (C1) (linked to the achievement of the Objective/Outcome 1) 3. Improved implementation and supervision of services (C1) 4. Improved performance purchasing capacity of MOH PBF unit and facilities (C2) Objective/Outcome 2. Improve the quality of primary health services in selected districts in Lesotho Outcome Indicators 1. Average Health Facility Quality of Care Score in target districts (%) 1. Number of health personnel in the target districts that received training focused on clinical services 2. Number of health personnel in the target districts that received Intermediate Results Indicators non-clinical health systems-related training 3. Community-based satisfaction score for PBF enrolled facilities in the target districts (%) Page 50 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) 4. Number of District Steering Committee meetings in target districts providing feedback and grievance redress mechanisms based on assessments to facilities and involve community representatives 5. Number of Quality checklists revised and streamlined 1. Improved implementation and supervision of services (C1) 2. Improved capacity of MOH and CHAL at central and district levels (C2) 3. Improved performance purchasing capacity of MOH PBF unit and Key Outputs by Component facilities (C2) (linked to the achievement of the Objective/Outcome 2) 4. Knowledge and skills of village health workers and health professionals improved, including in EmONC and supply chain (C2) 5. Better knowledge and skills in monitoring and evaluation at central and district levels (C2) Objective/Outcome 3. Improve contract management of select PPPs 1. MOF Central PPP Unit and MOH PPP Contract Management office Outcome Indicators established and fully staffed (%) 1. Number of Terms of References for key PPP positions in the Intermediate Results Indicators Government drafted and approved 1. Establishment of the MOF Central PPP unit and MOH PPP Contract Management Office (C3) 2. Recruitment of the full PPP management staff complement for MOF Key Outputs by Component PPP Unit and MOH PPP Contract Management Office (C3) (linked to the achievement of the Objective/Outcome 3) 3. Provision of technical assistance (TA) to strengthen oversight over the QMMH network PPP and other existing health PPPs, including PPP management capacity-building (C3) Page 51 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) ANNEX 2. PDO AND INTERMEDIATE RESULTS INDICATORS AT APPRAISAL, 2016 AND 2018 RESTRUCTURINGS PDO Indicators at Appraisal November 2016 Restructured Comments May 2018 Restructured PDO Comments PDO Indicators Indicators PDO 1. Pregnant women PDO 1. Pregnant women Revised for clarity of language Dropped because percentage delivering in health facilities (%) delivering in PBF enrolled health of institutional deliveries facilities in target districts (%) improved during project implementation and became Baseline: 56.2% Baseline: 53.2% relatively high. GOL and WB Target: 64% Target: 80% decided to include a more challenging to attain indicator, e.g. related to family planning PDO 2. Children 1 year old who PDO 2. Children under 1-year fully Revised for clarity of language PDO 2. Number of children Revised from percentage to received all basic vaccinations (%) immunized in PBF enrolled health under-1 fully immunized at numerical indicator, and facilities in the target districts (%) PBF enrolled facilities consistent with HNP Corporate Results Indicator Baseline: 62.1% Baseline: 60.1% Baseline: 22,834 (CRI) Target: 67% Target: 72% Target: 31,440 PDO 3. Currently married women PDO 3. Currently married women Revised for clarity of language PDO 1. Number of women Newly introduced indicator to using modern contraceptive using modern contraceptive using modern contraceptive cover all women to replace method (%) method in target districts (%) method in PBF enrolled former indicator on only health facilities in target married women, and revised districts from percentage to numerical indicator Baseline: 42.6% Baseline: 40.4% Baseline: 70,956 Target: 48% Target: 62% Target: 117,900 PDO 3. Number of Newly introduced indicator underweight children under 5 addressing malnutrition years detected and treated in (underweight) the target districts Baseline: 1319 Target: 2050 PDO 4. People receiving Upgraded from IR level due to PDO 4. Number of patients Newly introduced indicator to tuberculosis treatment in revised PDO started on TB treatment in measure initiation of TB Page 52 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) accordance with the WHO- the target districts treatment to replace former recommended “Directly Observed TB DOTS indicator Treatment Strategy” (DOTS) (Number) Baseline: 4925 Baseline: 3725 Target: 9500 Target: 4220 PDO 5. Pregnant women living Upgraded from IR level due to PDO 5. Number of people Newly introduced to measure with HIV who received ARV revised PDO currently on HIV treatment in all HIV patients on treatment prophylaxis or complete course of the target districts to replace former MTCT ARV to reduce the risk of MTCT in indicator target districts (Number) Baseline: 3910 Baseline: 128,037 Target: 13,000 Target: 178,300 PDO 4. Average health facility PDO 6. Average Health Facility Revised for clarity of language PDO 6. Average Health Facility No Change quality of care score (%) Quality of Care Score in target Quality of Care Score in target districts (%) districts (%) Baseline: 43.8% Baseline: 59.6% Baseline: 59.6% Target: 50% Target: 78% Target: 78% PDO 7. MOF Central PPP Unit and Newly Introduced to reflect PDO 7. MOF Central PPP Unit Language revised to include MOH PPP Contract Management introduction of PPP Contract and MOH PPP Contract “‘and fully staffed’” office established Management Support in PDO Management office established and fully staffed Baseline: 0% Baseline: 0% Target: 100% Target: 100% Intermediate Results Indicators November 2016 Restructured IR Comments May 2018 Restructured IR Comments at Appraisal Indicators Indicators IR 1. Pregnant women in a lowest Dropped because of a lack of wealth quintile delivering in availability of routine household health facilities (%) level data in between DHS/MICS surveys to accurately measure Baseline: 32.2% progress/equity and the absence Target: 35% of suitable substitutes from the HMIS. IR 2. Women with at least four Dropped because of a lack of antenatal care visits during availability of district-level Page 53 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) pregnancy (%) disaggregated household data to accurately measure routine Baseline: 70.4% progress across this indicator, Target: 74% and the absence of suitable substitutes from the HMIS. IR 3. Births attended by skilled Dropped due to the lack of health personnel (Number) reliable data from the HMIS to routinely measure progress Baseline: 17,453 across this indicator. Target: 77,000 IR 4. Mothers who received IR 1. Mothers who received Revised for clarity of language Dropped postnatal care within two days of postnatal care within two days of childbirth (%) childbirth in target districts Baseline: 42.1% Baseline: 42.1% Target: 47% Target: 70% IR 5. Pregnant women receiving IR 2. Pregnant women receiving Revised for clarity of language Dropped antenatal care from a health antenatal care during a visit to a provider (Number) health provider (Number) Baseline: 24,324 Baseline: 24,324 Target: 100,000 Target: 60,000 IR 6. Children receiving IR 3. Children Immunised Revised due to lack of Dropped as number of pentavalent vaccine (diphtheria, (Number) availability of HMIS EPI data children fully immunized tetanus, whooping cough, specific to children immunized incorporated as new PDO 2 hepatitis B and Haemophilus against DTP3, and consistent indicator. influenza type b) (Number) with HNP Corporate Results Indicator (CRI). Baseline: 26,474 Baseline: 26,474 Target: 98,000 Target: 50,000 IR 7. Tuberculosis treatment Dropped due to the lack of success rate (%) reliable routine data to effectively measure progress. Baseline: 69% However, a similar indicator was Target: 73% to be included as part of the results framework for the Southern Africa TB project. IR 8. People receiving Upgraded to PDO level due to tuberculosis treatment in the expanded PDO’s focus on Page 54 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) accordance to the WHO- TB. recommended “Directly Observed treatment Short Course” (DOTS) (Number) Baseline: 8,553 Target: 35,600 IR 9. Pregnant women living with Upgraded to PDO level due to HIV who received ARV the expanded PDO’s focus on prophylaxis or complete course HIV. of ARV to reduce the risk of MTCT (Number) Baseline: 4,972 Target: 21,000 IR 10. Children under 5 years Dropped due to the lack of whose weight and height are reliable data from the HMIS to monitored regularly (Number) routinely measure progress across this indicator, although children treated for malnutrition were to be incentivized under the revised PBF quantity indicators. IR 11. Number of health facilities IR 4. Number of health facilities Target revised due to scale IR 1. Number of health Target revised due to scale up with PBF contracts with PBF contracts down from nine to six districts. facilities with PBF contracts from six to ten districts. Baseline: 0 Baseline: 0 Baseline: 0 Target: 107 Target: 75 Target: 171 IR 12. Health facilities reporting Dropped because it was not stock-out of tracer medicines and considered relevant to the PDO. medical supplies at the time of However, stock-outs were to the health facility quality of care continue to be assessed through assessment (%) the routine facility quality of care assessments. Baseline: N/A Target: 5% IR 13. Health personnel receiving IR 5. Health Personnel Receiving Original IR13, through to IR19 IR 2. Number of health Condensed training indicator training in Advanced Midwifery Training (training related IRs) were personnel in the target split into two new indicators, and Neonatology (Number) condensed into one training districts that received training the first of which focused on Page 55 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Baseline: 0 Baseline: 0 related IR focused on clinical services clinical training Target: 20 Target: 1,500 IR 14. Health personnel receiving Baseline: 0 pre-service nurse anesthetists Target: 465 training (Number) Baseline: 0 Target: 12 IR 15. Nurses receiving training on the MOH adopted drug supply management manual (Number) Baseline: 0 Target: 150 IR 16. Hospital and DHMT pharmacists receiving ESAMI training courses (Number) Baseline: 0 Target: 18 IR 17. Personnel receiving IR 3. Number of health training in procurement and personnel in the target financial management (Number) districts that received non- Condensed training indicator Baseline: 0 clinical health systems-related split into two new indicators, Target: 16 training the second of which focused IR 18. Village health workers on non-clinical health systems trained (Number) Baseline: 0 training. Baseline: 0 Target: 6,500 Target: 1,500 IR 19. Monitoring and Evaluation officers and District Health Information Officers receiving formal M&E training (Number) Baseline: 0 Target: 12 IR 4. Community-based New indicator to measure satisfaction score for PBF community satisfaction enrolled facilities in the target districts Baseline: 75% Target: 89% Page 56 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) IR 5. Number of Terms of New PPP related indicator References for key PPP positions in the Government drafted and approved Baseline: 0 Target: 4 IR 6. Number of District Steering Committee meetings in target districts providing feedback and grievance redress mechanisms based on assessments to facilities and involve community representatives Baseline: 2 Target: 18 IR 7. Number of Quality checklists revised and streamlined Baseline: 0 Target: 4 Page 57 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) ANNEX 3. PDO UTILIZATION INDICATORS’ ACHIEVEMENTS COMPARING ‘WITH’ AND ‘WITHOUT’ PHASE 3 OF PROJECT IMPLEMENTATION PDO Indicator 1. Number of women using modern contraceptive method in PBF enrolled health facilities in target districts Achievement (%) Phases 1 (Quthing, Leribe) and II (Mafeteng, Mohale’s Hoek, Mokhotlong and Thaba -Tseka) 77.3 Phases I, II, and III (Berea, Butha -Buthe, Maseru and Qacha’s Neck) 43.9 PDO Indicator 2. Number of children under-1 fully immunized at PBF enrolled facilities in target districts Phases I and II 197.5 Phase I, II and III 148.4 PDO Indicator 3. Number of underweight children under 5 years detected and treated in the target districts Phases I and II 76.9 Phases I, II, and III 66.6 PDO Indicator 4. Number of patients started on TB treatment in the target districts Phases I and II -10.1 Phases I, II, and III -60.0 PDO Indicator 5. Number of people currently on HIV treatment in the target districts Phases I and II 268.77 Phases I, II, and III 169.5 Page 58 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) ANNEX 4. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Kanako Yamashita-Allen Task Team Leader(s) Chitambala John Sikazwe Procurement Specialist(s) Tandile Gugu Zizile Msiwa Financial Management Specialist Hocine Chalal Social Specialist Lungiswa Thandiwe Gxaba Social Specialist Melissa C. Landesz Social Specialist Supervision/ICR Christine Lao Pena, Omer Ramses Zang Sidjou Task Team Leader(s) George Daniel Procurement Specialist(s) Tandile Gugu Zizile Msiwa Financial Management Specialist Yvette M. Atkins Team Member Arundhati Inamdar Willetts Environmental Specialist Paolo Belli Team Member Michael Opagi Team Member Naoko Ohno Team Member Peter Boere Team Member Majbritt Fiil-Flynn Social Specialist Jason Lee Team Member Amer Dastgir Team Member Mantsebo Moipone Amelia Ndlovu Social Specialist Ntaoleng Celestina Mochaba Environmental Specialist Page 59 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY10 4.500 84,484.64 FY11 .575 30,864.11 FY12 44.113 238,589.26 FY13 63.140 400,193.11 FY14 0 0.00 Total 112.33 754,131.12 Supervision/ICR FY13 5.928 52,454.65 FY14 57.649 279,130.89 FY15 48.800 298,947.17 FY16 51.903 305,815.10 FY17 42.119 276,858.93 FY18 40.743 246,780.58 FY19 38.253 221,499.56 FY20 8.790 68,237.96 Total 294.19 1,749,724.84 Page 60 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) ANNEX 5. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Component 1: Improving Health Service Delivery 13.7 11.64 84.9% through Performance-Based Financing Component 2: Capacity Building Support to the 2.3 3.73 162.17% Ministry of Health Component 3: Enhance PPP Management Capacity within 0 .63 N/A the Government of Lesotho Component 4: Contingent 0 0 N/A Emergency Response Total 16.00 16.00 100.00 Page 61 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) ANNEX 6. EFFICIENCY ANALYSIS 1. The economic analysis for the project shows a sound economic rationale for the investment. However, as expected, with the changes in the last few years before the project ended, especially the decision to drop the AF has had a negative impact in the project’s implementation cycle and economic returns of the investment. 2. Overall, the project has contributed to reducing underweight prevalence, maternal mortality, under-1 mortality, and prevalence of TB and HIV/AIDS, and the benefits justify the costs. The project did so by improving staff motivation, allowing for less disruption of services through facilitated local procurement of services and commodities that are not centrally procured, and investing in outreach services to improve service coverage. Furthermore, the various clinical and non-clinical services provided have improved service delivery capacities across supported districts. The net economic benefits generated by the project’s inputs and outputs resulted in a positive net present value (NPV) of US$48.4M, an internal rate of return (IRR) of 14 percent, and a cost-benefit ratio of 3.1, with a conservative approach in estimating the benefits from the project. These numbers are, however, lower than the PAD (economic rate of return of 70%) as some of the project’s indicators did not meet the original target and the original investment was not completely disbursed, i.e. disbursement was approximately 94 percent. Moreover, the estimates at appraisal and implementation stage cannot be directly compared because the project underwent two restructurings that resulted in revised indicators, targets, component activities and funding, and project scope. The differences are also coming from different methodologies in the analysis. The PAD EA includes an input-output approach with different assumptions made compared to the further EAs performed during the project implementation and for the ICR. One cannot therefore have a reasonable direct comparison between the PAD estimates and the ICR estimates of the rate of return. Considering the differences in terms of design (components, scope, etc.) among projects in the Africa region and their different contexts, it is also not possible to have a reasonable direct comparison of rate of return with other projects Furthermore, the reduction of the project scope from 9 districts to 6 in 2016 and then scaling PBF to 10 districts in 2018, as well as basing the end-line economic analysis on the more reliable DHS 2014 data showing a higher maternal mortality may have contributed to the above difference. 3. The ICR team considered including information on the cost of PBF verification but it was not possible given the context. It was very difficult to separate verification costs from rolling out PBF because it was the first time this approach was being used in the health sector in Lesotho and it was part of the overall PBF package which included other aspects such as PBF sensitization, promotion, training, and others. It notes, however, that over the course of project implementation, the PBF scheme progressively adopted simpler ways to verify volume of services provided, namely: the pilot of risk- based verification of volume of services in two districts, implementation of exit interview surveys in lieu of Community-Based Organizations for Community-Based Verification at the hospital level, and piloting of a phone-based community client survey in selected facilities in Maseru district. 4. The analysis suggests that with an investment of $14.9 million, which is the total amount disbursed (IDA and Trust Fund) over 5 years, 3,300 children’s lives would have been saved, 239 fewer children under-5 would be stunted, and 2,192 women’s lives would be saved due to receiving project’s services (Table A6. 1). Investing in this performance-based financing project would result in an Page 62 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) estimated US$71 M in economic benefits over the lifetime of beneficiaries. Table A6. 1. Impact of the Project Year 0 Year 1 Year 2 Year 3 Year 4 Year 5 TOTAL Interventions'period 2014 2015 2016 2017 2018 2019 Assumed disbursement 397,611 963,593 1,594,277 2,584,204 4,495,829 4,954,372 14,989,886 ALL # maternal deaths averted 320 343 360 373 383 413 2,192 # child (<1) deaths averted 432 492 537 571 596 672 3,300 # children underweight averted (<5) 32 36 39 41 43 48 239 # deaths averted (TB,HIV) 167 194 215 230 242 276 1,325 Total number of deaths averted 919 1,029 1,112 1,174 1,221 1,361 6,817 # children underweight averted (<5) 32 36 39 41 43 48 239 Source: calculated based on service utilization data and their relation to mortality /mortality averted 5. The total costs amounted to US$14.9M disbursed (IDA and Trust Fund) over the project’s lifecycle as shown in Table A6.2. As mentioned in the previous section, the additional financing did not occur, so the analysis does not include the then planned additional financing of US$9M. The disbursement graph in figure A.1 shows an upward convex slope which demonstrates an inefficient disbursement in the beginning of the cycle. Table A6. 2. Project Disbursement by Year Year 2014 2015 2016 2017 2018 2019 Yearly 397,611 1,991,888 3,547,797 6,487,717 8,502,169 963,593 Accumulated 397,611 2,955,481 5,539,685 10,035,514 1,361,204 14,989,886 Page 63 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Figure A6. 1. Disbursement (in $) Disbursement 16,000,000 14,000,000 12,000,000 10,000,000 8,000,000 6,000,000 4,000,000 2,000,000 - 2014 2015 2016 2017 2018 2019 Accumulated 6. Sensitivity analyses showed that the results of the cost benefit analysis (CBA) were sensitive to changes in key modelling assumptions, but the main conclusions remain unchanged, i.e. that the investment was justified on economic grounds (Table A6. 3). A higher discount rate (from 5 to 10 percent) would reduce the economic return of the project’s investment, yet it would remain economically sound. Likewise, factoring the double counting factor (please see assumption box for more information) also yields 12 percent IRR with US$ 34M NPV, thus establishing a sound economic rationale for the proposed investment. Table A6. 3. Sensitivity Analysis BCR IRR NPV Base-case scenario (5% discount rate) 3.1 14% 48 M Sensitivity analysis Double counting (80%), same discount rate 2.5 12% 34 M Discount rate (10%) 1.5 14% 8M Double counting (80%), Discount rate (10%) 1.2 12% 3M 7. The cost benefits analysis (CBA) is built directly on the PDO and its indicators shown in the framework below ( 8. 9. 10. 11. Figure A6. 2), while recognizing the greater complexity of the underlying causal chain of the interventions on the outcomes and economic benefits. The project’s effectiveness/benefits were estimated separately for each project component and then transformed into monetary value in aggregated form. Page 64 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Figure A6. 2. Causal Chain Framework 12. The assumptions made to conduct the economic analysis are presented in Box A6.1 below. Page 65 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Box A6. 1. Assumptions used for the economic analysis The impact of the Project’s interventions was estimated separately for each component in terms of lives saved and human capital/potential productivity preserved (Figure A.2). The PDO is to increase utilization and improve the quality of primary health services in Lesotho with a focus on maternal and child health, TB and HIV. There are five PDO indicators (Figure 1) and we estimated the number of deaths averted and the number of cases of underweight children averted, among children below the age of 5. There are a number of studies that serve as the bases for calculating the impact of project interventions. 1. Nutrition services: We estimate improve child nutrition and prevent related diseases from Bhutta et al. (2008) paper. The most conservative study estimates are that all nutrition-related interventions reduce child deaths by 0.173 percent and child underweight prevalence by 0.241 (up to 24 months). 2. Reproductive health services: We estimated the number of maternal lives saved resulting from the estimated increase in use of modern contraceptive methods. Like for component 1, we used global evidence on the impact of contraceptive use on maternal deaths. The study by Ahmed et al. (2012) found that the total impact of increasing use of contraceptive methods, i.e. through spacing births or reducing the number of pregnancies and thus deliveries and unsafe abortions, reduces maternal deaths by 44 percent. 3. Vaccination: We estimated the number of children lives saved resulting from increasing the number of vaccinated children under 1. According to McGovern et al.’s (2015) paper, an increase in the mean vaccination coverage was associated with a decrease in cluster child mortality of 24%. 4. HIV therapy: We estimated the number of lives saved resulting from receiving HIV treatment. The global evidence noted by WHO argues that the gains in HIV treatment is responsible for a 26% decline in AIDS-related deaths. 5. TB DOTS: Similar to the case for HIV therapy, we estimated the number of lives saved resulting from receiving TB treatment. Beyene et al. (2016) argues that the risk of dying significantly reduced in patients receiving Cotrimoxazole preventive therapy (CPT) by 76.6 % compared to those not receiving CPT. Discount rate. The analysis assumes a 5% discount rate. The 5% discount rate computed through the subtraction between the treasury bond rate and inflation in Lesotho (15%-5%), which results in a more conservative estimate compared to the 3% (Ramsey Formula). For the sensitivity analysis, we included 10% discount rate in the calculation as the discount rate has been fluctuating from 10% to 5% in the last decade. Double counting factor. We also consider the double counting factor when providing the services to the target group. We expect that some beneficiaries from the baseline (i.e. females receiving modern contraception, or children receiving nutritional benefits) would receive the benefits in the following year and therefore, we eliminate the duplication. Here we expect that 80% of the beneficiaries will be new. Page 66 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) ANNEX 7. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS HEALTH SECTOR PERFORMANCE ENHANCEMENT PROJECT – IMPLEMENTATION COMPLETION REPORT GOVERNMENT OF LESOTHO COMMENTS PREAMBLE: The PBF project was relevant to the Health sector. It was aligned to the global and national priorities. The priorities of the Ministry of Health are to provide quality health services, reduce home-based deliveries and reduce maternal and child mortality and ensure that mothers deliver in health facilities. The project has met its objectives as there has been increase in mothers delivering in health facilities; health workers performance has improved; and health facilities were empowered to make decision on the utilization of the incentives by investing in critical items to improve infrastructure in their facilities such as doing minor works to maintenance of the facilities. The project did not quite meet its objectives in the first 2 years and was then restructured. The project was designed with lessons learnt from other countries without studying the context of the country. It was observed that it took some time for the Staff of the Ministry to understand and appreciate the objectives of the project. But once the results became apparent the buy-in for the project was tremendous and the various programmes began to assume ownership for the implementation and results. The thinking for the project was that the performance-based financing would be implemented across the Government system. The PAD does not stipulate clearly what the plans for sustaining the project are. The staff engaged in the project (Senior PBF Officer, PBF officer and M&E officer) who are supposed to be absorbed in the Ministry of Health Establishment, the ministry has not catered for them. It is hoped that the project has instilled the efficiency in the utilization of resources and improvement in the performance of Health professionals at facility level. Health Facilities will maintain the supervision methods that were introduced during the project and make it more systematic. The project was envisaged to end in June 2019, there was request and promise that additional funding would be available from World Bank and that the project be extended for one year. This was because the project had expanded to the last four districts of Maseru, Berea, Qacha’s Nek and Butha-Buthe in July 2018. The additional funding was not provided, and the project had to end. These caused a little disruption on the planned activities and payment of incentives. Government was still preparing for its institutionalization. Then there was another promise that PBF would be part of the proposed project on Lesotho Nutrition and Health Strengthening project to commence in January 2020. Government had to find ways of filling the gap (or vacuum) that would be created between the closure of the PBF project and the new one. Certain activities had to be undertaken to keep the spirit and morale of the human resources in the facilities. Government was assured of that there would be no break in the activities and that the Ministry should continue to undertake activities (including payment of incentives) using counterpart funds and that these would be duly re-imbursed under the new project. The questions are: given that the proposed second-Generation Model of PBF brings about a change in the application of the model, will this still be applicable and whether the new model will be relevant and effective. It is highly likely that there Page 67 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) will be no government counterpart funding available for the new Nutrition and Health Systems Strengthening project. BACKGROUND: When the project was designed, Lesotho remained off track to meet the Millennium Development Goals (MDGs) 4 - reducing child mortality and 5 - improving maternal health. According to 2014 Lesotho Demographic and Health Survey, Maternal Mortality Rate is still among highest in Sub-Saharan Africa (1,024 deaths per 100,00 live births); Under-5 mortality rate (85 deaths per 1,000 live births), TB per capita incidence ranks Lesotho first globally in terms of TB incidence (852/100,000) and high co-infection with HIV (74% of TB patients tested were HIV positive), and HIV prevalence among adults 15-49 years is the second highest globally (25%). In addition, the health system was faced with several gaps and challenges that included: acute shortage of human resources for health, inability to absorb all funds allocated to the health sector, outdated health legislation, inequalities and inequities in service delivery and a difficult terrain, as most parts of the country were hard to reach13. About 40 percent of the population lives in remote rural villages; often several hours walk through rough mountain paths to the nearest facility. Despite the GOL’s effort to improve access to health care by eliminating user fees from all public health centers including facilities affiliated with Christian Health Association of Lesotho (CHAL) and Lesotho Red Cross Society (LRCS) in 2008, access to health services remained a serious challenge. In an effort to reverse the above challenges and to improve the utilization, accessibility and quality of health care services, with major focus on the community level, the GOL through the Ministry of Health (MOH) took a conscious decision to adopt Performance-Based Financing (PBF) in the health centres, district hospitals and District Health Management Teams (DHMTs) of Lesotho. The PBF project was initially selected to cover nine of the ten districts and excluded Maseru district. The selected districts were Quthing, Leribe, Thaba Tseka, Mokhotlong, Mafeteng, Mohale’s Hoek, Butha Buthe, Berea and Qacha’s Nek). A three-phased approach was adopted to allow for adjustments in design based on lessons learned. The project was piloted in Quthing and Leribe in 2014 and 2015 respectively. In 2016 the project scaled–up to 4 districts (Mokhotlong, Thaba-Tseka, Mafeteng and Mohale’s Hoek) and in July 2018, was extended to cover the whole country through the scale-up to the 4 remaining districts (Maseru, Berea, Qacha’s Nek and Butha-Buthe). Due to the success of the project, a decision was made to include Maseru District and cover the entire country. In June 2019, when the project ended, PBF was in all the ten districts of the country, including the hard to reach areas; in 156 health centres; 17 hospitals from government, CHAL and LRCS facilities. The original Project Development Objectives were to: improve utilization and quality of maternal and newborn health (MNH) services in selected districts in Lesotho. 13 WHO Country Cooperation Strategy at a glance, May 2014. Page 68 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) PBF PROJECT ELEMENTS The PBF health service packages were aligned with the MOH essential health package while at the same time focusing on the improvement of the health MDGs. Both the health centres HC) and the district hospitals (DH) were involved, and PBF adopted both the; Minimum Package of Activities (MPA): for the health center and community level and the Complementary Package of Activities (CPA): for the district hospital. Based on the pre-determined indicators, the project incentivized health facilities on the performance or attainment of those targets through verification mechanisms such as health centre and hospital quantity and quality assessment using specially designed checklists, combined with a remoteness factor. The assessment tools covered a broad spectrum of services from maternal and neonatal services to other selected services such as those for HIV and AIDS, tuberculosis and nutrition. Promotion of utilization of services was done through a continuum of care throughout pregnancy, childbirth, postpartum and the neonatal period by simultaneously stimulating performance of health workers at the health. The incentives could then be equally distributed to fund investments in the improvement of services and to provide staff bonuses. Hospitals and health centers used their incentives to improve the quality of care through activities such as training, supportive supervision, case reviews, procurement of equipment and supplies. One hospital was notably able to procure an ambulance with their incentives. Given that the PBF scheme involved financing, data verification and auditing were conducted systematically and consistently every month and quarterly basis. Both the PPTA and the DHMTs would verify the quantity and quality of health services and would be validated by the District PBF steering Committees. Community participation was also promoted to strengthen project ownership and accountability. The PPTA engaged local NGOs or Community Based Organizations (CBOs) for tracing patients, randomly selected from health facility records, and verified the services received and determined their satisfaction with these services. Financial audits of the project were conducted annually by an external auditor. Internal audits were also conducted for health facilities that received PBF financing. KEY ACHIEVEMENTS Component 1: Improving Maternal and Newborn Health (MNH) Service Delivery at Community, Primary and Secondary levels through PBF. The objective of the component was to improve MNH service delivery at health facility and community level. The project contributed served to strengthen the quality and utilization of health services and promising results were observed. The quantity and quality of care at health facilities and hospitals improved during the five years of project implementation. This progress was reflected by the increase of qualitative and quantitative indicators as shown further below: New Out-patient Department Consultations The table below illustrates the number of new OPD cases that were seen in the health centres in the various districts. The total number increased from 37,338 when the project was piloted in Quthing to 2,407,219 country-wide, when the project ended. At each district level, the figures continue to reflect a steady and encouraging increase, especially since the figures for 2019 are for only two months (January Page 69 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) and February 2019 when the last quantity verification was held). The project was thus able to increase the utilization of OPD services. The health centres were implementing the strategies in the business plans like public gatherings and outreaches. It is worth noting that new OPD cases do not include patients who revisited the health centres for chronic diseases like hypertension or diabetes. Table 1 Verified Number of new outpatient consultations for curative care consultations (PBF Web) 2014 2015 2016 2017 2018 2019 Grand Total Berea 53520 20008 73528 Butha-Buthe 21975 8324 30299 Leribe 196885 206087 219790 185285 26785 834832 Mafeteng 21086 113479 122578 22626 279769 Maseru 121012 47002 168014 Mohale's Hoek 22096 99028 93978 17457 232559 Mokhotlong 27781 56873 53276 9793 147723 Qacha's Nek 30200 12602 42802 Quthing 37338 64024 68933 65266 53964 9503 299028 Thaba-Tseka 52823 114893 110391 20558 298665 Grand Total 37,338 260,909 398,806 669,329 846,179 194,658 2,407,219 Number of Children under 1 year Fully Immunized and Number of Women delivering in Health Facilities The number of children under 1 year fully immunized increased steadily to 149,814 in 2019 compared to 3,811 in 2014 while the number of women delivering in health facilities reached 143,376 compared to 4,440 in 2014. According to the baseline data that was collected prior the PBF implementation, 87 deliveries were conducted in Leribe district while at the end of the year 2018, 484 deliveries were conducted. The increasing trend was recognised in Mafeteng and Mohale’s Hoek districts. Most of the health centres started providing deliveries after the adoption PBF concept. Facilities in Quthing commenced to book mothers in the shelter when they reach 36 weeks of pregnancy and during their stay in the shelters, they were provided with food until they deliver. In all the districts HCs provided baby pack gifts to mothers who delivered in a health centre. These initiatives resulted in an increase the number of women delivering in health facilities and thus improved the overall delivery outcome. Pontmain Health Centre in Leribe district constructed the waiting mothers’ kitchen and procured utensils and furniture for the kitchen. The health facilities also hired temporary staff to assist in conducting deliveries where there was a need. The number of children under the age of one year old was stagnant in most of the districts but it decreased due to the stock outs of vaccines such as measles and BCG in the last quarter of 2018. Outreaches were conducted to improve the performance. At the end of the project the number of immunized children had increased upon the availability of the vaccines. Page 70 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Figure 1: Influence of the project on children under 1 year and pregnant women (MCDI/Health Works Lesotho Exit Report, May 2019) Similar positive outcomes were observed in the case of the other quantitative indicators such as the number of users of contraceptive methods. The number of users of long-term modern contraceptive methods has risen significantly by 1125 times. The clients who were provided with short-term modern contraceptive methods showed a constant performance while the long term contraceptive methods dipped in 2018. Possible reasons put forward included the fact that health facilities preferred using implants more than other long term contraceptives such as IUCDs and the fact that clients who were on implants were reported to have become pregnant while still using the contraception. This was found to be common in HIV positive women on ART. In order to increase the performance on this indicator, facilities increased number of health educations and outreach services. Figure 2. Project impact on the usage of contraceptive methods (MCDI/Health Works Lesotho Exit Report, May 2019) Page 71 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Overall, the quantitative indicators are 20-39 times more in 2019 compared to 2014 when the project started (data not shown) 14.. The indicator (number of patients referred who arrive at the district hospital) augmented beyond this range by 98 times. 15. As for qualitative data, there has been an enhancement since the beginning of the project. Throughout the PBF project implementation, the health facilities were steadily improving in quality scores. And because the quality of care scores in all district hospitals improved significantly, the MOH and the University of Pretoria revised the Quality Checklists at the district hospital level in June 2018 to ensure that the high scores truly reflect the quality of care provided to patients. The revised checklists put greater emphasis on the evaluation of health workers’ clinical skills such as essential steps in management of obstetrics emergencies. In the figures below, the average quality scores increased gradually except for the last three data points where the checklist changed and when new districts enrolled in PBF. Yet, starting the third quarter (Q3) in 2018, the scores resumed improving. The trend for the health centres is also increasing, however the dips observed were as a result of the new entry of districts into PBF – Q4 of 2016 the decrease was caused by new health centres (Mokhotlong, Thaba-Tseka, Mafeteng and Mohale’s Hoek health centres) and in Q3, 2018 when the last four districts joined (Maseru, Berea, Butha-Buthe, Qacha’s Nek). Figure 3. Variation of average quality scores for District Hospitals (MCDI/Health Works Lesotho Exit Report, May 2019) 14 MCDI/Health Works Lesotho Exit Report, May 2019. 15 MCDI/Health Works Lesotho Exit Report, May 2019. Page 72 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Figure 4. Variation of average quality scores for Health Centers (MCDI/Health Works Lesotho Exit Report, May 2019) There have been other positive changes brought about by the project:  Increased staff motivation and performance - facility staff adhere to the business hours and absenteeism considerably reduced.  Improved staff ownership due to the enhancement of health facilities autonomy to manage and plan activities for the health facility; the health workers have the opportunity to develop business plans and use the monetary incentives to improve health facilities based on their prioritization e.g. facilities hired nurses on short-term basis in order to be able conduct deliveries and other services  Increased community participation and feedback through client surveys helped improve service delivery and ownership by Health Centre Committees in managing the health centre activities and services implemented helped address local issues that had otherwise remained unnoticed e.g. advocacy for the repair of roads, access to water and electricity.  Increased and regular supervision of health facilities was done by the DHMTs, where feedback was provided to health facility staff, identified constraints and suggested solutions, and other information related to service delivery within the district. Component 2: Training of health professionals and VHWs and improving Monitoring and Evaluation (M&E) capacity Table 2. Training Supported by Project Training/Assessment Objective Trainees MOH adopted drug supply management Training on the findings of a review of the turnaround 77 health centre nurses for manual time in the working capital management of the Quthing, Leribe, Mafeteng, 5 days National Drug Service Organization (NDSO) and related Mohale’s Hoek and Thaba-Tseka processes at NDSO, MOH, GOL and CHAL health districts facilities which would allow the health centers to improve their forecasting and order preparation for Page 73 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) NDSO This would potentially reduce the delays in turning around and delivering orders and curtail stock- outs of drugs and medical supplies at the health center level Eastern and Southern African Management Build capacity on: 6 Pharmacists from Central level Institute (ESAMI) training courses (i) overview of supply chain management and and a few hospitals attended the (ii) Quantification of health commodities. training and then conducted a step-down training for 77 health centre nurses for Quthing, Leribe, Mafeteng, Mohale’s Hoek and Thaba-Tseka districts. Financial Management Training FM Refresher courses 2 MOH financial management staff attended TOMPRO training. Accountant trained in ICT Based Financial Management and Disbursements Course for Project Accountants for World Bank funded Projects in Kenya Financial management training on financial guidelines for using PBF funds was held for all government facilities Procurement Management Training Procurement Refresher courses 10 Procurement Officers attended the Basic and Advanced Course on Works, Procurement and Selection of Consultants Program and Short course on Goods and Equipment Procurement Part-time training at a university in South To equip Nurses to be able to provide the full 10 Nurses Trained in Advanced Africa for Advanced University Diploma in complement of Basic EmONC services Midwifery and Neonatology Advanced Midwifery and Neonatology To enhance the number of Advanced midwives in the country who in turn would train nurse midwives and provide mentorship and preceptorship of newly trained nurse midwives. Nurse Anesthetists training in African training To overcome the shortage of nurse Anesthetics in the 14 Nurses trained in Anaesthetics institutions. country Long Term Training on Oncology Nursing To overcome the shortage of Nurse Oncologists in the 2 Nurses trained country EmONC assessment, with MOH, UNFPA, To inform the need for on-the-job training for nurse 58 Health Professionals trained on UNICEF, and WHO. midwives and medical doctors providing obstetric EMONC services in districts. Essential Steps in Managing Obstetric To develop and maintain the skills for obstetric and A Doctor and Nurse from 18 Emergencies (ESMOE) master training emergency care Hospitals VHW training Capacity building on: Initial Trainings for 2235 Village - basic services such as family planning and referrals Health Workers in Leribe, Quthing, as well as taking care of mothers and children in the Mohale’s Hoek, Thaba-Tseka and postnatal period and promotion of exclusive Mokhotlong were conducted. breastfeeding. - conducting community head count and periodic update of village health registers for more accurate health facility catchment area data. Monitoring and Evaluation M&E Refresher courses 12 Assistant Statisticians trained Page 74 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) International for Monitoring and Evaluation M&E Refresher courses PBF M&E Officer Trained Development Results Biostatistics and Epidemiology Impact Assessment and Evaluation M&E Refresher courses 2 M&E Officers trained Audit Management for Donor Funded Capacity building hands-on skills on finance and audit 2 Auditors trained Projects management for donor funded projects and programmes PBF Training To provide an understanding of the PBF approach and PBF Director, Chief PPP, CHAL its implementation Executive Director, Director Finance, Economic Planner, PBF M&E Officer, PBF Officer, Senior PBF Officer, HMIS Officer Contract Management To provide an understanding of the contract Senior PBF Officer management process Project Management Cycle Understand the project management cycle PBF Officer Management of Public Health Reform Capacity building on understanding health systems Senior PBF Officer and Senior and their performance with structured approaches to Economic Planner developing health system reform policies for performance improvement Management and Control of Donor Funded To provide an understanding of the management of Director HPSD Projects donor funded projects Health Financing To build capacity on Health Financing Chief Economic Planner MNCH Score Card Study Tour in Tanzania To understand the Score Card and development 8 MOH Officials process so as to facilitate the development of such for Lesotho Health Centre Committees (HCCs) training Capacity building on roles and responsibilities of HCC. 13 members of Mokhotlong HCCs were trained from each health facility. Component 3: Enhance PPP Management Capacity within Government of Lesotho This component was to finalize the organizational structure of the MOH PPP Contract Management Office, and its relationship with the existing MOH PBF Unit, with a proposal to unify the two into a single unit; capacitate the Ministry of Finance (MOF) Central PPP and support the drafting of the PPP policy document. A key priority of this component was how best to make the QMMH Network PPP sustainable. A Clinical Services & Quality Assurance Advisor Consultant was engaged to ensure that the clinical service levels specified in the PPP Agreements are accurately monitored and reported. Direct clinical monitoring with a team of MOH officials and reviews of third-party monitoring reports, were undertaken on a quarterly basis. Other staffing positions identified included MOF Legal and Financial Management positions, where recruitment was at an advanced stage as they were about to be engaged when the project ended, and a MOH Contract Manager position which was never filled. Key MOH, MOF and Development Planning officials were trained on PPP Negotiations Skills to enhance capacity to handle the arbitration process and the possible re-negotiation of the PPP Agreement. IMPLEMENTATION CHALLENGES Delays in effectiveness: The commencement of the project which was supposed to take place in July 2013 was deferred to 2014 because negotiations between the Ministry of Health and the Firm that won the bid for the PPTA contract collapsed and the procurement process had to start again. The procurement process was initiated and on the 12th May 2014 MCDI HealthNet consortium reported on duty. Page 75 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) Lack of dedicated personnel in MOH PBF Unit: The lack of dedicated personnel in the PBF Unit to support key project activities adversely affected the pace of implementation. It was only in November 2016(with the recruitment of the Senior PBF Officer) and January 2017 PBF Officer (with the recruitment of the PBF Officer) that the PBF Unit restored its full staff complement. This then paved the way for an operationally and technically proficient Unit that was slowly able to absorb the functions of the PPTA. FM capacity constraints and Delays in disbursements: The absence of individual health facilities accounts for GOL facilities, for payment of the incentives timely and regularly, posed a great challenge. During the pilot stage in Quthing these were deposited in the District Council (DC) accounts and the system seemingly worked well. But it presented challenges in Leribe since the DC did not approve the use of their account. In July 2016 the health facilities in Quthing and Leribe opened individual bank accounts through health centre committees and the facilities in Quthing stopped using the DC bank account. Other bottlenecks included the delayed transfer of PBF funds from the Ministry of Health to the health facilities, and access to these very funds by the facilities themselves and this also affected the success of program. Motivation of health care workers was hindered, and this also prevented the investment in capacity building activities within the facilities themselves from taking place. PBF Design in Hospitals: Initially hospitals were assessing both quantity and quality but with the limitation that individual bonuses for hospital workers and village health workers (VHWs) were attached to the hospital. As a result, the ownership of and motivation for the PBF at the DH was lower than at the HCs. There was a need therefore for the project to consider the mode of incorporating individual bonuses for hospital workers and the VHWs attached to the hospital into the project design, such that the hospital team could claim a stake in the results. This was considered as part of the project design review during restructuring and hospital staff were now entitled to incentives. Additional Financing: Following a second restructuring where coverage of the project was expanded to all ten districts, a financial gap resulted that was further exacerbated by the general expansion of the project scope to add a new component aimed at enhancing Public Private Partnership management capacity within the GOL. The plan was also to extend the project by an additional year and request Additional Financing from the World Bank to cover the financial gap. GOL duly submitted a request for additional funds, but World Bank management made a decision to drop the Additional Financing and instead incorporate the HSPEP activities in the new Nutrition and Health System Strengthening Project. The Additional Financing request was therefore dropped and the project ended on 30th June 2019. This greatly affected the project operations and led to a serious shortage of funds. Most of the available budget had already been committed and there was a scramble to abruptly dovetail the remaining project activities within the remaining limited budget. The PPTA contract was ended prematurely, quantity assessments were also truncated and there were limited funds to cover the incentive payments – only the first quarter payments (January to March 2019) could be made, covered mostly by government counterpart funds. There remained a gap on where the second quarter (April to June 2019) payments would be sourced. Some procurement activities that were already in progress had to be cancelled. PBF Institutionalization: The Ministry had received Technical Assistance in August 2018 to initiate the process of institutionalizing PBF into the government system as a way of ensuring the sustainability of the PBF approach. Some initial consultations with the relevant stakeholder ministries had been undertaken. The project had proceeded to initiate the transfer of the verification function that was being conducted Page 76 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) by the international PPTA to a national organization as a means of curtailing the high costs of that activity. Evaluations for bids for local firms were at an advanced stage. However, due to the shortage of funds and impending end date of the project, this activity was abruptly halted. The Government was also never provided with a report on the findings of the technical assistance on whether such an endeavour is feasible and as such remains uninformed of the outcome of this assistance. Looking at Lesotho’s government payment system which is not so flexible the ministry was looking forward to the findings and the recommendations on how this can be addressed so that PBF could be institutionalized. Sustainability: The intention of the project was to provide mechanisms for improving the technical efficiency of health service provision and health facility performance through monitoring of key health indicators and outputs. It was hoped that this would increase efficiency in utilization of resources and improved performance of health personnel and health facilities and hence help the country to get more value for money. A combination of improved budget execution for health sector programs and improved efficiency and performance was anticipated. It was the government’s desire to expand PBF schemes within the public sector in general, with the health sector taking the lead in providing multiple opportunities for successful implementation in Lesotho. This would ensure continued implementation of the PBF interventions and that the lessons learned would be extracted for the purposes of other public sector programs. However, this was not explored during the project. Furthermore, the staff engaged in the project (Senior PBF Officer, PBF officer and M&E officer) were supposed to be absorbed in the Ministry of Health Establishment, however the requisite processes were not established hence ministry has not catered for them. International Technical Assistance: The project engaged a number of international technical assistance. This not only rendered the project efforts to be extremely expensive and funds being channeled externally. This impeded continuity in the activities and also meant that limited local capacity was built by the technical assistance. For instance, an international verification company (PPTA) was employed for what was supposed to be a limited period to build local competency on the PBF approach and implementation. But the company remained until the end of the project at an astronomical cost and did not seem to have yielded good value for money. Conclusion and Recommendations: The ministry wishes to acknowledge the good intentions of the project and the support provided by the World Bank. However, it would appear that thorough reviews were not undertaken when conducting comprehensive restructurings of the project. The ministry had set high expectations and felt that the project had ended pre-maturely, just when things were looking up. For instance the last four districts had less than one complete year cycle in the project when it ended. The ministry is hopeful that when moving to the new project, the second generation PBF model will identify and acknowledge the gaps left by the old PBF project and effectively address the expectations that had been set. The staff in the ministry and in health facilities were just beginning to understand the objectives of the project and were warming up to the project because of the benefits being realized by the project. The project ended when ownership was at an all-time high. There has also been a challenge of the high turn-over of senior government officials which has resulted in Page 77 of 78 The World Bank Lesotho Health Sector Performance Enhancement (P114859) a break in the knowledge and information of the project. Not all were on the same level of understanding. Some were not yet sensitized or some had not yet fully grasped the approach. The recommendation is, therefore, to enable the sharing or dissemination of the ICR Report to other government ministries or have an abridged version. Page 78 of 78