Document of The World Bank Report No: ICR3375 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-H1820 IDA-H6380 IDA-H7170 IDA-H7500 TF-17380 TF-95892) ON A GRANT ORIGINAL GRANT IN THE AMOUNT OF 99.3 MILLION SDR (US$150 MILLION EQUIVALENT) FIRST ADDITIONAL FINANCING IN THE AMOUNT OF SDR 50.9 MILLION (US$80 MILLION EQUIVALENT) SECOND ADDITIONAL FINANCING IN THE AMOUNT OF 18.6 MILLION SDR (US$30 MILLION EQUIVALENT) THIRD ADDITIONAL FINANCING IN THE AMOUNT OF SDR 47.3 MILLION (US$75 MILLION EQUIVALENT) TO THE DEMOCRATIC REPUBLIC OF CONGO FOR A HEALTH SECTOR REHABILITATION SUPPORT PROJECT October 25, 2015 Health, Nutrition, and Population Global Practice (GHNDR) Africa Region CURRENCY EQUIVALENTS (Exchange Rate Effective at Project Closing, April 30, 2015) Currency Unit = Congolese franc US$1.00 = 924.00 CDF US$1.00 = 0.72 SDR GOVERNMENT FISCAL YEAR January 1 - December 31 ABBREVIATIONS AND ACRONYMS AAP Agence d’Achat de Performance ACV Agence de Contractualisation et de Vérification ACT Artemisinin-Based Combination Therapy AEE Agence d’evaluation extern AEP Agence d ’Execution Partenaires AF Additional Financing AF-I-II-III Additional Financing I, II, and III ARV Antiretroviral BCECO Bureau Central de Coordination BOD Burden of Disease CAG Cellule d’Appui de Gestion CAGF Cellule d’Appui de Gestion Fiducaire CAS Country Assistance Strategy CCISD Centre de Coopération Internationale en Sante et Développement CEMUBAC Centre Scientifique et Médicale de l’Université Libre de Bruxelles pour ses Activités de Coopération CHW Community Health Workers COOPI Cooperazione Itallienne CREDES Centre de Recherche en Développement et Sante DEP Direction des Etudes et Planification DFID United Kingdom Department for International Development DHS Demographic Health Survey DMI District Medical Inspector DPT3 Diphtheria, Pertussis, Tetanus DRC Democratic Republic of Congo EEA External Evaluation Agency EHS Essential Health Services EMRRP Emergency Multi-Sector Rehabilitation & Reconstruction Project FBOs Faith Based Organizations FEDECAME Fédération des Centrales d’Approvisionnement en Médicaments Essentielles FM Financial Management GAVI Global Alliance for Vaccines and Immunization GF Global Fund GHNDR Health Nutrition and Population Global Practice HMIS Health Management Information System HNP Health, Nutrition, and Population HIV/AIDS Human Immunodeficiency Virus-Acquired Immunodeficiency Syndrome HRITF Health Results Innovation Trust Fund HSRSP Health Sector Rehabilitation Support Project ICR Implementation Completion Report ILI Intensive Learning ICR IMCI Integrated Management of Childhood Illness IP Implementing Partners I-PRSP Interim – Poverty Reduction Strategy Paper IPT Intermittent Preventive Treatment IPPF Indigenous People Planning Framework ISR Implementation Status Report ITN Insecticide Treated Nets KAP Knowledge, Attitude, Practice LLIN Long-Lasting Insecticidal Nets MAP Multi Country HIV/AIDS Project MOPH Ministry of Public Health MTR Mid-Term Review MWMNP Medical Waste Management National Plan NGO Non-Governmental Organization NIP National Immunization Program OOP Out Of Pocket OP Operational Policy OPV Oral Poliovirus PARSS Projet d’Appui à la Réhabilitation du Secteur de la Sante PBCs Performance Based Service Contracts PBF Performance-Based Financing PDO Project Development Objective PDSS Health Systems Strengthening for Better Maternal and Child Health Outcomes PHC Primary Health Center PIU Project Implementation Unit PMI Provincial Medical Inspector PNLP Programme Nationale de Lutte contre le Malaria QAG Quality Assurance Group QALP Quality Assessment of the Lending Portfolio RBF Results-Based Financing RF Results Framework RBM Roll Back Malaria SDR Special Drawing Rights SNIS Système National de l’Information Sanitaire TA Technical Assistance TTL Task Team Leader TSS Transitional Support Strategy UCP Unité de Coordination du Project UNFPA United Nations Population Fund UNICEF United Nations Children’s Fund USAID United States Agency for International Development WB World Bank WHO World Health Organization Senior Global Practice Director: Timothy Grant Evans Country Director: Ahmadou Moustapha Ndiaye Sector Manager: Trina Haque Project Team Leader: Hadia Samaha ICR Team Leader: Hadia Samaha ICR Primary Author: Claudia Rokx Democratic Republic of Congo Health Sector Rehabilitation Project CONTENTS A. Basic Information ............................................................................................................ i  B. Key Dates ........................................................................................................................ i  C. Ratings Summary ............................................................................................................ i  D. Sector and Theme Codes................................................................................................ ii  E. Bank Staff ....................................................................................................................... ii  F. Results Framework Analysis ......................................................................................... iii  G. Ratings of Project Performance in ISRs ...................................................................... vii  H. Restructuring (if any) .................................................................................................. viii  I. Disbursement Profile ................................................................................................... viii  1. Project Context, Development Objectives and Design ................................................... 1  2. Key Factors Affecting Implementation and Outcomes .................................................. 7  3. Assessment of Outcomes .............................................................................................. 17  4. Assessment of Risk to Development Outcome ............................................................. 26  5. Assessment of Bank and Borrower Performance ......................................................... 27  6. Lessons Learned............................................................................................................ 30  7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners............... 32  Annex 1. Project Costs and Financing .............................................................................. 33  Annex 2. Outputs by Component and Overview Changes to Indicators and Revised Targets............................................................................................................................... 34  Annex 3. Economic and Financial Analysis ..................................................................... 46  Annex 4. Bank Lending and Implementation Support/Supervision Processes................. 56  Annex 5. Beneficiary Survey Results ............................................................................... 56  Annex 6. Stakeholder Workshop Report and Results ....................................................... 57  Annex 7. Summary of Borrower’s ICR and/or Comments on Draft ICR ........................ 58  Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders ........................... 59  Annex 9. List of Supporting Documents .......................................................................... 60  Annex 10: MAP ................................................................................................................ 63  A. Basic Information DRC Health Sector Congo, Democratic Country: Project Name: Rehabilitation Support Republic of Project IDA-H1820,IDA- H6380,IDA- Project ID: P088751 L/C/TF Number(s): H7170,IDA-H7500,TF- 17380,TF-95892 ICR Date: 06/16/2015 ICR Type: Core ICR DEMOCRATIC Lending Instrument: SIL Borrower: REPUBLIC OF CONGO Original Total USD 150.00M Disbursed Amount: USD 332.32M Commitment: Revised Amount: USD 332.95M Environmental Category: B Implementing Agencies: Health Ministry UNFPA Cofinanciers and Other External Partners: B. Key Dates Revised / Actual Process Date Process Original Date Date(s) Concept Review: 09/30/2004 Effectiveness: 04/13/2006 04/13/2006 01/15/2010 Appraisal: 04/11/2005 Restructuring(s): 31/05/2011 10/18/2012 Approval: 09/01/2005 Mid-term Review: 03/15/2009 Closing: 06/30/2010 12/31/2014 C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Substantial Bank Performance: Moderately Satisfactory Borrower Performance: Moderately Satisfactory i C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) Bank Ratings Borrower Ratings Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory Implementing Quality of Supervision: Moderately Satisfactory Moderately Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Moderately Satisfactory Performance: Performance: C.3 Quality at Entry and Implementation Performance Indicators Implementation QAG Assessments Indicators Rating Performance (if any) Potential Problem Quality at Entry Project at any time Yes None (QEA): (Yes/No): Problem Project at any Quality of Yes None time (Yes/No): Supervision (QSA): DO rating before Moderately Closing/Inactive status: Satisfactory D. Sector and Theme Codes Original Actual Sector Code (as % of total Bank financing) Central government administration 3 3 Health 90 90 Sub-national government administration 7 7 Theme Code (as % of total Bank financing) Child health 25 25 HIV/AIDS 13 13 Health system performance 25 25 Malaria 24 24 Population and reproductive health 13 13 E. Bank Staff Positions At ICR At Approval Vice President: Makhtar Diop Gobind T. Nankani Country Director: Ahmadou Moustapha Ndiaye Pedro Alba Practice Trina S. Haque Laura Frigenti Manager/Manager: Project Team Leader: Hadia Nazem Samaha Eva Jarawan ii ICR Team Leader: Hadia Nazem Samaha ICR Primary Author: Claudia Rokx F. Results Framework Analysis Project Development Objectives (from Project Appraisal Document) The Project Development Objective (PDO) is to ensure that the target population of selected health zones has access to, and use, a well-defined package of quality essential health services. Revised Project Development Objectives (as approved by original approving authority) The objective of the Project is to ensure that the target population in Project Health Zones has access to, and uses a well-defined package of quality essential health services (EHS). (a) PDO Indicator(s) Original Target Formally Actual Value Values (from Revised Achieved at Indicator Baseline Value approval Target Completion or documents) Values Target Years Indicator 1 : Per capita annual curative consultation rate Value quantitative or 0.19 0.40 0.45 0.37 Qualitative) Date achieved 07/01/2007 07/01/2007 06/01/2012 03/03/2014 Target partially achieved: 0.07 pp/py below target. In 2011 the rate was 0.39 at Comments the original target date, and at which point all original project funds were (incl. % disbursed. However AF 3 revised targets upwards and as a result of the delays in achievement) AF 3 effectiveness, financing and utilization rates dropped and final target was only partially achieved. Indicator 2: Children 0-11 months vaccinated with DPT3 in the Project target zones Value quantitative or 54% 90% 90% 90% Qualitative) Date achieved 07/01/2007 07/01/2007 06/01/2012 03/03/2014 Comments (incl. % Target achieved. HMIS data 2014 achievement) Indicator 3 : Children 12-23 months who have received Oral Polio Vaccine (OPV3) Value quantitative or 46% 90% 89% Qualitative) Date achieved 07/01/2007 12/01/2010 06/01/2012 03/03/2014 Comments (incl. % Target achieved (HMIS 2014). achievement) Indicator 4 : Deliveries assisted by qualified personnel iii Value quantitative or 47 % 70 % 85% 75 % Qualitative) Date achieved 07/01/2007 07/01/2007 06/01/2012 03/03/2014 Comments Target partially achieved. The variance by Province Bandundu and Maniema (incl. % achieved 83 and 87 percent respectively. (DHS 2014) achievement) Indicator 5 : Women 15-49 years new users of family planning Value quantitative or 2% 6% 11% 2% Qualitative) Date achieved 07/01/2007 12/01/2014 06/01/2012 03/03/2014 Target not achieved. In 2011 the project had achieved 6%, a tripling of the Comments baseline rate and close to the original target. With AFIII, the target was increased (incl. % and the subsequent delays in financing resulted in drop in contraceptive supplies achievement) and drop in new users. Under-five children who slept under an insecticide- treated net (ITN) the Indicator 6: night before Value quantitative or 38% 70% 50% 66% Qualitative) Date achieved 07/01/2007 12/01/2010 06/01/2012 06/28/2013 Comments (incl. % Target overachieved. (UNICEF MICS 2013) achievement) Pregnant women who slept under an insecticide-treated net (ITN) the night Indicator 7 : before Value quantitative or 43% 70% 60% 63% Qualitative) Date achieved 07/01/2007 12/01/2010 06/01/2012 06/28/2013 Comments (incl. % Target overachieved. (UNICEF MICS 2013) achievement) Indicator 8 : Pregnant women who received 2 doses of IPT during last pregnancy Value quantitative or 26% 70% 45% 47% Qualitative) Date achieved 07/01/2007 12/01/2010 06/01/2012 03/03/2014 Comments Target overachieved. (UNICEF MICS 2013) (it should be noted the target was (incl. % adjusted downward with AFIII.) achievement) Indicator 9 : Total number of new polio cases reported quarterly Value quantitative or 98  30  20 0 Qualitative) iv Comments Target overachieved. This trend during the last 2 years of monitoring, beyond the (incl. % implementation period, no new cases were reported, by December 2014 - 0 new achievement) cases. Date achieved 05/31/2011 05/31/2011 06/01/2012 03/03/2014 Indicator 10 : Direct project beneficiaries of which female Value (quantitative 0 9 million 25.5 million 24.7 million or Qualitative) Date achieved 07/01/2007 07/01/2007 06/01/2012 03/03/2014 Comments Target partially achieved. Of the 24.7 million beneficiaries 54% were women, (incl. % while the target had been set for 60%. (Implementing Partners data sources.) achievement) (b) Intermediate Outcome Indicator(s) Original Target Actual Value Formally Values (from Achieved at Indicator Baseline Value Revised approval Completion or Target Values documents) Target Years Indicator 1 : Pregnant women enrolled in antenatal care Value (quantitative 63% 75% 90% 87% or Qualitative) Date achieved 07/01/2007 01/01/2010 06/01/2012 03/03/2014 Comments (incl. % Target partially achieved. Target was revised upwards with the last AFIII. achievement) Indicator 2: Health personnel receiving training Value (quantitative 0 2500 13,000 13,640 or Qualitative) Date achieved 07/01/2007 07/01/2007 06/01/2012 03/03/2014 Comments (incl. % Target achieved. (HMIS Data.) achievement) Health facilities constructed, renovated, and/or equipped Indicator 3 : (number) (cumulative) Value (quantitative 0 400 850 818 or Qualitative) Date achieved 07/01/2007 06/01/2007 06/01/2012 03/03/2014 Comments (incl. % Target partially achieved. (HMIS Data) achievement) Children immunized Indicator 4 : Value 250,000 1,400,000 578,000 602,000 (quantitative v or Qualitative) Date achieved 07/01/2007 07/01/2007 06/01/2012 03/03/2014 Comments (incl. % Target achieved. (SNIS Data.) achievement) Percentage of children 6-59 months of age having received Indicator 5 : Vit A Value (quantitative 53% 94% 95% 97% or Qualitative) Date achieved 07/01/2007 01/01/2010 06/01/2012 03/03/2014 Comments (incl. % Target achieved. (HMIS data 2013) achievement) % of provincial and health zones with OPV in stock and without stock Indicator 6: shortage Value (quantitative 98% 95% 99% or Qualitative) Date achieved 05/02/2011 06/01/2012 03/03/2014 Comments (incl. % Target overachieved. National Immunization Program data. achievement) % of provincial and health zones with adequate vaccines storage, cold Indicator 7: chain, and capacity Value (quantitative 60% 98% 90% 98% or Qualitative) Date achieved 03/01/2011 03/01/2011 06/01/2012 03/03/2014 Comments (incl. % Target overachieved. National Immunization Program data. achievement) Indicator 8 : % of households that possess at least one or more bed nets Value (quantitative 9.2% 60% 75% 70% or Qualitative) Date achieved 07/01/2007 07/01/2007 06/01/2012 03/03/2014 Comments Target not achieved. DHS 2013-14 results report 70% of HH possessing a net. (incl. % Worth noting UNICEF evaluation in targeted provinces reporting utilization of achievement) bed nets is high 95%. Under-five children with fever confirmed malaria and treated Indicator 9 : with ACTs in HSRSP zones Value (quantitative 0 60% 80% 53.3% or Qualitative) Date achieved 07/01/2007 12/01/2010 06/01/2012 03/03/2014 vi Comments Target not achieved. (HMIS Data 2013). Target was revised upwards with AFIII (incl. % which encountered immediate delays and resulted in declining utilization of achievement) services. Long-lasting insecticide-treated malaria nets purchased and/or distributed Indicator 10 : (number) Value (quantitative 2.9 million 11.8 million 11.8 million or Qualitative) Date achieved 06/01/2010 01/01/2011 03/03/2014 Comments (incl. % Target achieved. (HMIS data) achievement) % of reports including malaria control activities completed at Indicator 11 : health zone level and transmitted to provincial level Value (quantitative 70% or Qualitative) Date achieved 06/01/2012 Comments There was no reporting on this indicator in the project. However, IP reports (incl. % indicate reporting between these levels took place. achievement) Indicator 12 : % of children between 6-59 months receiving micronutrient supplements Value (quantitative 22% or Qualitative) Date achieved 06/01/2012 Comments There was no reporting on this data. Some activities were started, but at very (incl. % small scale and there are only planning reports, no final implementation reports achievement) available. Number of central and provincial level trained in health Indicator 13 : systems management and fiduciary responsibilities Value (quantitative 400 or Qualitative) Date achieved 06/01/2012 Comments There was no reporting on this indicator. Although some of this training took (incl. % place, there are no reports available. achievement) G. Ratings of Project Performance in ISRs Actual Date ISR No. DO IP Disbursements Archived (USD millions) 1 11/02/2005 Satisfactory Satisfactory 0.00 Moderately 2 04/05/2006 Moderately Satisfactory 0.00 Unsatisfactory 3 05/23/2006 Satisfactory Satisfactory 8.40 vii Moderately Moderately 4 10/06/2006 8.43 Unsatisfactory Unsatisfactory Moderately 5 02/17/2007 Unsatisfactory 8.72 Unsatisfactory Moderately Moderately 6 10/09/2007 15.91 Unsatisfactory Unsatisfactory 7 06/24/2008 Moderately Satisfactory Moderately Satisfactory 19.78 Moderately Moderately 8 12/22/2008 28.25 Unsatisfactory Unsatisfactory Moderately Moderately 9 06/29/2009 52.59 Unsatisfactory Unsatisfactory 10 12/22/2009 Moderately Satisfactory Moderately Satisfactory 72.55 11 06/28/2010 Moderately Satisfactory Highly Satisfactory 94.16 12 12/20/2010 Moderately Satisfactory Moderately Satisfactory 121.79 13 09/14/2011 Moderately Satisfactory Satisfactory 196.62 14 03/24/2012 Moderately Satisfactory Satisfactory 220.34 15 10/24/2012 Moderately Satisfactory Satisfactory 242.30 Moderately 16 04/25/2013 Moderately Satisfactory 261.02 Unsatisfactory 17 12/30/2013 Moderately Satisfactory Satisfactory 299.35 18 06/17/2014 Moderately Satisfactory Moderately Satisfactory 329.49 19 12/22/2014 Moderately Satisfactory Moderately Satisfactory 332.15 H. Restructuring (if any) ISR Ratings at Amount Board Restructuring Disbursed at Restructuring Reason for Restructuring & Approved Restructuring Date(s) Key Changes Made PDO Change DO IP in USD millions 01/15/2010 N MS MS 74.80 Additional Financing 31/05/2011 N MS MS 30.00 Additional Financing 10/18/2012 N MS S 241.12 Additional Financing I. Disbursement Profile viii ix 1. Project Context, Development Objectives and Design In 2006 when the project was being prepared, the health sector system in the Democratic Republic of Congo (DRC) was in serious disarray. There were significant gaps in service delivery, health outcomes were at appalling levels, health system capacity was very week, and the political situation was extremely fragile. In response, the World Bank designed and launched the Health System Rehabilitation Support Project (Projet d’Appui à la Réhabilitation du Secteur de la Sante - PARSS). While some aspects were underestimated overall the design was appropriate. The many implementation challenges the project faced, which are highlighted in this implementation completion report (ICR), were nevertheless overcome through the sheer doggedness and commitment of the Government, the Bank teams and especially the implementing partners and local health staff. The project fully disbursed its resources which had more than doubled since inception. More importantly, most project development objectives (PDO) indicators were achieved and in some cases surpassed. The benefits of this project can be found in the results achieved as well as in the lessons learned of what worked and what did not in a fragile post conflict country context. 1.1 Context at Appraisal 1. In 2005, after a decade of political instability and conflict and many years of mismanagement, the country was making considerable progress towards peace, reconciliation, and economic recovery. A series of political and military agreements were established towards political reconciliation and a Government of national unity was appointed in June 2003 for a 24-30-month period leading to elections. Political progress was accompanied by the reunification of the country. The high level of devastation was reflected in the significant low GDP per capita of US$96 in 2002. With sufficient security, economic activity slowly began to resume. 2. Although there were many encouraging developments, the overall situation remained fragile, challenges ahead were daunting, and the social situation was acute. There was still insecurity in parts of the country, especially in the East; the public debt was at record high and fiscal resources very limited; institutions were in shambles and the country was plagued with governance and fiduciary challenges; public finance management had deteriorated and the administrative capacity of the country was compromised. The majority of Congolese in 2005 lived on the equivalent of US$0.20 per person per day. The social crisis was reflected by the collapse of social services delivery and the erosion of social capital. Infrastructure suffered considerable damage, a key impediment for a large country like DRC. A determining element moving forward was the successful management of the period of political transition leading to elections. 3. DRC had very poor health outcomes. The country had among the highest levels of infant, child, and maternal mortality in the world. One in 10 infants died before their first year and at least one in five children died before reaching five years of age. The maternal mortality ratio (MMR) was one of the highest in the world at 1,298 maternal deaths per 100,000 live births. In addition, there were an estimated 3.8 million “excess” deaths 1 between 1998 and 2005 that were not due to the direct violence but to increased malnutrition and disease stemming from the displacement and socioeconomic disruption caused by the conflict. Diarrhea, respiratory infections, malnutrition and measles were the main causes of child mortality. Malaria was the number one killer of children under five and a significant cause of morbidity and mortality among adults. Ninety Seven percent of the population was at risk of endemic malaria, with drug resistance emerging as a serious concern. The prevalence of adult HIV/AIDS was estimated at 4-5 percent, indicating epidemic levels, and a very large part of the population was at increased risk of the disease, including displaced persons and the many rape victims. The prevalence of child malnutrition (38 percent), particularly wasting (16 percent) was high. Overall, the majority of Congolese consumed less than two-thirds of the daily calories needed to maintain good health in 2005. The large population of 58.3 million was growing fast, at an estimated annual growth of 2.9 percent. With its high fertility and mortality, 48 percent of the population was under the age of 15. Modern contraceptive use was at 4 percent. Based on the above, DRC was highly unlikely to achieve any of the health-related Millennium Development Goals by 2015. 4. The health system had largely collapsed due to lack of funding and support. Even though the total health budget had increased from one percent in 2002 to seven percent in 2004, it remained highly insufficient, serving mainly to make sporadic and very low salary payments to Government health workers, which resulted in a lack of motivation and poor performance. Health care expenditures per capita was very low at US$0.22 in 2002. Health funds were irregular and only partially disbursed, budget execution at approximately 70 percent for salaries and 40 percent for recurrent expenditures. Moreover, public spending for health was largely concentrated in the capital Kinshasa. Skilled health personnel were lacking in numbers (1,500 doctors for the country) and quality deficient. When services were available, utilization was poor due to high user fees. The majority of the population paid most of their health costs directly out of pocket. Supplies of drugs were meager, and inefficiently procured, the supply chain quite fragmented, and outreach and community health centers were very weak. An estimated 70 percent of the population had no access to health care at all, and hospitals and clinics, particularly in rural areas, had been devastated. The capacity of the Ministry of Public Health (MoPH) to carry out its stewardship and oversight function was extremely limited. 5. Yet, opportunities for a positive change in health sector performance did exist. First, the health zone (HZ) model, based on the decentralization and integration of primary care and first-level referral services, remained the basis of the health system. Second, there were strong public-private-partnerships with faith-based organizations (FBO) and international non-governmental organization (NGO)-run facilities integrated in the zonal health system.1 Finally, an increasingly results-focused culture at the MoPH was developing allowing a transition from process- to performance-based contracting between NGOs, health facilities, and health zone (HZ) management teams. 1 Since the 1980s and in view of the collapse of the state, non-state actors, including faith-based organizations, had taken over the delivery of essential health care services through partnership contracts in DRC. 2 6. Given this context, the rationale for the Bank to engage in the rehabilitation of the health sector in DRC was highly justified. The Government’s health strategy aimed to provide a package of essential health services (EHS) at the primary care level, with a particular emphasis on women and children, who were suffering the highest burden of morbidity and mortality. The Government was also keen on strengthening its stewardship role while devolving responsibility for the supervision and management of health activities to the provincial and district levels. World Bank support was in line with the May 2004 Round Table for Health and consensus from donors2 to help rebuild the health system. A clear cartography of donor support by areas and districts was developed by the MoPH’s Planning Department covering specific HZs out of the 515 HZs comprising the health system in DRC. 1.2 Original Project Development Objectives (PDO) and Key Indicators (as approved) 7. The Project Development Objective (PDO) is to ensure that the target population of selected health zones has access to, and use, a well-defined package of quality essential health services. 1.3 Revised PDO (as approved by original approving authority) and Key Indicators, and reasons/justification 8. The PDO was not changed. However, key performance indicators were revised to reflect the recommendations from the March 2009 Mid-Term Review (MTR) and the focus of the Additional Financings (AFs) that were approved in 2010, 2011, and 2012. During the MTR, several indicators were determined to be too specific, insufficiently related to the PDO or not measurable and were subsequently reworded or dropped (please see Annex 2 for all details). Also, the AFs introduced supplementary indicators related to the original project activities and changed the targets of existing indicators, reflecting the scaling-up of efforts. To that effect, new malaria and polio-specific indicators were introduced under AF- I and AF-II, and relevant targets were adjusted in 2011 and 2012 at the time of the 3rd AF. 1.4 Main Beneficiaries 9. The original project’s beneficiaries were meant to include an estimated 9.4 million people living in the 83 (out of 515) selected HZs in the provinces of Equateur, Katanga, Maniema, Bandundu, and Kinshasa. A special focus was on targeting women as well as children under the age of five. In addition, the MoPH and the different levels of the health system were expected to benefit from capacity building activities. 10. The number of project’s beneficiaries increased during the project lifespan. The additional Financing (AF) expanded the geographical scope of the project. AF-I increased malaria control interventions to cover the entire province of Katanga3 and two new 2 Other donors (African Development Bank, Belgium, EU, USAID) supported DRC with humanitarian assistance in conflict-affected areas and targeted provinces other than the Bank selected areas for enhancing the quality and accessibility of health care. 3 The original project only covered two districts in the province of Katanga. 3 provinces, South and North Kivu, to carry out the mass long lasting insecticidal net (LLIN) campaigns scheduled by the National Malaria Program (Programme National de Lutte contre Malaria - PNLP). AFI reached over 20 million people in 153 HZs in these provinces. AF-II supported the nationwide mass polio vaccination campaign and AF-III expanded the delivery of the package of EHS to the original HZs and South and North Kivu. 1.5 Original Components (as approved) 11. The original US$150 million project had the following four components: 12. Component 1 - Expand Access to and Utilization of a Proven Package of Essential Health Services to Selected Districts and Health Zones (US$104.0 million): This component was to support the MoPH to provide a package of EHS4 in the targeted 83 selected HZs in the five selected provinces through the financing of work programs to be implemented by international NGOs as implementing partners (IPs) under performance- based partnership contracts (PPCs) by: (a) making the package of EHS available; (b) improving usage of EHS in terms of costs by: reducing financial barriers to utilization; particularly of maternal and child health services; improving financial management, transparency, and accountability at the facility and HZ levels; creating the exemption programs for the poorest; and eliminating the then much-used practice of “taxation” of lower levels of the system by the intermediary levels; and in terms of quality of delivery of service by enhancing the competency of health personnel; and (c) developing the health system. This was to be achieved by (i) providing technical support and training to HZ health management teams in the areas of program coordination, financial and personnel management, and monitoring and evaluation (M&E), and improving the technical capacity of frontline health workers, including Community Health Workers (CHWs) and health facility personnel in the areas of the service packages to be offered under the project; and (ii) enhancing the public-private partnership between the Recipient and private non-profit service providers through PPCs. 13. Component 2 - Boost Malaria Control Interventions (US$30.0 million): This component was to support the scaling-up of key malaria control interventions in the same districts and HZs as under component 1, supporting the 2002-2006 National Roll Back Malaria (RBM) Strategic Plan by (a) financing and distributing drugs (intermittent preventive treatment –IPT- for pregnant women; artemisinin-based combination therapy - ACT) and long-lasting insecticidal nets (LLINs); carrying out rapid diagnostic tests; carrying out community communication strategies and training of health workers in the HZs; and (b) carrying out operational research and providing technical assistance (TA) to the MoPH and the IPs in the implementation of the malaria-related activities of the project. 4 The package of EHS consisted of: (i) child preventive health interventions; (ii) reproductive and neonatal health services; (iii) integrated management of childhood illnesses (IMCI); and (iv) community-level activities through community health workers (CHWs) (relais communautaires). 4 14. Component 3 - Strengthen Capacity for Oversight and Evidence-Based Management of the Health System (US$12.0 million): This component included: (a) strengthening MoPH’s stewardship and supervisory function and improving its capacity to carry out M&E activities; (b) strengthening the capacity of the provincial medical inspector (PMI) and the district medical inspector (DMI) in the fields of planning, coordination and supervision, M&E, budgeting and financial management (FM) and personnel management through the provision of goods, training, TA and operating costs at the district and provincial levels and remuneration of said PMI and DMI; (b) strengthening the capacities of the MoPH in the fields of public health budget preparation and execution and human resources management by assisting in the development and execution of a national human resource strategy; (c) supporting operational research to better inform strategic interventions in the health sector; and (d) strengthening MoPH’s capacity to improve health systems through training. 15. Component 4 - Project Coordination (US$4.0 million): This component was to ensure project execution, coordination and supervision, including dissemination of results and procurement and financial management and M&E activities associated with the project. 1.6 Revised Components 16. The original component design was not revised. However, the approval of the three AFs increased the scope of activities within the components (see below for details) and expanded the geographical coverage of the original project. Therefore, the total project amount more than doubled from US$150 million to US$335 million (see Table 1 below). In addition, two Health Results Innovation Trust Funds (HRITF) and a Canadian trust fund (TF) were linked to the project and added additional resources for technical studies and assistance. Their role in contributing toward the PDO is discussed in section 3.2 of the ICR. 17. The US$80 million AF-I, approved in December 2010, was prepared to support DRC’s efforts to scale up malaria control interventions. AF-I increased (a) funding for the activities under components 2, 3, and 4, with an emphasis on component 2, and (b) the geographical scope of the Project’s malaria component to provide province-wide mass distribution campaigns in the provinces of Katanga, South Kivu, and North Kivu in addition to the original five provinces targeted by the original project. 18. To support DRC to respond to a severe polio epidemic, a US$30 million AF-II was approved in June 2011. The AF-II was processed under OP/BP 8.0 Rapid Response to Crises and Emergencies. It aimed at supporting DRC’s efforts to control and prevent the recurrence of poliomyelitis outbreaks and to strengthen the routine vaccination program. This AF scaled up (a) activities under component 1 covering immunization services as part of the EHS, and (b) specific polio-related training and institutional strengthening under component 3. Some funds were made available for operational costs under component 4. 19. A third AF, AF-III, in the amount of US$75 million, was approved in June 2012, to continue support in providing a package of EHS at the primary level. The activities fell under components 1, 3 and 4 to cover a scale up of the basic EHS interventions under 5 component 1 with a focus on reproductive health, HIV/AIDS, immunization, and nutrition services, and to build capacity at the provincial level for project implementation and at the central level for the management of donor projects. AF-III also provided additional funding for operational costs under component 4. Table 1. Allocation of Project Costs (US$ million) by Component for the Original Project, AF-I, AF-II, and AF-III and TFs Component Original AF-I AF-II AF-III Total Project 1. Expand Access and Utilization of a Proven 104 27 62.5 193.5 Package of EHS 2. Boost Malaria Control Interventions 30 73.2 103.2 3. Strengthen Capacity for Oversight and Evidence- 12 4.8 2.5 11 30.3 Based Management of the Health System 4.Project Coordination 4 2 0.5 1.5 8.0 Total 150 80 30 75 335 HRTF – Haut Katanga PBF Pilot 0.75 HRITF – PBF scheme 10.0 Canadian Development TF – total BE+RE 3.5 Note: Original operation: approved in October 2005, effective in April 2006; AF-I: approved in December 2010; effective in May 2011; AF-II: approved in June 2011, effective in October 2011; AF-III: approved in June 2012, effective in November 2012; Haut Katanga PBF pilot: approved in February 2010, effective in March 2012; PBF scheme: approved and effective in June 2014; Canadian TF: approved in August 2011, effective in December 2011. 1.7 Other significant changes 20. The closing date of the project was extended four times for a total of 4.5 years. Under the April 2010 restructuring, the closing date was extended by 18 months, from June 30, 2010 to December 31, 2011 to make up for start-up delays. In order to accommodate the new activities under the first AF, a second 18-month extension of the closing date was granted from December 31, 2011 to June 30, 2013. Furthermore, at the time of the approval of the AF-III, the closing date of the AF-III was again extended until June 30, 2014. Finally, a six-month extension until December 31, 2014 was granted to allow for the achievement of the planned activities given the performance-based financing (PBF) scale up. 21. Key institutional and organizational changes occurred over the lifetime of the project. They included: (i) the change in fiduciary responsibility from the state fiduciary agency BCECO (Bureau Central de Coordination) housed in the Ministry of Finance (MoF) to the project coordination unit (PCU) at MOPH; (ii) the change in the responsibilities and level of autonomy of the IPs; (iii) the increasing responsibility of the MoPH in managing the project; (iv) the increased reliance on UN agencies to procure and implement some key activities: and (v) the change in approach from performance-based contracting at the IP level to performance-based financing at the provider level. 6 2. Key Factors Affecting Implementation and Outcomes 2.1 Project Preparation, Design and Quality at Entry 22. The project was prepared in a participatory manner with the Government strategy, policy and commitment deemed appropriate. The project was developed following the May 2004 Round Table for health, the Government’s presentation of the Minimum Partnership Program for Transition and Recovery in November 2004 as well as the FY05 Bank’s Health Country Status Report. The project preparation also benefited from coordination with and inputs by development partners. 23. Given the very challenging post-conflict context, the project was largely designed around experience learned in the implementation of Bank operations in DRC5 as well as in post-conflict countries. The project design included key lessons on the need for: (a) a large role for NGOs in support to service delivery to overcome capacity constraints while building the government capacity to coordinate and manage the sector as well as the capacity of the front-line service provision system; (b) a strong PCU at the central level and strengthened health authorities at the local level to effectively manage partnership contracts and ensure MoPH’s ownership at all levels; (c) selected IPs to be provided with flexibility to implement their work programs to the local circumstances based on performance contracts with built-in incentives to shift focus from inputs to results; and (d) effective and timely M&E of IP performance. 24. The components of the project were well selected. There was a strong focus of the primary health care level based on the HZ model and support was provided to the delivery of a minimum package of EHS for component 1, the focus on malaria control for component 2 (since malaria was the largest burden of morbidity among women and children), and the strengthening of the capacity of the health system for improved oversight and evidence-based management as well as a component on project management were appropriate. 25. The implementation arrangements for the project were justified given the weak Government’s capacity and dysfunctional health sector. The main feature was the outsourcing of the project implementers and the monitoring and evaluation (M&E) and fiduciary control functions. The division of responsibility was as follows: the MoPH was responsible for the oversight and management of the project; an independent PCU, within the MoPH Secretary General’s (SG) office, was in charge of coordination between the MoPH and the service providers; and the following service providers: (a) the State procurement agent, BCECO6, responsible for managing the fiduciary aspects of the project since there was no existing good procurement capacity in the MoPH-PCU; (b) given the vastness of the country, experienced NGOs as well as IPs contracted on a performance basis (with financial bonuses) to implement the project activities at the district level; (c) an 5 Especially the three- year implementation experience of the health component of the EMRRP project. The EMRRP health component in the amount of US$44 million supported the implementation of the same essential package of health services in 67 health zones. 6 BCECO was already responsible for WB financed projects (across sectors) to manage the fiduciary aspects of projects. It has overall responsibility for approving procurement, authorizing payment and managing the special account. 7 independent external evaluation agency (EEA) was used to monitor and evaluate IPs’ activities; and (d) externally recruited financial auditing and procurement auditing firms were used. 26. The risks were underestimated at the design stage and the readiness to implement the project was overestimated. While the selection of HZs to be targeted under the project was a result of a geographic mapping of donors done by the Government, the project ended up taking on the most difficult and remote zones with some of the worst health indicators. This was compounded by extremely difficult access to these zones and very limited transportation means. In addition, the anticipated coordination between the different executing and implementing actors was overly optimistic and risks were not thoroughly assessed. The technical, fiduciary and managerial implementation seemed to have been taken for granted based on past experience; no thorough assessment of the situation of the ground seems to have been carried out; no in-depth assessment of the governance and political economy challenges for project implementation was undertaken; and some key sector risks (disseminated health facilities, low salaries, low motivation of health workers, cost recovery issue, low demand for services) that may likely have affected the project execution were only partly identified. Lastly, the Result Framework was incomplete with the provision of the baseline and targets for utilization indicators left to the EEA once recruited and procedural manuals for project execution were not finalized. 2.2 Implementation 27. The project was implemented in a very challenging political, security, capacity and governance context and faced complex operational challenges over a period of more than nine years which included 3 AFs, making it one of the largest and complex projects in the DRC portfolio. The implementation period can be split into two separate periods (pre- and post-restructuring), each representing and explaining key factors behind implementation progress or lack of progress and changes made during the course of the life of the project. A. From project approval until the April 2010 restructuring 28. The project became effective more than 8 months after approval. Effectiveness was delayed from December 2005 to April 2006 due to the turbulent political climate in the country in the wake of the first elections in 40 years and the inexperienced project management. All conditions of effectiveness (establishment of a financial and accounting system and selection of independent auditors) were met on February 28, 2006 but the project could not be declared effective until the project was ratified and the legal opinion issued. 29. The project experienced a very difficult and long start-up period. Project implementation suffered from (a) the difficult political context of 2006 with the contested July 2006 presidential election results and post elections violence which led to the organization of new elections in October 2006, and (b) persistent problems related to procurement and fiduciary management. The project relied on three major procurement 8 items (contracts for the nine IPs to implement project activities, the EEA, and the PNLP) which comprised the bulk of the project and were managed centrally due to the governance challenges faced by the country. Consequently BCECO was overwhelmed with managing numerous projects and ended up performing very poorly, especially in handling complex procurement for the service providers, leading to long delays in procurement procedures, serious procedural mishaps on several contracts and inability to complete some important tenders essential to the project’s infrastructure (i.e. FM and internal auditor firm, external procurement audit firm, and the TA firm/NGO to support the provincial and districts level). The Bank also contributed to the delays, with long procedures for providing non-objections for contracting the IPs, as overhead costs were considered very high. Finally, payments cleared by IDA for purchases by the PCU and IPs were delayed by BCECO. As a consequence, the implementation rating (IP rating) was downgraded to unsatisfactory range. 30. Once the IP contracts were finalized, implementation at the local level started, albeit still at a very slow pace. Over a year and a half after effectiveness, contracts for seven out of the nine IPs were finally signed and implementation started in October 2007 through the establishment of field offices. By mid-2008, all the IPs had started their activities, including planning, training health personnel, providing incentives (primes) to health personnel, supporting health services through the distribution of inputs to health facilities, and carrying out mass-campaigns. 31. The start-up period remained very slow due to (a) continued security challenges, (b) loss of original IP staff due to the long delays in signing contracts and the subsequent need to recruit new employees; (c) continued delays in initial procurement of equipment and vehicles which constrained IPs’ operations; and (d) many problems with transporiont due to very poor infrastructure and costly and dangerous travel by air and water. Bottlenecks in project management and implementation continued to hamper project execution throughout 2008. These were due to: (a) cumbersome approach for the management of IP contracts. The contracts requirements were burdensome and bureaucratic requiring centralized procurement of key inputs which slowed down the operations of the IPs and led to serious delays in getting the needed inputs (drug, vehicles, etc.) to HZs. In addition, the numerous IP activities which required non-objections by the PCU impacted the autonomy and flexibility in planning, budgeting and procurement7 for IPs and impeded their ability to supervise implementation; (b) unclear institutional arrangements between the PCU and BCECO: the division of fiduciary responsibility was diffused, leading to continued delays in procurement and reimbursement of IPs and complicating Government oversight of IP contracts which ended up focusing on process rather than performance as envisaged by the project design; (c) a PCU operating as a standalone unit: there was very little coordination collaboration and accountability between the PCU and the ministry’s Secretary General and technical units; (d) poor relations between the PCU and CREDES, 7 Initially the IPs were responsible for the procurement of drugs and equipment. However, delays were incurred that were compounded by the very limited availability of firms to offer the required inputs at the local level. 9 the EEA following the very long delay in signing the contract, the EEA immediately ran into difficulties resulting in delays in carrying out the baseline survey due to misunderstandings between the PCU and the Public Health Institute of the MoPH on an issue of non-compliance with IDA procedures to procure goods and services for its own needs. The baseline data was finally available mid-year and the end of project targets in November 2008; (d) geographic inaccessibility and the difficult security situation in some areas exacerbated the implementation challenges and delayed distribution of inputs. Component 3, which focused on building capacity at the intermediary level and among provincial and district medical inspectors, was also seriously delayed in contracting a firm. Given the substantial delays and lack of reliable data to assess progress, the PDO and IP ratings should not have been upgraded to moderately satisfactory in the June 2008 Implementation Status Report (ISR). These ratings were downgraded back to moderately unsatisfactory in the two subsequent ISRs. 32. A Quality Assessment of the Lending Portfolio (QALP) was carried out in May 2008 as part of the Africa Portfolio Review of risky projects. The QALP January 2009 final report rated the project’s quality of design and implementation as unsatisfactory due to a weak Results Framework poor quality, and an inadequate risk assessment of the, Government’s support for the project, and implementation delays. The QALP determined that the prospect of achieving the PDO was unlikely and recommended that the project be restructured at the Mid-term Review to remove BCECO from the project, increase IPs’ autonomy and flexibility to manage their project activities; and revise the scope of the essential health package. 33. The MTR highlighted many of the persistent problems plaguing the project. By the time of the May 2009 MTR, only 28 percent of total project funds had been disbursed, mostly from the UNICEF contract under component 2. The key problems included: (a) limited governance capacity for project management; (b) weak appropriation of the project objective at all levels of the health system; (c) PCU operating very much in parallel with the MoPH and not much accountable to the ministry; (d) lack of clarity about responsibilities between the PCU and IPs; (e) weak implementation and management capacity at the central and local levels; and (f) an incomplete RF. The MTR recommended to: (a) continue with the project design but strengthen capacity in management, procurement, supervision, and M&E; (b) transfer fiduciary responsibility from BCECO to the PCU; (c) revise the IP contracts to allow for a more performance-based focus; (d) improve the specificity of indicators and targets; and (e) extend the closing date by 18 months from June 30, 2010, to December 31, 2011 to compensate for the long start-up delays. 34. Project implementation improved substantially over the course of 2009. Factors contributing to the improved performance included: (a) revised IP contracts with clarified procurement plans and procedures between the PCU and IPs; (b) improved management and implementation of the IP contracts; and (c) improved cooperation between PCU and BCECO allowing key centrally procured inputs to be delivered to the project districts. Also, while the contract with CREDES and the relationship between the PCU and CREDES remained problematic, the division of responsibility for M&E for the project was clarified 10 in order to rely on IPs for timely routine monitoring while a household survey would be done by the CREDES to assess the overall performance of the project. Also, although the consultancy firm (CEMUBAC) was in place for the support to district and provincial health administration under Component 3, effective support had not yet started. The improvement in implementation led to the upgrading of the PDO and IP ratings to marginally satisfactory. B. From the April 2010 restructuring until the completion of the project 35. The April 2010 restructuring led to a significant improvement of project performance. After the MTR and the QALP recommendations the project was restructured. The entire process took more than 18 months due to the multiple complexities and issues that needed discussions and decisions at various levels. With the project restructuring, fiduciary management was transferred from BCECO to the PCU now that the key contracts were finalized. This was accompanied by seconded staff from BCECO with a mandate to continue to build capacity within the MoPH. The implementation capacity of the PCU improved markedly. IP contracts were further revised to simplify procurement and FM procedures and to clarify tasks, indicators and targets. Improved communication between the PCU and the MoPH was confirmed by the introduction of monthly review meetings chaired by the SG to strengthen the MoPH oversight. IPs accelerated project activities and the capacity-strengthening activities in health management at the provincial and district levels resulted in improved supervision and trust at the local level. The data collection and timely reporting also improved and a common template to facilitate the quarterly reporting of IPs was developed. However, disagreement persisted on the facility and households surveys delaying their completion. Finally, the constraints with the EEA contract were addressed with the replacement of the team leader and revision of the terms of reference. Disbursements picked up, completion of large contracts led to a significant acceleration of disbursements, and by the revised closing date of December 2011, all funds from the original grant were disbursed and as result, the DO and IP ratings were upgraded to the satisfactory range. 36. Three AFs were approved in a period of 18 months between December 2010 and June 2012. These AFs were prepared at the request of the Government and responded to clear and specific health and financing needs to support the overall health system in DRC. AF-I was prepared to respond to a resource gap for the Government’s strategy of LLIN coverage and mass campaigns as DRC was a key priority country under the Malaria Booster Program; AF-II was prepared to respond to a wild poliovirus outbreak with a high risk of reaching epidemic proportions not only in DRC but also in neighboring countries; and AF-III was requested to continue support to primary health care services to avoid a financing gap until new funding from development partners was made available. The proposed activities under the AFs fell under the components of the original project and were consistent with contributing to impacting on maternal and child morbidity and mortality. They also continued to support training and development of institutional capacity at central and provincial levels. AF-III activities rightly focused on reproductive health/family planning (FP), nutrition, HIV/AIDs, and immunization. The AF instrument reflected the Bank’s management preference at the time to prepare AF rather than add new projects to the portfolio given the difficult country context. 11 37. The first two AFs8 were approved in December 2010 and June 2011 and a third one was in preparation for approval by end 2011. AF-I became effective in May 2011 after some delay to obtain Government ratification and AF-II became effective in October 2011. To remedy the start-up delays due to the pre-elections context, a US$5 million project preparation advance was approved for AF-II to allow timely support to vaccination campaigns. The new activities and geographical scope of interventions for the AFs impacted heavily on the workload of all project implementers while their capacity was still limited and necessitated the recruitment of additional PCU fiduciary staff. Key activities under the two AFs were contracted to UN agencies.9 The November/December 2011 events, when elections led to civil strife, geographic inaccessibility as well as insecurity in some target provinces contributed to delays in the distribution of bednets until April 2012. After a slow start, implementation of AF-II went as planned due to support of the technical leadership of the National Immunization Program, one of the best organized structures under the MoPH, which had subunits in all the provinces. 38. Meanwhile, lack of funding between December 2011 and July 2012 impacted the delivery of EHS. During the period between the final disbursement of the original project funds and the approval of AF-III funds were depleted. Therefore, activities were limited to the bare essentials. Salary top-ups or “primes” were not paid out to health facilities and the delivery of drugs and equipment was delayed. AF-I and AF-II pre-financed activities to maintain the minimum level of EHS. AF-I reallocated some of its funds toward financing some of the activities under component 1; it was then restructured as a level 2 to ensure retroactive financing of the activities to continue the delivery of the essential health package. The delay in Board approval and effectiveness of AF-III was due to the political situation, a countrywide six-month suspension of disbursement because of governance issues that were beyond the control of the project, and delays in obtaining the legal opinion. An anti-fraud anti-corruption plan with performance measures, verification mechanism and clear responsibility for implementation was developed to strengthen governance mechanisms and dialogue among partners as well as to improve FM, accounting and procurement mechanisms and to ensure proactive disclosure of project related information. AF-III became effective in November 2012. 39. With AF-III10, approved in June 2011, a full PBF approach for EHS delivery was put in place and the role of the IPs changed from being the main implementers to that of capacity builders and results verifiers. Over time, as the capacity of the MoPH province- and district-level staff grew, MoPH decided that staff was ready to take on responsibility for complete implementation. Other considerations in the move to MoPH implementation were the significant overhead cost of contracting IPs and the time- consuming procurement process. Additionally, some of the IPs indicated that they had no interest in continuing to support the project. As a result, implementation arrangements 8 The closing date of the project was extended from December 31, 2011 to June 30, 2013. 9 UNICEF was contracted to purchase, distribute, monitor, and report on the ITNs and to carry out the mass campaigns. AF-II financed temporary staff for mass vaccination and social mobilization activities through a contract with UNICEF. WHO was contracted to provide technical expertise and played an important role in reporting and disease surveillance through its network of provincial officers. 10 A final extension of the closing date to December 31, 2014 to make up for previous delays was granted. 12 shifted: provincial-level PCUs were established11 to take over the role of the IPs in contracting suppliers and paying incentives (primes) in the HZs and work with the district health administrations. With the transfer of responsibility for implementation to the local MoPH affiliates, the role of the remaining IPs diminished and evolved from service providers to performance purchasing agencies, responsible only for verification of services provided and results. 40. After a delayed launch of AF-III, steady progress was made in implementing the PBF scale up12 to all the HZs. The effective launch of AF-III took six months and delays occurred with the signature of key contracts pertaining to the implementation of EHS at HZ levels. By end 2013, significant progress were achieved on all key contracts (purchase of essential drugs, inputs for FP and nutrition, recruitment of TA firm to be deployed at district levels, new EAA, Memorandum of Understanding with UNFPA). UNFPA was contracted to conduct a number of gender-based violence (GBV) interventions and to purchase FP commodities. After a long delay due to contract disagreements with the Bank, UNFPA purchased the commodities, however, distribution was not clearly defined in the contract and the commodities remained in customs at the time of the closing of the project in December 2014. UNICEF implemented micro-nutrition interventions. Since the signing of the two performance purchasing agencies contracts, the implementation of the PBF program progressed steadily and PBF contracts were signed with the health facilities. Other project activities were implemented once needs assessments for nutrition inputs, family planning activities, ARV, and the strengthening of regional drug distribution centers were finalized. Data on new indicators for AF-I and AF-II was not available caused by the delay in launching and delivering the Demographic Household Survey (DHS) as well as the UNICEF impact evaluation on bednet distribution. A final extension of the closing date was granted on July 24, 2014, with a new Closing Date of December 31, 2014 to make up for previous delays. 41. Despite the very difficult beginning and constraints during the initial project implementation, after restructuring, the project ended up being fully disbursed and all planned activities were implemented. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 42. Design: Key indicators were identified to measure improvement in geographical access, reduction of financial barriers to utilization and enhancement of quality of care. The decision to rely on an EEA was the right choice at a time when the MoPH’s capacity to monitor and evaluate was very weak. The causal links, or results chain, between the project’s inputs, activities, outputs, and intended outcomes seemed reasonable. Baseline, MTR, and end-line surveys were planned accordingly. However, while the PDO clearly 11 Each provincial-level PCU comprised a field coordinator with as small administration team while recruiting two NGOs/IPs as service purchasing agent regrouping the 10 districts into two pools, the West and the East regions. Eight field coordinators were recruited. Many of the former IP staff were recruited as individual consultants to manage the provincial-level PCUs. 12 An RBF pilot in Haud Katanga was introduced in 2009. The RBF approach was not new for DRC, as many RBF schemes had been taking place at the country level since 1996 and the EU project implemented an RBF approach in their selected health zones. The IE of the RBF pilot informed the design of the PBF scheme. 13 stated two main areas of achievement, access to and utilization of quality services, and although several of the intermediate indicators measure access, few PDO indicators directly measured improved access to services although there are a number of intermediate outcome indicators which measure access. The assisted deliveries were likely intended to be the access indicator at appraisal (percent of women who receive delivery care from a trained provider), however, it was reworded to “percent of assisted deliveries” during project restructuring and became a better indicator for quality of services and utilization rather than access. The many changes in indicators and targets over the course of the restructuring and AFs added much complexity to monitoring and evaluating project progress. The project also comprised a component to help strengthen the capacity of Government officials at all levels in planning, monitoring and evidence-based management and included intermediate indicators to measure progress. 43. Implementation: The baseline data for the project only became available at the MTR due to the long delays in contracting the EEA. The quality of the data was questionable and often resulted in disagreements between the EEA and IPs on achievements. Project management relied on the EEA’s reports but also used project data, the HMIS, and other studies and reports conducted by partner organizations. Collaborative efforts by the donor partners to finance a DHS around the end of project implementation were made despite delays in the launch of the DHS as planned in 2012. Data were collected in 2013 and results made available in October 2014. Other key studies were delayed (UNICEF impact evaluation report, EEA household survey and final evaluation reports on malaria activities. These delays resulted in a lack of reporting on impacts until quite some time after the original project activities had been implemented. Over time, more responsibility for project data collection was given to the IPs, especially after the restructuring, this improved the timeliness and quality of the data reporting. Despite continuing management issues between the PCU and EEA, the work was evaluated by CREDES during supervisions and determined to be of satisfactory quality, and it provided reliable data that was later confirmed by findings from the 2014 DHS. 44. A significant number of indicators were revised at the MTR to accommodate the added focus and targets were adjusted to more ambitious levels. Additionally, a large number of intermediate indicators were dropped from the initial list (please see Annex 2 for details). The link between the PDO and the PDO indicators after all the changes during the AFs and restructuring could have benefited from more careful revision and better consideration for more realism with regards to ambitious targets. Despite that, the majority of indicators met or exceeded targets although 3 intermediate outcome indicators had no reporting and therefore cannot be evaluated by this ICR. 45. Utilization: During the first years of implementation, the delays in obtaining monitoring data did not inform or influence decision making and/or adjustment. However, reporting from the IPs and project data were used extensively in the yearly health forums, which were designed to take actionable decisions with regard to project activities based on evidence from the field. At the same time, the AF-I was the result of data analysis and a demonstration of the impact of the interventions. Studies by the Kinshasa School of Public Health with UNICEF and WHO demonstrated that the malaria control interventions were 14 having very positive impact, resulting in a decision to very rapidly scale up the malaria interventions. 2.4 Safeguard and Fiduciary Compliance 46. Social and environmental safeguards: There were no reported social or safeguard issues during the implementation of the project, and the rating has been consistently satisfactory. The original project and the subsequent AFs were classified B (partial assessment). The environmental issue related to the safe handling and disposal of medical waste was generated from the minimum rehabilitation of the health facilities. These were addressed in the Medical Waste Management National Plan (MWMNP) (Report E844- December 2003), which was prepared in the context of the Bank-funded HIV/AIDS project. The MWMNP was made available at public sites accessible to all project-affected facilities and communities. In May 2011, a comprehensive environmental audit was conducted and concluded that the environmental impact of the renovation of the health facilities conducted under the project was minor. A higher risk was posed to the environment from the placement of equipment such as waste-burning plants and latrines. The report made recommendations on reducing those risks, such as taking into account wind direction, other construction, and other types of medical waste disposal. 47. Indigenous people safeguard (OP 4.10): The safeguard was triggered by the AF-I in recognition that though the Batwa people (indigenous pygmy population) are located in the targeted HZs, the Strategy Framework for the Indigenous People Development Programs, supported by the World Bank and the DRC Government, started only in 2009. Therefore, terms of reference (TORs) for an Indigenous People Planning Framework (IPPF) were prepared and its financing included under the AF-I. The IPPF was prepared and disclosed in December 2, 2010 and September 22, 2011. In 2011, an indigenous people audit was carried out and found that project activities had had a positive impact on the indigenous people. No negative effects were reported. Access to health services increased from 40 to 65 percent in the areas where indigenous people reside. Moreover, prenatal care, hospital care, and malaria control all saw improvements. 48. Procurement performance: Overall procurement performance was poor. Although the decision to contract BCECO as the fiduciary agent at the time of preparation was the right one, early on during implementation it became clear that the arrangement was not working as expected. BCECO, the fiduciary agency for numerous other projects, did not have adequate time for PARSS. The problems in procurement led to an almost two-year start- up delay, and major disruptions in implementation due to procurement-related delays continued during project implementation. The reasons for the poor performance were discussed above. Unfortunately, given the challenging political economy and governance context in DRC, the procurement management changes made no substantial difference in the end, as the official in charge at BCECO during the first years of the project was seconded from BCECO to the PCU. Later, when the BCECO contract ended and its oversight role was terminated, the same person was hired by the PCU to lead procurement. At various times, the quality and performance of the procurement process were unsatisfactory. 15 49. Financial Management: At the closure of the original project and after the first two AFs, an in-depth financial management review took place, US$471,093.38 in ineligible expenses were identified at the time of the audit, while the financial management supervision mission identified another US$117,371.75 due largely to further review of the reports. After an in-depth review, the total amount was found to be closer to US$150,000, which was communicated and paid back by Government to the Bank. One important lesson that emerged from this issue was the need for more regular and frequent supervision reviews. Although during most of the project’s life financial management was rated moderately satisfactory, during the last supervision mission, the project team downgraded the rating to moderately unsatisfactory. This was due to a US$6 million financing gap13 at closing that resulted mostly from the project’s over-commitment in the procurement of essential drugs and its underestimation of the cost of their distribution. Project management has not been efficient in administering the logistics of drugs, and issues have persisted up until the time of the writing of this ICR. 2.5 Post-completion Operation/Next Phase 50. Continuing services and avoiding interruptions in the provision of health services to the population in DRC has been at the forefront of project management concerns. This resulted in the AF-III and the new health system strengthening project approved by the Bank on December 18, 2014. In its last six months of implementation, the project financed preparatory activities and used funds to avoid serious disruptions until the next operation is effective. 51. Based on lessons learned from the project in terms of scope, implementation arrangements, fiduciary practices, and interventions, three new operations have been prepared and approved in order to assist the country in sustaining reforms and institutional capacity: (i) the US$220 million Health Systems Strengthening for Better Maternal and Child Health Outcomes (PDSS) approved on December 18, 2014 scales up the geographical coverage of the project to include an additional 60 HZs in the provinces of Equateur, Bandundu, and Katanga. Increased attention is being given to building MoPH’s capacity in line with the ministry’s increased level of responsibility in managing the health services. The PDSS is being implemented within the MoPH ensuring in-country system use and capacity building. The PDSS also offers a platform for technical and financial partner alignment14; (ii) the US$107 million Great Lakes Emergency Sexual and Gender-Based Violence and Women’s Health Project (of which US$73.86 million is allocated for DRC), approved by the Bank on June 26, 2014, aims to provide integrated health and counseling services, legal aid, and economic opportunities to survivors of sexual and gender-based violence and to strengthen health services for poor and vulnerable women in Africa’s Great Lakes Region. It is the first World Bank project in the Africa region with a major focus on addressing the needs of survivors of sexual and gender-based violence; and (iii) the US$15 million Human Development Systems Strengthening Project 13 It was agreed that the financing gap be financed under the new project. 14 A Memorandum of Understanding was signed with the Global Alliance for Vaccines and Immunization (GAVI), the Global Fund, and UNICEF to ensure complementarity between the key strategic partners, which will continue to provide the inputs required while the PDSS finances the outputs. 16 (P145965) approved on April 23, 2014. Includes a component on improving the HMIS and strengthening the pharmaceutical sector, particularly FEDECAMES. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation Relevance of Objectives 52. The project objectives were Highly relevant and fully consistent with the Government and Bank’s strategic approaches for the health sector at the time of its preparation. The PDO was aligned with the objective of the May 2002 Interim Poverty Reduction Strategy, to enable the poor to have access to primary health care and the Government’s strategy (Minimum Partnership Program for Transition and Recovery) to increase access to the package of essential primary and first referral health services as well as the National Roll Back Malaria Strategic Plan for the 2002-2006 period. Social development was one of the strategic elements of the Bank’s 2004 Transitional Support Strategy (TSS) which emphasized recommended additional resources for health activities through both advisory and investment operations. The project objective was also highly relevant to the Bank’s HNP strategy, its malaria booster strategy and the Africa HNP strategic priorities. 53. The project objectives remained Highly relevant throughout the project life. Despite considerable improvements over the past decade, as witnessed by a significant decline in child mortality rates, overall health outcomes in DRC remain among the worst in the world. The PDO remained fully relevant to the 2011-2015 National Health Development Plan which identified as a key pillar the support to the development of the HZs to deliver basic health services. Pillar three of the 2006 Poverty Reduction Strategy Paper on improving access to social services and reducing vulnerability and the Strategic element 3 of the FY08-11 country assistance strategy (CAS) intended to help improve the provision of social services and reduce vulnerability. The FY10-16 CAS also recognizes the continued need for high-impact support in the health sector. 54. The relevance of the project development objective is rated as High, both before and after restructuring and the three AFs. Relevance of Design and Implementation 55. The relevance of the project design was and remained consistent with the project objectives and with the needs of the country. The design was based on available knowledge of the key causes of morbidity and mortality in the country (i.e. Bank Country Health Status Report, 2002-2006 National Roll Back Malaria Strategic Plan). Analysis of the health situation clearly highlighted the high burden of morbidity and mortality from preventable and treatable diseases at primary and first referral levels. Therefore, the focus on basic health services was highly appropriate. In addition, the emphasis on decentralized service delivery through HZs with well-defined structures and faith-based organizations 17 and NGOs as key IPs was very relevant given the decentralization approach of the Government and the limited capacity of the Government. Given the precarious state of child and maternal health in DRC, the EHS package, which included an integrated set of maternal and child health services, was deemed sound and consistent with international practices and targeted the most pressing health problems. It was also already utilized in several HZs supported by donors. The provision of the EHS package, with the additional services added over the ensuing years, remains the most important basic health service in line with the burden of disease in DRC. Malaria control, included then as a specific component because it was the largest burden of morbidity among women and children, continues today the highest burden of disease and contributor to mortality. The project activities were highly relevant and responded to the objectives of the project. The project opted for the recruitment of an EEA and ensured that the project coordination component had adequate funding for M&E due to the inability in DRC to obtain and analyze reliable data in a timely manner. Last, the choice of target HZs was well coordinated and the project supported the most deprived districts with no adequate support from other donors. 56. While the project design was responsive to the country context, some elements of it could have been further assessed. At the time of project preparation, the health system had all but collapsed, and what was remained was there due largely to community-based and religious organizations. Therefore, the reliance on third-party implementers for service delivery at the community level and for logistical support was the right decision. At the same time, given that this was an ambitious project targeted at the country’s most challenging regions in a very difficult environment, a more comprehensive capacity assessment to allow for more effective capacity building at all levels should have been conducted at the outset. Also, a more comprehensive performance assessment of the IPs should have been conducted, as IP performance in practice was variable, not least because the HZs were selected in some of the most challenging regions in the country. Also, while the first two AFs responded to clear needs, they overloaded the PCU as well as IPs adding complexity to an already challenging project implementation. Although AF-III was prepared in response to clear demand from the Government, it could be argued that this should have been a new project rather than a bridging of financing and capacity gaps for many ongoing interventions until a new heath project was prepared. The AF instrument reflected the Bank’s management preference at the time to prepare AFs rather than add new projects to the portfolio. 57. Introducing Performance Based Financing (PBF) as an output strategy vs. an input to ensure results. This decision was based on the positive results achieved in the African region where PBF was introduced as a strategy to motivate and strengthen the delivery of health services, and given the results in the neighboring countries. While not all the intended results were achieved due to some design and some implementation issues in a PBF pilot15 conducted and evaluated in DRC, it was deemed the right approach given the need to empower the HZ to better achieve results and be able to deliver the EHS in a 15 Between 2009 and 2013, in line with the Government health sector strategy of promoting contractual approaches, a trust-funded PBF pilot was implemented in Haut-Katanga. An impact evaluation was conducted and lessons learned documented. 18 quality manner. Lessons learned from that pilot as well as from other country implementation led to the introduction of PBF in all 83 HZ in 2013. 58. The Bank’s implementation assistance was largely responsive to the changing needs and the operation remained important to achieving the country’s objectives. Although the project was prepared in what was officially a post-conflict situation, security issues remained in certain areas, and regular election-related and other violence continued during the life of the project and contributed to delays and changes in government. The reliance and close integration of local private non-profit service providers (faith-based organizations), partners (IPs) and UN agencies to provide services and to build capacity, especially at intermediate level (districts and provinces) and to front line service providers, and the establishment of performance-based contract between the IPs and public and private service providers at the decentralized levels in the early years were in line with the prevailing weak capacity of the MoPH. The Bank also relied on donor partners to undertake capacity building of the MoPH. At the time, given the demonstrated capacity of the NGOs in DRC, it was a sensible decision to contract them to implement health services. In fact at the time the only health services still being provided in the project selected areas were by Faith Based Organizations (FBO). Using NGOs/FBOs proved to be a key factor in ensuring that the interventions by in large continued during the political crisis. To date, FBOs manage over 35 percent of all HZs in DRC.16 This led to increased government responsibility in the later years of the project with the move to the PBF approach. Also, the coordination of the project was reinforced and particular attention was paid to strengthening donor planning and coordination at the district level throughout the project. The risk rating at the design stage was high due to the potential recurrence of conflict and continued weaknesses in governance and fiscal management. The risk was and continued to remain significant at times it turned out to be high since security was an issue in some regions and uncertainties during the election period were substantial. Despite major political upheavals disruption and delays, the project fully disbursed. 59. The relevance of the project design and implementation is rated as Substantial both before and after restructuring and after the three AFs. 60. Given the High relevance of objectives and Substantial relevance of design, the overall relevance of the project is rated as Substantial. 3.2 Achievement of Project Development Objectives 61. Overall efficacy is rated as Substantial. This is being assessed against the two main objectives of improving utilization and access. The achievement of utilization improvements is measured as the percentage of women and children sleeping under bed nets and the share of women receiving IPT for which the expected targets were overachieved. The fourth indicator of utilization, new cases of polio, was also overachieved as no new cases of polio were reported since the interventions were started. The malaria control and polio interventions in particular were highly successful, thanks to the scaling- 16 World Bank Health PER 2015. 19 up of a recognized effective approach to malaria control and contracting UNICEF for implementation. An evaluation by the Kinshasa School of Public Health on the impact of the successfully implemented malaria activities under the PARSS demonstrated an increase in ownership and utilization of ITNs and a concomitant reduction in fever episodes. Studies carried out in 2013 demonstrated a reduction in morbidity and mortality from reduced malaria fevers among the targeted population (UNICEF 2013). The evaluation also confirmed the success of the LLIN campaigns in obtaining rapid results, leading to the Government’s request to expand activities to the provinces of Katanga and South and North Kivu to support its aim of universal LLIN coverage. Thanks to the project, over 11 million LLINs were distributed to millions of households, and more than half of all women and children under five in the targeted areas now sleep under bed nets. 62. Another key indicator of utilization is the annual curative consultation rate which was partially met. This assessment needs some nuancing as it was close to its target at the end of the implementation period of the original project. The last additional financing adjusted the target upward, however its effectiveness was delayed and as explained in the implementation section, utilization rates suffered. Utilization rates almost doubled between 2007 and 2011 (the original closing date) from 0.19 to 0.37 visits per person per year, due to price-setting reforms and increased public resources for the health sector in project areas. The final target was increased to 0.45 per person per year with the AF-III; it took a long time for AF-III to become effective, however, which resulted in a financing dip and a subsequent reduction in the utilization rate to 0.37 in 2014. Overall, the achievement against the revised target is considered as partially achieved. 63. A similar phenomenon as explained for the utilization rates indicator occurred with the family planning rates, which had increased from 2 to 6 percent by 2011, but was not achieved at the end of the project when the rate dipped back to 2 percent. The non-availability of funding led to a dip in the availability of contraception commodities. Family planning is the only indicator the project did not achieve, even though the targets were already set very low. Achieving improvements in family planning is very difficult, if not impossible, if health facilities do not have the contraceptives in stock and suffer from frequent stock-outs in supply. The very slow start up and continued procurement and distribution problems with FP commodities hampered the family planning component throughout the entire period of project implementation. Even in the closing month of the project, the FP commodities had been sitting for over 9 months in the port of Kinshasa, due to customs issues and lack of clarity about responsibilities and transport costs. 64. The immunization indicators, that is, children 0–11 months vaccinated against diphtheria, pertussis, and tetanus (DPT) and the percent of children 12-23 months immunized against the oral poliovirus, were achieved on target. These indicators and assisted deliveries, are good proxies for improved access, the second part of the DPO. Assisted deliveries increased significantly but fell just short of achieving their target in the PARSS provinces. Disaggregating the results to the provincial level, however, shows that the most challenging provinces in term of access (Equateur, Bandundu, and Maniema) all saw very important increases (except for Katanga) in assisted deliveries: from 51 percent in 2007 to 64 percent in 2014 in Equateur, from 70 to 83 percent in 20 Bandundu, and from 69 to 87 percent in Maniema. These are direct achievements of the project interventions. 65. The improvement of access is measured as increased availability of the EHS package through the construction of new facilities and training of health staff. During the project period, 818 health centers were constructed and over 13,640 health personnel was trained in the project areas. In particular, from an equality perspective access improved: the geographic selection of the most challenging regions with the lowest health outcomes provides the project with very high marks on equity. In addition, health services are now being provided where before the project started there were no services nor plans by government to establish new services. The project (i) contributed to the management, administration, and supervision capacity of the MOPH at the provincial and district levels, including improvements in infrastructure and equipment; (ii) improved and assured the availability of essential drugs and medication, including malaria control medication at the health facilities in the targeted 153 health zones covering about 20 million people (about 11 million benefited in the 83 health zones where the essential package was delivered but 20 million were covered by the malaria control activities); (iii) provided logistical support to health facilities in the targeted zones (vehicles, motorbikes, and ambulances); (iv) improved working conditions by assuring better payments through bonuses (primes) to health workers; (v) assured free distribution of LLINs; and (vi) financed research support, especially in the area of health financing. Millions of households have received free ITNs, immunization campaigns were successfully conducted, and awareness among the population has been raised. Annex 2 lists the different component activities that were financed by the project and the additional financings. 66. Financial barriers to access of services were removed with the increased resource availability, creation of the exemption program for the poorest and financial incentives packages for health staff, as well as increased accountability at the health facility level. The share of out of pocket expenditures of total health expenditures decreased to 40 percent between 2005 and 2013 (see Annex 3: Economic Analysis). 67. A number of indicators could not be verified for lack of data and are therefore not evaluable (see Table 3 below). Whether children received micronutrients is unclear as there is no data to verify this. The indicator on micro-nutrients was added only with the last AF-III; thus there is no baseline, or end-line data. Three other interim indicators, including reporting on malaria control, training of health staff in health zones and number of health staff trained in management and fiduciary responsibilities, lack data to report achievements. The latter three however, are interim indicators for which other indicators provide a reasonable proxy such as for example the number of trained health people overall in the PARSS project represents those in PARSS zones. 68. As Table 3 below shows, the project has achieved the majority of the indicator targets which can be measured. Even if the three non-evaluable indicators are counted, the overall achievement is close to 80% for both PDO and IO indicators. Table 3. PDO Indicators Achievement: Summary 21 Status PDO IO Indicators Indicators Surpassed 4 5 Achieved 2 this2 Partially achieved 3 2 Not achieved 1 1 Not evaluable (NE) 3 Total 10 13 % surpassed, achieved and 90% 90% partially achieved 70% if NE is included in the denominator 69. With regards to overall trends in health outcomes17 the most recent DHS data shows significant declines in infant and child mortality in DRC overall. Although it is difficult to attribute this directly to the project, the project’s contribution to preventing malaria among children under five, the biggest killer of children in this age bracket, cannot be understated. In two of the most inaccessible PARSS provinces, Bandundu and Maniema, the decrease in infant and child mortality at 45 and 52 percent, respectively, was more than the national average decrease of 37 percent between 2007 and 2014 (table 4). Given the remoteness of these regions, and the fact that PARSS was the only project in those regions, this was a significant achievement for the project. Other indicators such as assisted deliveries also confirm strong results in PARSS areas (Table 4). Table 4. DHS Health Outcomes Trends Indicators DRC Kinshasa Bandundu Equateur Maniema Katanga 2007 2014 2007 2014 2007 2014 2007 2014 2007 2014 2007 2014 Infant 92 58 73 50 103 57 102 65 129 62 94 72 mortality % decline 37 32 45 36 52 23 Under-five 148 104 102 83 154 89 168 132 219 105 154 121 mortality % decline 30 19 42 21 52 21 Assisted 74 80 97 97 70 83 51 64 69 87 70 64* Deliveries % increase 8 0 19 24 26 - 9* Source: DHS 2007, 2014. * worsening 3.3 Efficiency 70. The economic justification of the original project and the subsequent AFs was very clear. In 2005, after years of conflict, economic crisis, and mismanagement, DRC needed an injection of resources into the severely underfunded health sector. At the time, the burden of financing the health sector was largely borne by households in a country 17 Not a PDO indicator in the project, however a higher level objective of Government National Health Strategy as well as HNP Africa strategy. 22 classified as one of the poorest in the world. Keeping in mind the limitations, the economic analysis conducted for this ICR (Annex 3) demonstrates that, using the most plausible parameters, the project is likely to have generated positive returns for the Congolese people. In fact, if the DALYs are valued at the recommended rate of three times the per capita income, the estimates indicate that the project provided value for money throughout the entire range of the 95 percent uncertainty interval, even when increasing the discount rate to 5 percent. Project resources were used for strategic development, following national priorities and country partnership goals, as well as increased equity by investing in the most vulnerable areas. Additional Financing I and II had very strong economic rationales, malaria being the highest burden disease at the time and a polio outbreak would have put the entire region at risk. 71. Financial access to health services, through better financial protection, has improved. According to WHO, a more than 40 percent reduction in the share of out-of- pocket expenditure18 was observed in the country’s total health expenditure between 2005 and 2013, from approximately 57 to 33 percent. The decline in out-of-pocket expenditure is most likely explained by an increase in both external financing and public expenditure. The reduction of some of the user fees may have also contributed to this trend. Whatever the main cause, the data suggest that individuals are using less of their own money to pay for care and that out-of-pocket expenditure is no longer the main source of health financing in DRC. The forthcoming World Bank PER shows an increase in public funding for health from US$1.9 per capita in 2008 to US$2.8 per capita in 2013. However, an analysis of public health expenditures as a percentage of GDP shows that there was a decrease from 4.6 percent in 2008 to 4.5 percent in 2013. The contribution from external financing increased from 33.5 percent of the total health budget in 2008 to 37 percent in 2013. 72. Allocative efficiency is substantial. Using a combination of project data and international data in its assumptions, the economic analysis shows a cost-benefit return of 3 dollars to each dollar invested by the project. The analysis only uses the improvements - averted deaths and disability- from the increases in skilled deliveries, children and pregnant women sleeping under bed-nets and children vaccinated with DPT3 (for which evidence is available in the international literature), thus even this impressive return is likely underestimated. Even the most conservative estimate, the lower bound shows an estimated return of almost half a billion on the USD 332 million investment (for details see Economic Analysis Annex 3). 73. In terms of technical efficiency, the project used the resources to maximize health outcomes in the most cost-efficient manner with regards to activities. The high cost effectiveness of the distribution of LLINs, the mass immunizations, and the distribution of vitamin A has been demonstrated in numerous international studies.19 Implementation efficiency improved during the project period. At the start the project applied the most 18 The project contributed to a reduction in out-of-pocket payments by subsidizing an essential health package, thereby reducing consultation and other fees at health facilities through price-setting policies and performance payments. A small survey by one of the IPs, CCISD, demonstrated a reduction in the costs of services, prenatal care, facility-based deliveries, and consultations by half in the district of Mbandaka. Health facility staff explained that they could reduce prices thanks to the performance bonus they obtained and the increase in utilization (economies of scale). 19 World Bank 1993; Claeson, Mawji, and Walker 2000; Jamison, Jha, and Bloom 2008. 23 efficient implementation arrangements by contracting to experienced entities and those that had proven successful in health service delivery. Progressively, after capacity building, the more sustainable approach of implementing using national health system structures, was utilized with the local health sector authority taking increasing responsibility for service delivery. Toward the end of the project, after piloting the approach, RBF was applied to service delivery at the facility level. Although RBF has proven to be an efficient approach in other countries, the pilot in DRC was not convincing due to the many changes in design (see lessons learned). The AFs applied the most efficient approaches, based on the interventional evidence, with provision of impregnated bed-nets and mass-immunization against polio. 74. The lack of actual costing data does not allow for a cost-efficiency analysis. 75. In terms of implementation efficiency, delays were significant and as described in the earlier sections, very disruptive throughout the entire period. Efficiency of the AFI and II were substantial, although the efficiency of the original project and AFIII, together making up two-thirds of disbursement, was modest. Due to the delays throughout the project implementation, activities on the ground, even though relevant took much more effort rendering them only modestly efficient and significantly reduced the capacity building opportunities, and subsequently reducing sustainability ratings (see further). 76. On balance, taking into account Substantial allocative efficiency but Modest efficiency of implementation, the overall rating for efficiency is Modest. 3.4 Justification of Overall Outcome Rating Rating: Moderately Satisfactory 77. The Project relevance is rated as Substantial, Efficacy is a combination of ratings for each phase, averaging to Substantial, as explained above, and Efficiency is rated as Modest given the significant delays in project implementation. Therefore, the overall Outcome rating is Moderately Satisfactory and is also supported by Table 5 which presents the weighted disbursement averages before and after restructuring and for the three AFs. Table 5: Weighted average before and after restructuring and by each AF Against Against AF I AF II AF III Weighted original Revised average PDO targets total 48 million 102 80 million 30 million 75 335 disbursed million million Rating U MS S S MS Rating value 2 4 5 5 4 % disbursed 23.8% 9% 22.3% 100% 14% 30.4% Weighted 0.28 1.2 1.19 0.45 0.89 4.01 value Final rating MS 24 3.5 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 78. Gender aspects: The project focused most of its interventions on the most vulnerable groups (women and children) and the monitoring system included disaggregated data. Four of the nine PDO indicators were particularly focused on women. AF-III included a specific component to address GBV in non-conflict areas since the 2014 DHS confirmed that GBV was also a very frequent event in non-conflict areas in DRC, especially among the Pygmy population, where rape appeared to be a culturally acceptable event. UNFPA was contracted to raise awareness and train public health professionals in assessing GBV. Once the interventions started, a situational analysis revealed a higher prevalence of GBV than expected. Thanks to the project, this theme has been included in the national programming of the MoPH. The project also added value in this area with the strengthening of the health information system’s adjustment for reporting on gender-related violence. (b) Institutional Change/Strengthening 79. Keeping in mind the starting point of the health sector in 2005, the project made satisfactory progress in building in-country capacity at the decentralized levels. The main implementation responsibility is at the HZ level, as management and supervision occur at the province and district levels and policies are set the central level. In recent years, the province level has been provided with much more responsibility in the spirit of decentralization to increase the effectiveness and efficiency of health sector performance. Additionally, a very large focus has also been assigned to the HZs level, where service delivery occurs. 80. There has been an incremental shift toward reliance on in-country NGOs to serve as purchasing agencies rather than international organizations in order to focus on building in-country capacity and cost-efficiency measures. The IPs changed roles several times over the duration of the project and gradually handed over fiduciary and coordination responsibility to the district level. These tasks have now been assumed by district health administration staff as a result of the knowledge and skills transfer accomplished by the PARSS. The new project will be managed by the MoPH itself through the combined collaboration of the DEP (Direction des Etudes et Planification), the Financial Management Unit, and the Procurement Unit. Hence, there is now less reliance on an autonomous implementation agency as was done under the PARSS project, there is now greater reliance on in-country systems. 81. In the last 18 months as part of the third AF implementation, the Bank has shifted its strategy to scale up PBF, a strategy that has been piloted in DRC over the past 10 years. PBF supports various levels of the health system (starting with the health facilities) in developing results-oriented action plans and making payments based on the results (quality and quantity) achieved. This approach is the central focus of the new health system strengthening project, which will cover 140 HZs, reaching approximately 25 percent of the 25 population of DRC. The expanded reach is possible due to the harmonization of financing with key partners, notably UNICEF, the Global Fund, GAVI, and UNFPA. (c) Other Unintended Outcomes and Impacts (positive or negative) 82. Although this was not foreseen as a specific capacity-building opportunity during preparation, the close collaboration of the Public School of Health of the University of Kinshasa, the Catholic University of South Kivu, and the Free University of the Great Lakes with international universities to conduct the malaria-control surveys strengthened their capacity in survey methodologies, reporting, and results dissemination. 83. Donor coordination: The Bank’s presence in DRC in the health sector played a pivotal role in strengthening donor coordination as the project on the one hand relied on few partners such as UNICEF, UNFPA, WHO to implement part of the project but also was a conduit for other partners such as the Canadian Department of Foreign Affairs and Trade to invest in the health sector and align some of their activities to the project. The new US$220 million health project is founded on a strategic alignment with USAID, GAVI, Global Fund, UNICEF, and UNFPA. The Bank is also leading the sub-group on health care financing and is present in all the other sub-groups focusing on key pillars of the health system thus strengthening the technical and coordination of the health sector and ensuring better alignment to the Government’s health strategy. Finally, with DRC being one of the pilot countries for the Global Financing Facility this alignment will be further strengthened in the coming years to support the achievement of maternal and child health results by 2030. 3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops N.A 4. Assessment of Risk to Development Outcome Rating: Significant 84. The risk rating at the design stage was high due to the potential recurrence of conflict and continued weaknesses in governance and fiscal management. The risk was and continues to remain significant; and at times became high due security concerns in some regions and violence and uncertainties during the election period. The risk rating at the design stage was moderate for progress in other sectors that could influence health (road rehabilitation, water supply, food security, and education), and attention was paid to financial barriers to access to health services. 85. The main risk to the development objectives is the ability of the health sector to provide essential services at the local level. The overall risk rating at the design stage of the project was moderate and it remains as such. The risk mitigation mechanism in place now is the close donor coordination on the PBF approach being implemented nationwide. 26 A recent public expenditure review will be followed by a health financing strategy and investment case to help government enhance sustainability. 86. Government expenditure in health has increased, whether measured as a percentage of GDP or as a proportion of total public expenditure. Unfortunately, the data suggests some crowding out in recent years, with private sector expenditure, measured as a share of GDP, declining by almost 50 percent since 2009. As a result, after a brief (countercyclical) burst between 2008 and 2010, the country’s total expenditure in health remains at very low levels, below 4 percent of GDP (see Annex 3, figure A3.1). The forthcoming DRC Health PER shows an increase in public funding for health from US$1.9 per capita in 2008 to US$2.8 per capita in 2013. However, an analysis of public health expenditures as a percentage of GDP shows that there was a decrease from 4.6 percent in 2008 to 4.5 percent in 2013. The contribution from external financing increased from 33.5 percent of the total health budget in 2008 to 37 percent in 2013. 87. There is some evidence that improvements in some indicators waned during a brief period in which the project encountered implementation problems due to delays in financing. This can be seen as a “natural experiment” that could suggest that the intervention was indeed having a positive health impact. However, at the same time it shows clearly the continuing dependence on very heavy donor involvement, both financially and technically. Finally, despite the growing capacity at the local health levels, institutional capacity remains weak and requires continued substantial support in the foreseeable future. Project and financial sustainability is rated moderate. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Rating: Moderately Satisfactory 88. The Bank team made considerable efforts but had little control over issues that primarily affected readiness for implementation during the first two years of the project. The issues were linked to the challenging operational environment post-conflict, very weak governance systems, and completely deteriorated health system, coupled with the selection of the most challenging regions in which to focus project activities. Governance and personality issues were beyond the control of the project team but required government intervention that at that stage could not be counted on. (b) Quality of Supervision Rating: Moderately Satisfactory 89. The project was well supervised, with multiple missions over the period of the project, an average of two missions annually. For about half of the project period, the task team leader was based in the field, allowing for rapid and intensive follow-up and dialogue. An MTR and QAG were conducted after three years of implementation. The ISRs have been 27 candid, proposing mitigating measures and requesting timely management support. However, the kind of rapid decision making needed at all levels (management and team) in the unstable environment in which the project was implemented, especially in the early years, was at times lacking, this contributed to the project’s implementation delays. For example, the MTR and the subsequent restructuring took about a year to finalize and eventually resulted in improvements in the field, with disbursements going up and activities accelerating. 90. During the first years of the project, supervision focused almost entirely on dealing with procurement problems delays and getting the main contracts out. The nature of the project was heavily procurement focused with large funding. The Bank decisions and support on procurement in particular were slow and insufficient. More pro- active and timely support could have significantly contributed to improvements on the ground. This weak support on procurement from the Bank team continued throughout the project implementation period. 91. Team presence on the ground (including long presence of TTLs) helped to move implementation forward. The team conducted regular fiduciary reviews, although there are few written documents. Progressively throughout the project the Bank team tackled many of the issues and managed slowly but steadily to focus on the right areas with the right actors to implement the project. Reporting on these areas was candid in the ISRs, and follow-up was provided to governance problems—sometimes before they happened—such as early rejection of and recommendations on questionable procurement recommendations. The team also made careful preparations to ensure the good performance of the management entities, including formulation with the Borrower of an Action Plan and auditors TORs. Safeguard missions were less well documented, in particular with regards to follow up on recommendations. . 92. Despite the Bank’s efforts, attention to building the capacity of the MoPH was deficient. Much attention was given to resolving fiduciary issues and the need to effectively start project implementation. This focus deterred the need to build capacity for MoPH stewardship and donor harmonization. After implementation finally started following the 2010 restructuring, the focus of the Bank team shifted to the preparation of the three AFs that were prepared in a short time period, sidelining again the capacity building and harmonization dimensions. Only after procurement hurdles were eliminated in the last years of the project and implementation went more smoothly did the Bank team have time to focus on sustainability issues, such as promoting capacity building and reviewing new approaches to implementation/service delivery, including PBF. Only with the third AF, a change in service delivery approach, and a step-up in donor harmonization was better and more attention paid to building MoPH capacity. (c) Justification of Rating for Overall Bank Performance 93. Rating: Based on the Moderately Satisfactory rating for quality at entry and Moderately Satisfactory rating for quality of supervision, overall Bank performance is rated Moderately Satisfactory. 28 5.2 Borrower Performance (a) Government Performance Rating: Moderately Satisfactory 94. One of the main concerns voiced during the implementation completion review mission by the MoPH, donor partners, IPs, and the PCU itself was their lack of real engagement with the ministry. Although technical departments were involved, they remained on the margin until late into the project. At the intermediary level, the situation was much better, and medical inspectors and health managers were fully engaged, again especially after the first implementation hurdles were eliminated and contracts were in place. At the local health center level there was very good collaboration. Much of the lack of central-level engagement was the result of the strong leadership of the PCU coordinator, who himself came from the ministry and felt only a limited need to consult at regular intervals. However, the ministry also did not make much effort to remain engaged and left the PCU to manage the project very independently. (b) Implementing Agency or Agencies Performance Rating: Moderately Satisfactory 95. The strong leadership of the PCU was an advantage and a constraint for project implementation, as it stood in the way of relations with the MoPH and the BCECO, which led to impasses and delays in contracting during the important start-up years. After all had been smoothed out, the capabilities of the coordinator became more visible with the quality of the PCU’s organization of the health forums, project documentation, and management and supervision of the IPs. However, a well-organized forum is not sufficient for success, as commitment and real engagement from all stakeholders are needed as well. The latter was difficult to foster in a climate of limited collaboration between the PCU and the ministry. One key concern that long remained an obstacle and was insufficiently addressed during the entire implementation period was the persistently weak procurement capacity, which often led to urgent requests requiring rapid rather than comprehensive reviews of procurement packages, thereby reducing quality and increasing the risks of even more delays. During the last years of the project and the management of the AFs, capacity was built at the provincial and district levels of the MoPH and overall management has improved. At the same time, the decision to contract out very large parts of the implementation to partner organizations, and the management of those contracts, can be rated Satisfactory. After initial procurement of the large contract issues, the implementation and monitoring advanced smoothly and contributed to a large extent to the success of the malaria control and polio control programs. (c) Justification of Rating for Overall Borrower Performance Rating: Moderately Satisfactory 29 96. Limited engagement at the central level was in part due to the urgency of getting the project started, this pushed the PCU to focus on moving ahead with the contracting and to neglect various channels of communication. The Project Coordinator at the time came from the ministry and followed its strategy, but he failed to communicate regularly with his colleagues and report on project interventions, problems, and achievements. After restructuring and with the three AFs, during the last years of implementation the situation improved. More responsibility for implementation was moved to the ministry, regular communication took place (which then became a requirement), and much more attention was focused on the capacity building of the health sector itself. The new project will be implemented by the DEP directly with support from the other ministerial units. 6. Lessons Learned 97. Comprehensive capacity and realistic readiness assessments at project appraisal are very critical in fragile states/post-conflict areas. The project’s ambitiousness given the collapsed health system, its challenging geographical coverage and the country context, coupled with very weak institutional capacity and significant governance challenges, should have benefited for more explicit assessment of institutions and risks and stronger measures to mitigate the occurrence of project-related risks. Therefore, the project team and Bank management could have applied more stringent criteria to readiness and preparation to address the weaknesses in implementation capacity. The reliance on local- level NGOs —even with their demonstrated experience—was overestimated as was shown at the outset, mainly because of the unexpectedly challenging environment of the selected regions, which had been underestimated. 98. Realism in target setting and timely documentation of progress is essential. The many changes in indicators, including wording, targets and completely new indicators added over the course of the restructuring and AFs added much complexity to monitoring and evaluating the project progress. Better documentation on progress on these indicators and more realistic assessment before changing targets would have simplified subsequent assessments. 99. Parallel implementation structures are justified in fragile environments but need to be accompanied by a strong focus on sector capacity building and rigorous follow- up. In DRC, the health sector had all but collapsed and was in great need of rebuilding, which, at the time, the national health system was not equipped to handle. Given that situation, the project contracted out most of the interventions to third parties and donor partners (UNICEF, WHO, UNFPA). Although all contracts were accompanied by clear mechanisms to concurrently build national health sector capacity, due to the many implementation problems, attention to this area often fell short. After most of the project supervision management went to solving procurement problems and getting contracts signed and intervention started, little communication and real capacity building of the national health system took place except at the intermediary levels. After restructuring, most attention went to the AFs and the focus was on quick results, understandably, given the malaria and polio threats. One exception needs mentioning which is the polio 30 vaccination effort driven by the NIP. Only during the last years of the project did the implementation arrangements change to give more responsibility and the needed capacity building to the sector, it is now showing results. The new project will rely much more on the sector itself for implementation. 100. Rebuilding health workers’ trust in the health system is paramount to service delivery. The project clearly shows the importance of motivating health workers and rebuilding their trust. The project’s component 1 included a provision of motivational bonuses with some discretion for facilities on how to divide the bonuses. Added to this were the improvements in the work environment and the enhanced provision of medicine and pharmaceuticals. The introduction of the “bonus de démarrage” (start-up bonus) with the AF-III especially gave new life to the project’s objective of improving health facilities. This bonus was used to purchase new beds for maternity wards and to provide improved infrastructure such as latrines and sanitation. 101. Bank-supported projects can bring in new focus and leverage funding to pilot interventions. The gender-based violence pilot in a non-conflict region is a good example of making funds available and contracting out a pilot in a theme-area that was not recognized as an urgent issue. The pilot managed by UNFPA was very successful and will be scaled up in the next project. The PBF pilot is another example of how the project, with leveraged funds, introduced a new approach to financing the system. The pilot was evaluated and produced results that confirmed that RBF could work in DRC. Many of the lessons and findings of this pilot are also included in the new health project, which will scale up and implement the PBF approach nationally. 102. A vertical approach to rebuilding a decimated health system should be accompanied by a health systems strengthening approach. The project was started with the purpose to rebuild a deteriorated health sector. The approach to rebuilding at the time, in a very weak environment, was to focus on getting the vertical service delivery functional again. In the ten-year implementation period, the situation evolved and the importance of not only rebuilding the basic services at the local level but also the system itself, became important. The project, with the help of the Trust Funds, initiated some work on strengthening the health system, in the areas of health financing and financing mechanisms, but much more remains to be done. The project provides many lessons to be taken on in the new operation. 103. The design and implementation of the DRC – Katanga pay-for-performance pilot project was incorrect in many aspects and this has led to an absence of results. 15 years of accumulated lessons learned, the collective science of implementation for Performance-based financing (PBF) projects has led to a wealth of lessons learned related to design and implementation aspects that taken together, much enhance the probability of showing strong effects [Reference: Chapter 17, Fritsche, György Bèla; Soeters, Robert; Meessen, Bruno. 2014. Performance-Based Financing Toolkit. Washington, DC: World Bank. © World Bank. https://openknowledge.worldbank.org/handle/10986/17194 License: CC BY 3.0 IGO.”]. DRC has been at the core of lessons learning for PBF design and implementation, as it has benefited of various pilot PBF programs since 2000. The exact 31 areas of design and implementation that were at fault have been well- documented in the World Bank policy note on the Katanga Impact Evaluation and will not be repeated here [reference: Policy note]. Suffices to say that design and implementation characteristics of scaled up PBF projects in Rwanda and Burundi [References: Basinga, P., P. Gertler, et al. (2011). "Effect on maternal and child health services in Rwanda of payment to primary health-care providers for performance: an impact evaluation." The Lancet 377: 1421- 1428; Bonfrer, I., R. Soeters, et al. (2014). "Introduction of Performance-Based Incentives in Burundi was Associated with Improvements in Care and Quality." Health Affairs 33(12): 2179-2187; Bonfrer, I., E. v. d. Poel, et al. (2014)]. "The effects of performance incentives on the utilization and quality of maternal and child care in Burundi." Social Science & Medicine 123: 90-104].was chosen as a basis to re-design the PBF project, and this re- designed PBF project forms the core of the current PDSS program. 104. Institutional continuity in leadership over the project and adequate supervision resources are important. Not only was the PARSS an ambitious project in a very challenging environment, over its lifetime, three AFs were added, two restructurings took place and three Trust Funds were used to support the project. Over that same period, four task team leader changes took place at crucial moments in the project. The experience with these changes - the delays that were incurred, the steep learning curves for those taking over – call for a more careful consideration for the institutional continuity and adequacy in resources to manage a project of this magnitude and complexity and challenging context. 105. Was the use of AFs appropriate in the DRC context? A final lesson of the PARSS relates to the use of the three additional financings as instruments. AF-I was justified as providing very timely additional resources to address an urgent problem. It also provided an easy instrument to use available resources at the time. The AF-II also was justified as providing timely resources to prevent a potential epidemic from happening. Both were implemented very successfully and had they been evaluated by themselves, would have been highly satisfactory projects. However, they (the AFs) came at a cost to the original project which had to divert attention and resources to managing very large contracts, with limited procurement capacity, and coordinating with many partners at the same time. The AF-III was less well justified as an additional financing, as it really picked up several pieces and provided a financing gap which should not have occurred; a new project should have been prepared instead and this would have allowed the project to address the then clear implementation problems more effectively. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/implementing agencies (b) Cofinanciers (c) Other partners and stakeholders 32 Annex 1. Project Costs and Financing (a) Revised Estimates of Project Costs Financed by IDA (amounts in ‘000US$) Component Activity Original AF-I AF-II AF-III Total budget revised 1 Expand Access 104.0 27.0 62.5 193.50 and Utilization EHS (Polio) 2 Malaria 30.0 73.2 103.2 Control Interventions 3 Strengthen 12.0 4.8 2.5 11 30.3 Capacity of Evidence- Based Management 4 Project 4.0 2.0 0.5 1.5 8.0 Coordination Total 150.0 80 30 75 335 (b) Financing Loan Approv Closing Original Cancelled Disbursed Undisburse number al date date d IDA-H1820 01-09- 06-20- 150 91,761.62 152,924,580.88 2005 2013 million IDA-H6380 12-20- 06-30- 80 million 929,755.16 79,643,643.10 2010 2013 TF-095892 02-25- 09-30- 10.8 106,331.55 749,218.45 2010 2013 million IDA-H7170 06-28- 06-30- 30 million 987,203.73 27,372,177.56 2011 2013 TF-017380 06-11- 12-30- 2.5 0 9,985,522.13 14,477.87 2014 2014 million IDA-H7500 07-06- 12-31- 75 million 0 72,426,167.98 11,171.48 2012 2014 (c) Commitments and Disbursements by Components (in USD) Components Commitment Actuals % disbursed 1 Expand Access and 206,955,550 214,504,890.74 103.65% Utilization of EHS 2 Malaria Control 103,200,000 101,471,651.82 98.33% Interventions 3 Strengthen Capacity of 30,300,000 4,160,582.48 13.73% Evidence-Based Management 4 Project coordination 8,000,000 25,990,979.72 324.89% Total 348,455,550 346,128,104.76 99.33% 33 Annex 2. Outputs by Component and Overview Changes to Indicators and Revised Targets Outputs by component: Original project Component 1: Expand Access and Utilization of a Proven Package of Essential Health Services to Selected Districts and Health Zones Key Activities:  Replacing and upgrading cold chain equipment  Purchase of vehicles and delivery of vaccines  Renovation and upgrading of health facilities  Vaccination during pre-school consultation visits to health facilities  Maternal vaccination at health facilities during pre-natal visits  IPT treatment for pregnant women  Iron/folate and Vitamin A supplementation for pregnant women and children under five  IMCI protocol treatment for children under five presenting at health facilities  ORS and zinc treatment for children presenting with diarrhea at health facilities  Implementation of new malaria treatment policies  Upgrading of first referral hospitals ensuring appropriate referral  Provision of overall essential health services including HIV/AIDS, TB and other priority diseases such as onchocerciasis in endemic areas  Development of Community Health Services in selected areas.  Reform of payment system  Development of Health Zones: Capacity building of management and coordination at the provincial and district level Key Outputs: see table A2.1 Original project Component 2: Boost Malaria Control Interventions  Intermittent Preventive Treatment for pregnant women  Malaria treatment for children  Purchase and distribution of long-lasting insecticide treated bed-nets  Technical assistance  Operational research 34  Operational costs and management Key Outputs: see table A2.1 Among the main inputs that contributed to the activities and below listed outputs are:  Drugs and medical equipment, including ACT (US$ 50 million)  Long-lasting impregnated bednets (US$ 12.2 million)  HIV/AIDS rapid testing US$ 3 million)  Cold chain and vaccines (US$ 11.3 million)  Contraceptives (US$ 3.2 million)  Nutrition inputs (US$ 3 million)  IT equipment (US$ 650.000)  And support to distribution and access to services with 528 motorcycles, 158 four-wheel drive cars, 14 pick-up jeeps, 7 all-terrain vehicles, 2 mini-bus, 1 truck for drug transport, 7500 bicycles, a boat and several smaller non-motor water vehicles. Original project component 3: Build Oversight and Management Capacity:  Contract with Implementation Partners  Contract with External Evaluation Agency  Support to Districts and Province Health Teams  Technical Assistance  Operational Research Key Outputs: see table A2.1 Additional Financing I additional activities:  Contribution to government LLIN mass distribution campaign in Katanga, South and North Kivu  Provision of malaria-related preventive, diagnostic and treatment services in PARSS zones  Support to health system strengthening through training of health care providers and support to key functions within the Ministry  Monitoring and Evaluation activities Additional Financing II additional activities: 35  Scaling up the immunization related activities under the original component 1 and 3  Contract partner to purchase and distribute Oral Polio Vaccin  Support cold chain equipment  Logistics training  Purchase of vehicles  Social mobilization Additional Financing III additional activities:  Scaling up original project activities  Add nutrition interventions  Strengthen HIV/AIDS interventions  Pilot gender violence in non-conflict areas  Strengthen capacity building at the Province level  Expand Results-based financing approach Table A2.1. Outputs by Component Intermediate indicators Baseline MTR 2011 Final Target 2007 /Restructurin Original Results at 2014 g 2010 Project Closure disbursed 2014 1. Pregnant women enrolled 63% 84% 84% 87% 90% in antenatal care (HMIS) 2.Health personnel 1,077 10,500 13,640 13,000 receiving training (HMIS) 3.Health facilities 387 808 818 850 constructed, renovated, (HMIS) and/or equipped (cumulative) 4.Children immunized 250,000 1,580,000 1,580,000 602,000 578,000 (SNIS) 5.% of children between 6 53% 83% 97% 95% and 59 months having (HMIS) received vitamin A 6.% of provincial and 90% 99% 95% health zones with OPV in (NIP) stock and without stock outage 36 7.% of provincial and 60% 98% 90% health zones with adequate (NIP) vaccine storage, cold chain, and capacity 8.Households that possess 9.2% 50.2% 50.2% 70% 75% at least one or more bednets 9.Under-five 5 with fever 53,3% 80% confirmed malaria and (HMIS) treated with ACTs in PARSS zones 10.LLIN malaria nets 2,9 million 2,9 million 11,8 11,8 purchased and /or million million distributed (HMIS) 11.% of reports including No data 75% malaria control activities completed at health zone level and transmitted to province level 12.% of children between 6 No data 22% and 59 months receiving micronutrient supplements 13.Number of central- and No data 400 provincial-level staff trained in health systems management and fiduciary responsibilities Table A2.2. Overview Changes to Indicators and Revised Targets PDO: The objective of the project is to ensure that the target population of selected health zones has access to and use of a well-defined package of quality essential health services. Indicators: MTR (2009) AFI AFII (June, AFIII (June, (December, 2011) 2012) 2010) Annual per No change No change No change No change capita curative Target 2011: Revised target: Revised target: consultation rate 0.40 0.37 0.45 in target health zones % of pregnant Reworded: No change No change No change women who % deliveries Revised target: Revised target: receive delivery assisted by 75% 85% care from a qualified trained provider personnel 37 in target health zones New: % of Reworded: % No change No change women using women 15–49 Revised target: modern who are new 11 contraception users of family planning Target: 7% % of pregnant Modified: % Modified: % of No change No change women sleeping of children pregnant Revised target Target revised: under an ITN in and women women who 74% 60% target zones who have slept under an access to ITN the night ITN before No target Target: 70% % of children Modified: % of No change No change under 5 sleeping children <5 Target revised: under an ITN in who slept under 50% target zones an ITN the night before Target: 70% New: % of No change No change pregnant Target revised: women 45% receiving 2 doses of IPT against malaria Target: 70% % of children Revised: % No change No change No change ages 12–23 of children Revised target: Target revised: months with 0–11 months 95% 90% DTP3 vaccinated vaccination in with DPT3 target health Target: 75% zones New: % No change: children target revised: immunized 90% with OPV3 Target: 94% New: Total No change number of Revised target: polio cases >20 new cases reported quarterly 38 Target: < 30 new cases Component 1: Expand Access to and Utilization of a Proven Package of Essential Health Services to Selected Districts and Health Zones Indicators: % of children New: % No change No change No change: presenting at women Target revised : PHC facilities enrolled in 90% in target health antenatal zones who are care treated under Target: 75% IMCI protocols Caesarian Dropped section rate in (no data target health available) zones % of cases of Dropped illness in (replaced previous 2 with no. of weeks in target beneficiaries) health zones who did not seek care due to cost New: Direct No change No change No change beneficiaries Target: 5 of which are million, of female which 60% are (CORE) women New: % of No change provincial and Target revised: health zones 95% with OPV in stock and without stock outage Target: 98% New: % of No change provincial and Target revised: health zones 75% with adequate vaccine storage, cold chain and capacity 39 Target: 98% New: % of No change children Target revised: between 6–59 95% months having received vitamin A Target: 94% New: % of children between 6–59 months receiving micronutrient supplements Target: 22% Component 2: Boost Malaria Control Interventions Indicators: Proportion of Moved to women who PDO receive indicators intermittent Target: preventive 60% malaria treatment during last pregnancy Proportion of No change No change No change No change households with Target: Revised target: Target revised: at least one 60% 80% 75% LLIN Number of Dropped health centers (too specific, with no reported replaced stock-outs of overall anti-malarial stock-out ) drugs New: No change No change No change Number of Target: 2500 Revised target: health 13,000 personnel receiving training (CORE) 40 New: Health No change No change No change facilities Target: Revised target: constructed, 400 850 renovated, and/or equipped (CORE) New: No change No change No change Children Target revised: immunized 578,000 (CORE) Target 2011: 1.4 million New: No change No change No change Number of No change in pregnant target women receiving antenatal care during a visit from a health provider (CORE) Target 2011: 1.1 million New: ITN No change No change No change purchased Target revised: and /or 11.8 million distributed (CORE) Target 2011: 2.9 million New: % of No change No change children < 5 Revised target: with fever 80% treated with ACTs Target: 60% Component 3: Strengthen Capacity for Oversight and Evidence-Based Management of the Health System Indicators: Number of Dropped completed health facility surveys 41 Completed Dropped study on health financing Development of Dropped health human resources strategy New: Project Modified: % of No change No change management reports, Revised target: compiled including 75% data from malaria control NGOs and activities, reports completed at quarterly on health zone performance level and transmitted to province level Target: 70% New: % of No change health zones where staff has been trained Target: 515 New: Number of central- and provincial-level staff trained in health systems management and fiduciary responsibilities Target: 400 Component 4: Project Coordination Indicators: % of problems Dropped raised by IPs effectively addressed by the PIU % of IP Dropped resources raised from disease- specific grant 42 programs external to the project New: % Dropped disbursement of NGO contracts Table A2.3: PDO indicators achievement 2005 2007 2009 MTR- 2011 2013/4 Target Project indicators Baseline 2010 Original Final 2013/4 (original and revised as Restructuri Project Results per table 2 in Section 1) ng 100% (data- disbursed / source) or latest data PDO indicators 1. Per capita annual 0.19 0.32 0.39 0.37 0.45 curative consultation rate (HMIS) 2. Children 0–11 months 54% 61% (MICS) 83% 90% 90% vaccinated with DPT3 in 93% (HMIS) (HMIS) the project target zones 3. Children 12-23 months 89% 90% who have received Oral (HMIS) Polio Vaccine (OPV3) 4. Deliveries assisted by 47% 70% 70% 75% 85% qualified personnel (HMIS) 5. Women 15–49 years new 2% 5% 6% 2% 11% users of family planning (HMIS) 6. Under-five children who 14.3% 38% 38% 66% 50% slept under an ITN the night (MICS) before 7. Pregnant women who 15.3% 43% 43% 63% 60% slept under ITN the night (MICS) before 8. Pregnant women who 26% 49% 49% 47% 45% received 2 doses of IPT (MICS) during last pregnancy 9. Total number of new 98 0 new <20 new polio cases reported cases cases quarterly 43 Direct beneficiaries (of 7,110,000 7,110,000 24,7 25,5 which female) (51% 51% million million of women) female) (54%) which 60% (IPs) women Table A2.4. PDO Indicators Achievement PDO Baseli Final data End Exceeds/on/cl Difference between target Indicator ne 2014 target ose/below and achieved/comment target Per capita 0.19 0.37 0.45 Target partially 0.07 pp/py below target; in annual curative achieved. 2011 the rate was 0.40; a consultation dip occurred afterward in rate implementation due to lack of resources Children 0–11 54 90 90 Achieved months vaccinated with DPT3 in the project target zones Children 12-23 46 89 90 Achieved months who have received Oral Polio Vaccine (OPV3) Deliveries 47 75 85 Target partially 10% below. Variance by assisted by achieved. province *: Bandundu, qualified Equateur, and Maniema personnel increased by over 10% points; Kinshasa has already achieved close to 100%; Katanga saw decline. Women 15–49 2 2 11 Target not FP interventions started new users of achieved. very late and in 2011, FP family use was 6%. A dip occurred planning (FP) after all funds were disbursed. Under-five 38 66 (June 50 Overachieved 10% over target children who 2013) slept under ITN the night before 44 Pregnant 43 63 (June 60 Overachieved 3% over target women who 2013) slept under ITN the night before Pregnant 49 47 45 Overachieved 2% over target women who received 2 doses of IPT during last pregnancy Total number 128 0 >20 new Overachieved After mass vaccination no of polio cases cases new cases have been reported reported in December 2014. quarterly 45 Annex 3. Economic and Financial Analysis This section presents the findings from the economic and financial analysis of the project. Economic context and the health sector As noted in the main document, in 2004–05 when the project was being prepared, the Democratic Republic of Congo (DRC) was beginning to recover from the Congo Wars that ended in 2003 with the appointment of the Transitional Government. Since 2005, the country has been experiencing high rates of economic growth (averaging over 6 percent per year), a trend that was partially interrupted in 2009 by the international financial crisis, when GDP growth fell below 3 percent. These high rates of growth are important to a country that is still among the poorest in the world, with a population that continues to increase by almost 3 percent per year and has a fertility rate of approximately five children per woman. World Development Indicators (WDI) data suggest that these economic improvements have been only partially transferred to the financing of the health sector. The data show a general pattern of growth until 2009 and a decline after that. This pattern is observed in terms of public expenditure in health (whether measured as a percentage of GDP or as a proportion of total public expenditure) and with respect to private sector expenditure. As a result, the country’s total expenditure in health has remained very low, generally below 4 percent of GDP, with 2013 levels not much different from those seen in 2004 after the wars (see figure A3.1 below), and the country continues to have one of the lowest per capita health expenditures in the region. Figure A3.1. Patterns of Health Expenditure (Index, 2009 = 100) 120 100 80 Index (2009 = 100) 60 40 20 0 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Public Expenditure as % of GDP Private Expenditure as % of GDP Total Health Expenditure as % of GDP Public Expenditure as % of Total Gov. Expenditure Source: WDI. 46 Perhaps the most positive outcome in the financing of reform in the health sector is the more than 40 percent reduction in the share of out-of-pocket expenditure in the country’s total health expenditure observed between 2005 and 2013, from approximately 57 to 33 percent. The decline in out-of-pocket expenditure is most likely explained by an increase in both external financing and public expenditure, as figure A3.2 indicates.20 It should be noted that the sharpest decline in out-of-pocket expenses occurred when total health expenditures were increasing the most (i.e., between 2006 and 2009), and the decline continued even when the share of external financing dropped (e.g., between 2006 and 2007). Whatever the main cause, the data suggest that individuals are using less of their moneys to pay for care, and out-of-pocket expenditure is no longer the main source of health financing in DRC. Figure A3.2. Trend in Out-of-Pocket Expenditure and Other Sources of Health Financing 70 35 60 30 50 25 Constant 2011 International Dollar 40 20 % 30 15 20 10 10 5 0 0 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Health expenditure, public (% of total health expenditure) External resources on health (% of total health expenditure) Out‐of‐pocket health expenditure (% of total health expenditure) Health expenditure per capita, PPP (constant 2011 international $) Source: WDI and WHO. Despite these mixed results in health financing, health outcome indicators improved throughout the period in which the project was implemented, for instance: - life expectancy at birth increased between 2005 and 2013 for both women and men, and relatively more for the former, from 49.4 years to 51.8 years and from 46.3 years to 48.2 years, respectively; - although still high, infant mortality declined between 2005 and 2013, from 104.9 to 86.1 deaths per 1,000 live births. 20 The reduction/elimination of some of the user fees may have also contributed to this trend. 47 Economic analysis It should be obvious that these improvements cannot be directly attributed to the Bank’s intervention. The same applies for the discussion that follows. The extent to which the project can explain the improvements observed in the health outcomes monitored can only be assessed with a proper impact evaluation. In this sense, it must clear that this analysis does not have a counterfactual against which results can be can compared and that it assumes that the project does explain the changes in outcomes. Having said that, an important caveat must be noted: as discussed in the main document, there is some evidence that improvements in some indicators waned during a brief period in which the project encountered implementation problems. This can be seen as a “natural experiment” that could suggest that the intervention was indeed having a positive health impact. This economic analysis assesses the benefits generated by the Bank’s intervention as they relate to four of the nine health outcomes or indicators of the Project Development Objective (PDO): i. percentage of deliveries assisted by qualified personnel; ii. percentage of pregnant women who slept under insecticide-treated nets (ITN) the night before; iii. percentage of children less than five years of age who slept under an ITN the night before; and iv. percentage of children 0–11 months vaccinated with diphtheria, pertussis, and tetanus (DPT3). Given the limited time available for this analysis and the scarcity of data, these were the only indicators that could be translated into health benefits (i.e., truly represented outcome indicators) and/or for which data could be found and evidence in the literature that estimate the health impact of these interventions.21 Despite these constraints, they do reflect the project’s main actions in malaria and maternal and child health. The 2010 Global Burden of Disease (GBD) dataset provided by the University of Washington’s Institute of Health Metrics and Evaluation (IHME) was the main source of information used in the analysis.22 The dataset contains age- and sex-specific estimates for mortality, disability, disease burden, life expectancy, and risk factors. As the data available are aggregated at the national level and therefore not specific to the areas of project intervention, they may not truly reflect the local reality. In this sense, as the areas selected for the project had some of the country’s worst health conditions, this exercise may be underestimating the health benefits derived from the intervention. 21 Regarding the other PDO indicators, it would be almost impossible to provide a reasonable estimate to measure the health impact of an increase in the rate of curative consultations as well as the use of family planning. There was no baseline for children immunized with the oral poliovirus (OPV3), and data on the burden of polio or literature on the effectiveness of the two doses of intermittent preventive treatment (IPT) on malaria in pregnant women could not be found. 22 http://www.healthdata.org/gbd/data. 48 Age- and sex-specific estimates were calculated for years of healthy life lost due to non- fatal health conditions (or years lived with a disability [YLD]) and years of life lost (YLL) due to premature mortality, which, combined, generate the disability-adjusted life years (DALYs) lost. The direct health benefit from the project arises from the YLD, YLL, and therefore DALYs averted by its intervention. Because the 2010 GBD 2010 data are nationwide, as noted above, they were adjusted for the population size of the project and the changes generated during implementation by the Additional Financings (AFs), particularly in the case of malaria. Following the recommendations from the second edition of the Disease Control Priorities in Developing Countries (DCP2) (see Jamison et al. 2006) and the Copenhagen Consensus (see Jamison, Jha, and Bloom 2008), each DALY averted was converted into monetary value at three times the per capita income of the year (World Development Indicators [WDI] data at constant U.S. dollars of 2005). Because individuals (and societies) tend to prefer to receive benefits sooner rather than later and to incur costs in the future rather than now, it is necessary to find an instrument to convert the future values of benefits and costs into a common present value. Discounting does that, as it generates the (net) present value of the stream of benefits and costs. Following the guidelines from the World Health Organization (WHO) (Tan-Torres Edejer et al. 2003) and DCP2, the stream of the monetary value of the annual DALYs averted was discounted at 3 percent. As per these same guidelines, project costs were also discounted at 3 percent. We have used the evidence provided by the literature to estimate the effectiveness of specific interventions supported by the project to combat malaria in children less than five years of age and in pregnant women (Morel, Lauer, and Evans 2005; Pulkki-Brännström et al. 2012). More specifically, it was assumed that ITNs reduce the incidence of malaria by 50 percent and case fatality by 20 percent. Evidence from the literature was also used to estimate the impact of birth deliveries with skilled professionals on maternal and neonatal mortality, in particular the regression equations derived by Berhan and Berhan (2014) (see also Graham, Bell, and Bullough 2001). The DRC mean age at childbearing (29 years) was used to define the age-specific DALY. Regarding the impact of the DPT3 vaccine, it was assumed that an immunization coverage of 80 percent and above would protect the entire cohort of children. In order to ensure consistency with the project duration, the 1–4 age period was used to circumscribe the DALY. Baseline results Table A3.1 provides the baseline estimates of the ratio between the net present value (NPV) of benefits and costs obtained using the assumptions described above. It shows the results according to the parameters recommended by WHO, DCP2, and the Copenhagen Consensus (i.e., a discount rate of 3 percent for both benefits and costs and DALYs valued at three times the per capita income). The mean value is highlighted and shows a benefit- cost ratio of approximately 3, which means that each dollar invested by the project generated 3 dollars in benefits. Under these parameters, both the lower and upper boundaries of the 95 percent uncertainty interval show benefit-cost ratios greater than one. 49 These results provide some confidence regarding the return on investment (disregarding the problem of attribution noted earlier). Table A3.1 Benefit-Cost Ratios – Baseline Estimate (3% discount rate and DALYs valued at three times per capita income) Benefit-Cost Ratio Lower Bound Estimate 1.47 Mean Value 3.20 Upper Bound Estimate 6.22 Source: Author’s calculations. Sensitivity analysis In order to assess the robustness of these results, we conducted a sensitivity analysis exercise varying the different parameters used. Table A3.2 shows the baseline results, together with a more conservative estimate generated using a valuation of the DALYs equal to the country’s per capita income. As is to be expected, while the mean value of the benefit-cost ratio is still (marginally) greater than one, the estimate at the lower boundary of the 95 percent uncertainty interval falls below unity. Accordingly, if benefits were to be valued at the average per capita income of DRC, we would not have a very high degree of confidence regarding the positive returns from the project. Table A3.2. Benefit-Cost Ratios – Baseline Estimate and Estimate with DALYs valued at per capita income One Time Three Times GDP p.c. GDP p.c. Lower Bound Estimate 0.49 1.47 Mean Value 1.07 3.20 Upper Bound Estimate 2.07 6.22 Source: Author’s calculations. Because the NPV can be very sensitive to the specification of the discount rate, we have estimated benefit-cost ratios at various scenarios. Again, following the recommendation of WHO and DCP2, we have estimated NPVs and benefit-cost ratios using a 5 percent 50 discount rate for the benefits. These results are shown in table A3.3. The highlighted columns display the estimates at the new discount rate. Table A3.3. Benefit-Cost Ratios – Sensitivity Analysis with 5% Discount Rate for Benefits Benefit Parameters One Time GDP One Time GDP Three Times GDP Three Times GDP p.c. and 5% p.c. and 3% p.c. and 5% p.c. and 3% Discount Rate Discount Rate Discount Rate Discount Rate Lower Bound 0.43 0.49 1.29 1.47 Estimates Mean Values 0.94 1.07 2.82 3.20 Upper Bound 1.83 2.07 5.49 6.22 Estimates Source: Author’s calculations. As can be seen from the table, an increase in the discount rate from 3 to 5 percent applied to the benefits does not move the benefit-cost to below the unity, even at the lower boundary of the confidence interval, when DALYs are valued at three times the country’s per capita GDP. This provides additional confidence regarding the “social profitability” of the project under the recommended parameters for the economic evaluation of health projects in developing countries. However, with a 5 percent discount rate and DALYs valued at the per capita GDP, the project ceases to present returns at even the mean value. We also generated estimates for different values of the discount rate applied to costs. In addition to the original 3 percent, we have used a discount rate equal to 7 percent (to reflect the average inflation rate observed in the country since 2010). 51 Table A3.4. Benefit-Cost Ratios – Sensitivity Analysis with 5% Discount Rate for Benefits and 7% Discount Rate for Costs Benefit Parameters Cost Parameters One Time GDP One Time GDP Three Times GDP Three Times GDP and 5% Discount and 3% Discount and 5% Discount and 3% Discount Rate Rate Rate Rate Lower Bound Estimates 3% 0.43 0.49 1.29 1.47 7% 0.54 0.62 1.63 1.86 Mean Values 3% 0.94 1.07 2.82 3.20 7% 1.19 1.35 3.57 4.05 Upper Bound Estimates 3% 1.83 2.07 5.49 6.22 7% 2.32 2.63 6.95 7.88 Source: Author’s calculations. The table shows that at the mean value, the project provides positive returns under all scenarios, except if benefits are discounted at 5 percent and DALYs are valued at one time the per capita income. The estimated benefit-cost ratios vary from a minimum of 0.43 to a maximum of almost eight. The results presented in the table are also graphically displayed in figures A3.3–A3.5 below. Figure A3.3. Sensitivity Analysis, Mean Values 4.5 4 3.5 3 Benefit‐Cost Ratio 2.5 2 1.5 1 0.5 0 3*GDP p.c. ‐ 3% 3* GDP p.c. ‐ 5% 1*GDP p.c. ‐ 3% 1*GDP p.c. ‐ 5% Benefit Parameters 7% 3% Source: Author’s calculations. 52 Figure A3.4. Sensitivity Analysis, Lower Boundary 95% Uncertainty Interval 2 1.8 1.6 1.4 1.2 Benefit‐Cost Ratio 1 0.8 0.6 0.4 0.2 0 3*GDP p.c. ‐ 3% 3* GDP p.c. ‐ 5% 1*GDP p.c. ‐ 3% 1*GDP p.c. ‐ 5% Benefit Parameters 7% 3% Source: Author’s calculations. Figure A3.5 Sensitivity Analysis, Upper Boundary 95% Uncertainty Interval 9 8 7 6 Benefit‐Cost Ratio 5 4 3 2 1 0 3*GDP p.c. ‐ 3% 3* GDP p.c. ‐ 5% 1*GDP p.c. ‐ 3% 1*GDP p.c. ‐ 5% Benefit Parameters 7% 3% Source: Author’s calculations. 53 Final discussion As is always the case with cost-benefit and cost-effectiveness analysis, several assumptions were used in this exercise. In the particular case of this project, we were forced to estimate benefits for only part of the interventions supported by the project, thus underestimating the total value generated by the Bank’s efforts. On the other hand, while we have used the total cost of the project, we have not incorporated any expenditure incurred by the Government of DRC in project-related activities, which underestimated the project’s true cost. Since in the case of DRC public expenditures tend to be small, it is likely that benefits may have been underestimated to a greater extent than costs, but this is an educated “guess estimate.” Finally, it should be remembered once again that the strongest assumption of this analysis rests in the fact that we have attributed all the benefits captured by the outcome indicators to the project intervention, a fact that cannot be corroborated without a proper impact evaluation. Figure A3.6 displays the evolution of the value of the benefits generated by the project (i.e., the value of the DALYs averted) using the recommended parameters of a 3 percent discount rate and DALYs valued at three times the country’s per capita income. It shows that the benefits generated by the project increased after 2010, as the project gained force and expanded its coverage with the various AFs. Figure A3.6. Evolution of Benefits at Baseline Scenario 450000 400000 350000 300000 DALYs (000s) 250000 200000 150000 100000 50000 0 2006 2007 2008 2009 2010 2011 2012 2013 2014 Mean Estimate 3*GDP p.c. Upper Bound 3*GDP p.c. Lower Bound 3*GDP p.c. Source: Author’s calculations. Keeping in mind the limitations of this study, the economic analysis conducted shows that using the most plausible parameters, the project is likely to have generated positive returns 54 for the Congolese people. In fact, if the DALYs averted are valued at the recommended rate of three times the per capita income, the estimates done in this exercise indicate that the project provided value for money throughout the entire range of the 95 percent uncertainty interval, even when increasing the discount rate to 5 percent. 55 Annex 4. Bank Lending and Implementation Support/Supervision Processes Annex 5. Beneficiary Survey Results N.A. 56 Annex 6. Stakeholder Workshop Report and Results The Government of the Democratic Republic of Congo produced in March 2015 a completion report which was validated through a stakeholder workshop. The report confirms based on their own assessment as to storyline, ratings and lessons learned are in line with the Bank's assessment. They have identified lessons learned which in line for the most part with the Bank’s own report. Below are the lessons learned and recommendations from the Government’s report. 1. Recommendations for the Government/ Ministry of Public Health a) Negotiate with the World Bank and other partners for a continued financing of the project activities (without delays) in order to conserve and consolidate the achievements of the project; b) Negotiate the scaling up of the health zones reached in the new project for better coverage of the country; c) Consider a better integration of the institutional arrangements of the project in the health system. The teams of technical assistants of the new project should be integrated into the Ministry’s structures as well as the supported districts. d) Identify and contract out providers (public or private) in order to reach all the Provinces in DRC; e) Change the input approach which turned out to be inefficient as many commodities purchased through the project at the central level didn’t reach the facilities in time as planned. The Government/Ministry of Public Health could provide funding directly to the health centers which can pass their own requests once pre-qualified suppliers have been selected; f) Consider mechanisms at the provincial level for a better use of human resources. Initiatives with the private sector may allow a more efficient use of the health workers at different levels of the health system; g) Harmonize the World Bank disbursement procedures with national practices for relief of these procedures; h) Negotiate with the World Bank and other partners the extension of the PBF approach in other sectors other than health (eg education). 2- To the World Bank a) Sustaining the achievements of the project through the implementation of a new project using the performance based approach; b) Encourage a best practice approach of the performance-based financing strategy c) Consider the introduction of the PBF approach in other sectors other than health (eg education); d) Encourage the support to the victims of sexual violence; e) Alleviate funds disbursement procedures in the context of the harmonization/ alignment (Paris Declaration) 57 Annex 7. Summary of Borrower’s ICR and/or Comments on Draft ICR A consultation with representatives from the Ministry of Public Health on the Implementation Completion Report was held on September 28, 2015. The report was discussed with emphasis on the outcomes and the final performance ratings. The representatives agreed with the final ratings, after further explanations from the team and appreciated the discussion on the lessons learned described in the report. The government representatives’ highlighted additional lessons learned that were incorporated in the final version. Furthermore, the French version of the ICR report was shared with the Government who advised the Bank that the Ministry of Health didn't have any additional comments pertaining to the ICR and take notes of the project accomplishments as well as lessons learned 58 Annex 8. Comments of Cofinanciers and Other Partners/Stakeholders 59 Annex 9. List of Supporting Documents Berhan, Yifru, and Asres Berhan, 2014. “Skilled Health Personnel Attended Delivery as a Proxy Indicator for Maternal and Perinatal Mortality: A Systematic Review.” Ethiopian Journal of Health Sciences (Special Issue) September: 69–80. CCISD (Centre de Coopération Internationale en Santé et Développement). 2007. “Rapport Narratif Trimestriel Bandundu.” CCISD, Kinshasa. ———. 2014. “Rapport Troisième Trimestre Couvrant la Période du 1 Octubre au 31 Décembre, 2013.” CCISD, Kinshasa. ———. 2014. “Rapport Troisième Trimestre Couvrant la Période du 1 Janvier au 31 Mars, 2014.” CCISD, Kinshasa. CEMUBAC (Centre Scientifique et Médicale de l’Université libre de Bruxelles pour ses Activités de Coopération). 2010. “Rapport Trimestriel, Trimestre II, sur le Renforcement des Capacités du Niveau Intermédiaire du Système de Santé.” CEMUBAC, Kinshasa. ———. 2010, “Rapport Trimestriel, Trimestre IV, sur le Renforcement des Capacités du Niveau Intermédiaire du Système de Santé.” CEMUBAC, Kinshasa, COOPI (Cooperazione Internazionale). 2013. “Project de Communication de Masse à Base Communautaire sur l’Allaitement Exclusive et la Nutrition de la Femme Enceinte, Rapport Trimestrielle Mai – Juillet 2013.” COOPI, Kinshasa. ———. 2013. “Project de Communication de Masse à Base Communautaire sur l’Allaitement Exclusive et la Nutrition de la Femme Enceinte, Rapport Trimestrielle Aout – Septembre 2013.” COOPI, Kinshasa. ———. 2014. “Project de Communication de Masse à Base Communautaire sur l’Allaitement Exclusive et la Nutrition de la Femme Enceinte, Rapport Trimestrielle Octobre – Décembre 2013.” COOPI, Kinshasa. CREDES (Centre de Recherche en Développement et Santé). 2010. “Enquête sur les Performances à Mi-Parcours des Formations Sanitaires des Districts et Zones de Sante Ciblent du PARSS au Décembre 2009.” CREDES, Kinshasa. Dimomfu, L. 2011. “Cadre de Planification en Faveur des Populations Autochtones, Rapport Final.”PARSS, Kinshasa. Doucoure, D. 2011. “Audit de Gestion Environnementale pour les Projets de Construction: Rapport Final.” PARSS, Kinshasa. 60 Ecole de Santé Publique Université de Lubumbashi. 2012. “Rapport Final d’Enquête avant la Campagne de Distribution des MILD dans la Province du Katanga.” Université de Lubumbashi, Kinshasa. GIZ (Deutsche Gesellschaft fur Internationale Zusammenarbeit). 2014. “Achat de Performance dans les Districts de Lualaba, Haut Katanga et Sud Maniema, Rapport Final.” GIZ, Kinshasa. Graham, Wendy J., Jacqueline S. Bell, and Colin H. W. Bullough. 2001. “Can Skilled Attendance at Delivery Reduce Maternal Mortality in Developing Countries?” Studies in Health Services Organisation and Policy 17: 97–129. Jamison, Dean T., Prabhat Jha, and David Bloom. 2008. “Disease Control.” Copenhagen Consensus 2008 Challenge Paper: Diseases, Copenhagen Consensus Center, Copenhagen. Jamison, D.T., J.G. Breman, A. R. Measham, G. Alleyne, M. Claeson, D. B. Evans, P. Jha, A. Mills, and P. Musgrove, eds. 2006. Disease Control Priorities in Developing Countries. New York and Washington, DC: Oxford University Press and World Bank. Morel, Chantal M., Jeremy A. Lauer, and David B. Evans. 2005. “Cost Effectiveness Analysis of Strategies to Combat Malaria in Developing Countries.” The BMJ 331. MEASURE DHS. 2007. “République Démocratique du Congo Enquête Démographique et de Santé 2007, Rapport Final.” ICF International, Rockville, MD. ———. 2014. “République Démocratique du Congo Enquête Démographique et de Santé 2013–2014; Rapport Préliminaire.” ICF International, Rockville, MD. O. Connel, T. 2005. “Consultant Mission Report on Malaria in DRC.” World Bank, Washington, DC. “PARSS (2007–2014) Rapports d’Avancement du Project d’Appui à la Réhabilitation du Secteur de Santé.” Pulkki-Brännström, Anni Maria, Claudia Wolff, Niklas Brännström, and Jolene Skordis-Worrall. 2012. “Cost and Cost Effectiveness of Long-Lasting Insecticide-Treated Bed Nets - A Model- Based Analysis.” Cost Effectiveness and Resource Allocation 10 (5). SAPHIR Développement. 2015. “Projet d’Appui à la Réhabilitation du Secteur de la Santé (PARSS) Rapport d’Achèvement.” Saphir Consortium, Ouagadougou. Tan-Torres Edejer, T., R. Baltussen, T. Adam, R. Hutubessy, A. Acharya, D. B. Evans, and C. J. L. Murray, eds. 2003. Making Choices in Health: WHO Guide to Cost- Effectiveness Analysis. Geneva: World Health Organization. 61 UNFPA (United Nations Population Fund). 2014. “Rapport d’Avancement du Projet La Prévention et la Réponse aux Violences Sexuelles et Basées sur le Genre Faites aux Femmes, aux Hommes, aux Jeunes et aux Enfants dans les Zones Cibles du PARSS.” UNFPA, Kinshasa. UNICEF (United Nations Children’s Fund) DRC. 2013. “Shining a Light on Malaria in DRC: Support Distribution of Long-Lasting Insecticide-Treated Bed Nest in the DRC (Katanga, North and South Kivu) / Final Narrative and Funds Utilization Report.” UNICEF, Kinshasa. World Bank. Forthcoming. “Investir dans la Couverture Sante Universelle: Opportunités et Défis pour la République Démocratique du Congo.” Report xxxx, World Bank, Washington DC ———. 2005, 2011, 2011, 2012. “Development Grant Agreement Health Sector Rehabilitation Support Project.” Reports H-182, H-638, H-717, H-750, World Bank, Washington, DC. ———. 2005, 2011, 2011, 2012. “Project Appraisal Document Health Sector Rehabilitation Support and Additional Financing I, II and III.” World Bank, Washington, DC. ———. (2007–14) Implementation Status and Results Reports 1–18. ———. 2009. “Quality Assessment of Lending Portfolio Report: DRC Health Sector Rehabilitation Support Project.” World Bank, Washington, DC. ———. 2013. “DRC Health Systems and Financing: Haut-Katanga Performance-Based Financing.” Policy Note, World Bank, Washington, DC. 62 Annex 10: MAP 63