Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) Report Number: ICRR0022434 1. Project Data Project ID Project Name P143849 CG Health Sector Project Country Practice Area(Lead) Congo, Republic of Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-53570,TF-15744 28-Jun-2019 19,306,214.60 Bank Approval Date Closing Date (Actual) 20-Dec-2013 30-Nov-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 10,000,000.00 10,000,000.00 Revised Commitment 19,945,203.74 9,972,453.34 Actual 19,306,214.60 9,972,453.34 Prepared by Reviewed by ICR Review Coordinator Group Salim J. Habayeb Judyth L. Twigg Joy Maria Behrens IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Financing Agreement of 2/19/2014, p. 5, the objective of the project was to increase utilization and quality of maternal and child health services in targeted areas. The statements of objectives were identical in the PAD and ICR. b. Were the project objectives/key associated outcome targets revised during implementation? Page 1 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) No c. Will a split evaluation be undertaken? No d. Components 1. Improving utilization and quality of health services at health facilities through performance-based financing (PBF) (Appraisal US$107.5 million; Actual US$33.9 million). The component included the following: (a) Provision of PBF grants to health service providers for delivery of health subprojects to women and children: two health service packages were designed for use under the PBF approach for health centers and hospitals. Health service packages include a Minimum Package of Activities of 23 preventive and curative services for community and health center levels, and a Complementary Package of Activities with 18 services for first level hospitals. Service packages were designed based on international good practice. The component also included a quantified quality checklist for each level of service. (b) Strengthening capacities within the Ministry of Health and Population (MOHP), including health administration units, health verification teams, technical PBF unit, internal verification, and civil society organizations. The component also aimed at strengthening external verification through the provision of technical advisory services, goods, non-consulting services, training, and operating costs. 2. Strengthening health financing and health policy capabilities (Appraisal US$12.5 million; Actual US$5.5 million). The component included: (i) development of criteria for the introduction of fee waivers for the poor and vulnerable, fee exemptions for selected services, and development of tools for budget formulation; (ii) technical assistance and training to develop the policies contributing to the development of Universal Health Coverage; and (iii) strengthening M&E mechanisms. Note: The project was initially identified as “Health System Strengthening Project II” in the Financing Agreement and the PAD. Its title was changed to “Health Sector Project” at the project start-up, as was reflected in all ISRs. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost and financing. The original project cost was estimated at US$120 million, consisting of an IDA Credit of US$10 million (SDR 6.6 million), a grant of US$10 million from the Multi-Donor Trust Fund for Health Results Innovation, and US$100 million in government counterpart financing. At that time, projects of the Congo portfolio sought to use small amounts of IDA resources to leverage significant government funding from oil revenues. The total actual cost was only US$39.3 million due to a lack of counterpart funding Page 2 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) caused by plummeting oil revenues. The IDA Credit and the trust fund were fully disbursed (both aggregating at US$19.3 million), and actual government financing was US$20 million. Dates. Project appraisal was finalized on 11/25/2013, and the project was approved on 12/20/2013. It became effective only on 1/21/2015 (see Section 5). The Mid-Term Review was carried out on 1/23/2017. The project closed on 11/30/2019, five months beyond the original closing date of 6/28/2019. The project underwent three level-2 restructurings (on 5/29/2017, 3/20/2018, and 11/17/2019) that were triggered by financing gaps caused by insufficient counterpart funding. The restructurings revised disbursement arrangements, financing percentages of eligible expenditures, and allocations between disbursement categories. 3. Relevance of Objectives Rationale At appraisal in 2013, the Republic of Congo was one of the fastest growing economies in Sub-Saharan Africa, but health status indicators remained poor. According to the Demographic and Health Survey 2011- 12, the maternal mortality ratio was 426 per 100,000 live births, under-five mortality and infant mortality rates were 68 and 39 per 1,000 live births respectively, and the prevalence of chronic malnutrition was 24.4 percent in children less than 5 years. Utilization of services was low in both rural and urban areas (PAD, p. 4), and was explained by the burden of out-of-pocket spending through user fees and charges for medicines, low quality of services, limited range of available services, poor performance of health workers, and underfunding of health facilities. During 2012-2013, the Bank supported a PBF pilot that resulted in increased service utilization, thus encouraging the government to plan a scale-up of the PBF approach. At entry, the objectives were aligned with the Country Partnership Strategy (CPS) FY13-FY16 that included support to achieving better health outcomes and improving basic public services. The ICR (p. 7) stated that the approach of CPS FY13-FY16 was to use the country’s large oil revenues to promote a competitive and diverse economy that facilitated poverty reduction with improvements in health and social outcomes, and increases in accessibility to public services. The Bank envisaged the project to be designed, like most projects in the Congo portfolio at that time, to use a small amount of IDA resources to leverage significant government funding from oil revenues. Page 3 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) The objectives remained aligned with National Health Development Plan (2020-2024), including for access to quality health care services, health sector management strengthening, and decentralization with transfer of resources to regional and local health stakeholders. At project closing, the objectives remained fully consistent with the Country Partnership Framework (CPF) FY20-FY24, specifically under Focus Area 2 to "Build Human Capital and Enhance Resilience for Social Inclusion and Sustainable Growth." Under CPF Objective 2.1 on "Improved Quality and Access to Health Delivery Systems”, the CPF stated that the Bank’s program will continue to focus on improving quality and utilization of maternal and child health services in targeted areas through PBF which will be scaled up; that the program is also directly focusing on geographical targeting to rural areas and marginalized populations where services are needed to address disparities and ensure access to quality healthcare; that improving quality of health services is expected to increase higher volume of curative and preventive services; and that health system strengthening will continue through policy dialogue on issues related to health financing and importantly supporting the Ministry of Health to improve its use of public funds. The CPF also stated that the project will collaborate with the Lisungi Safety Nets Project to use its technique for targeting the poorest households for fee exemptions. CPF indicators included similar indicators used by the project, such as immunization for children, antenatal care visits for pregnant women, nutrition services, and fee exemptions for the poor. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Increase utilization of maternal and child health services in targeted areas Rationale Targeted areas involved seven out of 12 health departments in the country with a total population of about 3.8 million people or 86 percent of the population. Direct beneficiaries included women and children under-5. According to the PAD (p. 9), health departments were chosen based on their involvement with a previous PBF pilot, either as pilot executers or as control groups, and on the desire to introduce urban PBF in two other departments, i.e., Brazzaville and Pointe Noire. Theory of change Page 4 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) It was reasonably expected that PBF financing of maternal and child health services, capacity building, implementation of quality standards, and fee exemptions would plausibly lead to increased availability of health services that would contribute to increased utilization and quality of health services. These outcomes would be further facilitated by capacity building in management and health policy, and by strengthening M&E and the use of information and communication technology. Outputs and Intermediate Results: Project activities consisting of the provision of PBF grants, application and monitoring of quality checklists for services provided by PBF facilities, training of health workers in maternal and child health, verification of results, and provision of medicines, vaccines, goods, vehicles, and motorcycles contributed to the following:  Under the project, 1,199,715 people received essential health, nutrition, and population services.  The number of new and existing acceptors of modern contraceptive use increased from a baseline of 3,600 in 2014 to 96,615 acceptors in 2018, short of the target of 218,000 acceptors.  The number of poor people benefiting from fee exemptions reached 51,905 persons, short of the target of 850,000.  The percentage of pregnant women who received two doses of Intermittent Preventive Treatment of malaria increased from a baseline of 15 percent in 2014 to 62.6 percent in 2018, exceeding the target of 30 percent.  The percentage of women counseled and tested for HIV increased from a baseline of 22 percent in 2014 to 86.9 percent in 2018, exceeding the target of 30 percent.  The number of health workers receiving training in maternal and child health was 2,523 persons, exceeding the target of 1,666 workers. Training was intended to be assessed by knowledge tests and competency tests, but training assessment was not realized because of the lack of related baselines and funding. Outcomes The project was able to reach only about 48 percent of the targeted population with PBF activities rather than the originally envisaged 86 percent of the population (ICR, p. 13), given that only one-third of the total planned budget materialized (US$40 million out of US$120 million). The economic crisis that started in mid- 2014 following a sharp decline in oil prices and oil revenues resulted in the government being able to fulfill only US$20 million of the total US$100 million that it had initially committed to the project (ICR, p. 12 and p. 26). According to task team clarifications on 12/16/2020, the provision of health services under the PBF scheme ended in April 2018 due to the lack of funding. Page 5 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) The number of children fully immunized increased from a baseline of 70,284 children in 2014 to 112,358 children in 2018, short of the target of 330,300 children. The percentage of children aged between 6 months and 59 months receiving nutritional services increased from a baseline of 15 percent in 2014 to 23 percent in 2019, short of the target of 45 percent. The percentage of pregnant women having at least 3 antenatal care visits before delivery increased from a baseline of 50 percent in 2014 to 78.2 percent in 2018, close to the target of 80 percent. The percentage of new curative consultations per capita per year increased from a baseline of 20 percent in 2014 to 27 percent in 2018, short of the target of 35 percent. Rating Modest OBJECTIVE 2 Objective Increase the quality of maternal and child health services in targeted areas Rationale Theory of change The same as above under Objective 1. Quality checklists served as a quality assessment tool that included more than a hundred criteria measurements on various aspects of service provision, such as hygiene, drug availability, drug management, rational prescriptions, clinical care aspects, equipment availability, financial management, and laboratory functions (ICR, p. 10, and PAD, p. 11). Outputs: the same as under Objective 1, above. Note on quality checklists: The use of quality checklists was an integral part of PBF. Checklists contained criteria for ten services. The information provided granular data on each service. At the end of each quarter's Page 6 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) assessment, a summary of main recommendations was discussed with health facility managers and their teams. Checklists were also used by facilities for regular internal self-assessment to determine the level of progress made before the next quarterly assessment. All assessment information was compiled in a PBF database platform Outcomes The ICR reported that the average score of quality checklist increased from a baseline of 28 in 2015 to 69.7 in 2018, exceeding the target score of 63 (ICR, p. 19, and Results Framework, ICR, p. 36). However, taking into consideration the diminished project reach (see outcomes under Objective 1, above) for improving both utilization and quality aspects, the achievement of the objective is considered borderline substantial. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The aggregation of achievements of both objectives to increase utilization and quality of maternal and child health services in targeted areas indicate that the project partly achieved its objectives; therefore, overall efficacy is rated modest. Overall Efficacy Rating Primary Reason Modest Low achievement 5. Efficiency At appraisal, the PAD estimated the cost-effectiveness ratio of project-supported services at US$70 per disability-adjusted life year (DALY) averted, indicating favorable cost-effectiveness (PAD, p. 23). At project closing, the ICR repeated a cost-effectiveness analysis that estimated cost-effectiveness at US$49 per DALY for services provided at health centers and at US$66 per DALY for services provided at hospitals. The ICR carried out a separate cost-benefit analysis that estimated the number of lives saved through the project, and contrasted benefits with costs. The project’s internal rate of return varied between a low of 4.9 percent and a high of 5.6 percent per year, depending on the assumptions made (ICR, p. 45). However, the ICR Page 7 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) did not provide detailed information on assumptions made, discount rates used, or sensitivity analysis. The reported benefit to cost ratio was favorable, with a low estimate of 2.04 and a high estimate of 3.42. Implementation inefficiencies ranged from significant to major. Inefficiency from the perspective of Bank funding was undermined by a considerable drop of 80 percent in counterpart financing (ICR, p. 20), extended delays in implementing activities, inability to continue activities, staffing issues, and shortcomings in fiduciary aspects. There was a lag in starting implementation, as the project became effective in January 2015, 14 months after approval, following protracted discussions largely caused by financial constraints that resulted from declining oil prices in the late months of 2014. The complexity of implementing the PBF program, even in a context with some PBF exposure under the pilot, meant that a longer preparatory phase than initially planned was needed. Putting in place coordination mechanisms, contracting and verification arrangements, training, and evaluation arrangements contributed to implementation delays. Commitment to the PBF scheme was variable at MOHP level. The ICR (p. 34) reported that the Inspector General for Health commitment and some government branches, that implemented their contracts, showed commitment, but that some other departments within MOHP either implemented their contracts slowly or refused to integrate performance frameworks in their work (ICR, p. 34). Staff turnover due to poor selection of profiles and prolonged absence of an in-country project coordinator slowed implementation in 2015 and 2016. Project coordination improved following the mid-term review of 2017, including with the hiring of a new coordinator. Shortcomings in fiduciary aspects and governance negatively affected implementation, and financial management performance was rated by the project team as moderately unsatisfactory throughout most of the project period (see Section 10 for details). The ICR (p. 21) reported that procurement of commodities was inefficient. Difficulties in procuring medicines and consumables were reported throughout implementation, largely due to delays in obtaining approvals from MOHP (ICR, p. 28). The project purchased a buffer stock of drugs for the Central Medical Stores that took 2.5 years to arrive in the country, and a container remained in customs one month after project operations were stopped (ICR, p. 21). Efficiency Rating Modest Page 8 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 86.00 ICR Estimate  4.90  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is rated high, as the objectives remained fully aligned at project closing with the Country Partnership Framework and the country’s National Health Development Plan. Efficacy is rated modest, as the project partly achieved its development objectives, largely due to reduced government funding and decreased coverage. Efficiency is rated modest because of significant shortcomings in the efficiency of implementation. Therefore, the overall outcome is rated moderately unsatisfactory, indicative of significant shortcomings in the project’s overall preparation, implementation, and achievement. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome The two main areas of risks to development outcomes are related to macro-economic constraints and sector- specific risks. The country’s undiversified economy renders it vulnerable to sharp oil market fluctuations. Financial constraints may further exacerbate the pattern of low health budget execution where execution rates remained low at an average of 40 percent during the period 2015-2019. Institutional and governance weaknesses in the health sector remain significant. Nevertheless, since project closing, the ICR (p. 33) noted that there was a stronger national engagement for implementing PBF throughout the country, underlined by explicit commitment to PBF in the National Health Development Plan 2018-2022, and by the adoption of a decree expanding PBF in the health sector. The decree was signed by the Minister of Health in June 2019, setting up laws to expand PBF nationwide. Such expansion is currently being implemented in Likouala region through the LISUNGI Safety Nets System Project. A follow-on project is currently under preparation, the Kobikisa Health System Strengthening Project (P167890), according to task team clarifications on 12/17/2020. The proposed new project would support the Page 9 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) delivery of and access to quality maternal and child health services by building on the PBF approach of this project, while incorporating measures to enhance access for the poor. The follow-on project would use performance-based conditions to address critical steps and systemic issues in the results chain, with emphasis on those that would improve governance and management of public financing. The tentative estimated cost is US$100 million and a tentative Board approval date is planned for Q4 FY21. 8. Assessment of Bank Performance a. Quality-at-Entry According to the PAD (p. 8), preparation built on experience gained from a PBF pilot implemented in 2012 in three departments (Niari, Pool, and Plateaux). Project design was also informed by good practices and lessons learned from PBF operations in Africa, including from Rwanda, Burundi, DRC, Zambia, Zimbabwe, Nigeria, Benin and Chad. The project design considered lessons related to improving service quality, incentives to expand coverage of essential health services, monitoring, verification, and reducing financial access barriers (PAD, pp. 16-17). MOHP was responsible for project management and coordination through a Technical PBF Unit supported by MOHP Directorates. Financial management was the responsibility of an MOHP Implementation Unit that was coordinating implementation of the Bank-supported Health Development Project-I, and the unit was performing adequately, according to the PAD. However, the financial management assessment carried out by the Bank concluded that overall residual financial management risk was substantial due to the country’s weak capacity context (PAD, p. 25). The procurement unit of MOHP was planned to handle procurement functions with its existing capacities. Its performance and capacity were assessed by the Bank to be moderately satisfactory, the related risk was considered substantial, and mitigation measures were prepared, including by developing a procurement plan for the first 18 months of the project, updating record management, and providing staff training on Bank procedures and guidelines (ICR, p. 28). At the same time, preparation underestimated the timelines required to set the foundation for scaling up PBF operations (ICR, p. 31), such as for recruitment, training, PBF management, and verification arrangements. Although many facilities had already participated in the PBF pilot, there was a two-year gap between the pilot and project implementation, and there was turnover of previous health personnel. With regard to the high level of counterpart financing at 83 percent of project resources, and as stated in Section 3, the Bank approach in the Congo portfolio at that time was to use a small amount of IDA resources to leverage significant government funding from oil revenues, although the risk of counterpart funding was raised during project preparation (Restructuring Paper, No. RES27190, 5/29/2017, p. 4). Page 10 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision According to the ICR (p. 31), the task team was proactive and consistent in its supervision and implementation support functions. The task team recruited an in-country health specialist to enhance engagement with the government and to provide-hands-on support. The ICR stated that reporting was adequately documented in ISRs and Aide-Memoires, although there were lapses in reporting on safeguards. Procurement support was extensive, including training provided by a procurement specialist. According to the ICR (p. 31), regular supervision missions were undertaken and dialogue was maintained with different stakeholders at various levels and throughout project implementation. The task team adequately facilitated and managed three restructurings, largely for reallocations and to revise financing ratios in response to government financial constraints. In addition, the team prepared in March 2017 a restructuring paper to adjust the results framework in the context of reduced project funds. The proposed restructuring also intended to add a third component to respond to government needs in addressing financing gaps in acquiring vaccines and essential drugs, including antimalarials and antiretrovirals, mosquito nets, and other HIV/AIDS commodities. However, Bank Management decided not to pursue the planned restructuring, and instead, to incorporate the amendments instead into a proposed additional financing of US$20 million. At that point in time, the Bank portfolio in the country was suffering from a general government inability to meet its counterpart funding commitments across all sectors. Following further discussions with the government, a Bank management decision was made to prepare a new health project rather than pursuing additional financing (ICR, pp 14-15) for the project. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Project objectives were clearly stated and the indicators reflected stated objectives on utilization and quality of maternal and child health services. Some baselines were available during preparation and others were determined through a baseline survey conducted shortly after project approval in 2014. MOHP had Page 11 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) institutional responsibility for M&E, and it planned for three data collection sources: the existing health management information system, PBF database, and impact evaluation. b. M&E Implementation There were significant shortcomings in M&E implementation. Internal verification arrangements were put in place, but only the ex-ante verification was performed, thus diminishing data robustness. The project was not able to conduct the planned 2017 hospital quality review or the provincial health department and purchasing agency performance evaluations. External verification was conducted only once, and the contract was not renewed after mid-2017 (ICR, p. 27). The planned impact evaluation was not completed. c. M&E Utilization Data were used for regular project monitoring and allowed the project to monitor activities of health facilities to determine whether to keep them under PBF contracts based on performance. At the same time, the ICR (p. 28) reported that "that limited decisions were made at the central level to address issues encountered based on data generated by health facilities", but the ICR did not clarify what this meant. M&E Quality Rating Modest 10. Other Issues a. Safeguards Environmental safeguards. The project was classified as Category B, as it triggered Environmental Assessment (EA) OP/BP 4.01 because increased use of medical facilities was expected to generate more medical waste. Reporting on the Environmental Assessment Category lacked attention in the PAD, Integrated Safeguards Data Sheet (ISDS), ICR, and Restructuring Papers. In the PAD and ISDS at appraisal, reporting was inconsistent. Page ix of the PAD classified the project as EA Category C, while page 27 noted that the safeguard policy on Environmental Assessment (OP/BP 4.01) was triggered. Similarly, the EA Category was recorded as C in the appraisal ISDS (p.1, Report No. ISDSA5999, disclosed on 11/20/2013), while page 8 of the same ISDS stated that EA OP/BP 4.01 was triggered. In the ICR, the EA Category on page 1 was blank, but the ICR (p. 29) stated that the project was classified as Category B, as it triggered Environmental Assessment OP/BP 4.01. The EA Category was incorrectly recorded as C category in the Restructuring Paper of 2017, and was not recorded in the two subsequent Restructuring Papers of 2018 and 2019. The National Medical Waste Management Plan (NMWP) was updated to meet World Bank requirements. The plan included provisions for training, management of medical waste, and investments in equipment. Subcontractors were recruited to conduct biomedical waste management in accordance with the NMWP. Page 12 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) Compliance was not consistent, and there were delays in the preparation of safeguards plans, implementation of environmental protection measures, and monitoring and reporting. The task team urged regular supervision by the Project Implementation Unit. Stakeholders received training on how to better implement environmental and safeguards policies with a focus on health and safety aspects related to waste management. The first safeguards supervision mission was organized in November 2016. During the safeguard supervision mission in 2018, the task team identified issues related to odor nuisances from incinerators and insufficient knowledge of NMWP by health facility personnel. The task team made recommendations for improvement that were agreed with the government team, including: (i) rehabilitation of incinerators in health facilities situated near community neighborhoods; (ii) access ramps to buildings; and (iii) making the NMWP plan known to health centers. Since some reference hospitals and large health centers did not have a mechanism to manage waste, related functions were outsourced to local waste management companies, with the recommendation that the Directorate of Health and the District Chief ensure that contracted companies apply good practices for biomedical waste management. The ICR (p. 30) stated that the project did not submit an environmental and social monitoring report to confirm compliance with recommendations and points agreed with the Bank. An overall safeguard rating was not recorded in the Bank’s Operations Portal. However, according to task team clarifications (12/16/2020), the team maintained regular contacts with Environmental and Social Framework colleagues during project implementation, and there were no issues brought up other than holding PBF facilities accountable for the quality of infection prevention and waste disposal measures. The task team also stated that observance of environmental guidelines was part of quality checklists that were used to pay for performance in contracted PBF facilities. Social safeguards. The safeguard policy on Indigenous Peoples OP/BP 4.10 was triggered. An Indigenous Peoples Planning Framework (IPPF) was prepared and disclosed in the country in November 2013 and at the external site of the World Bank. The Bank's Social Development specialists provided technical support. Although IPPF implementation was delayed and there was no Grievance Redress Mechanism, overall performance of social safeguards implementation was considered satisfactory by the task team. b. Fiduciary Compliance Financial management performance had multiple shortcomings and was rated by the task team as moderately unsatisfactory throughout most of the project period. Main issues observed during project implementation included: (i) weaknesses in the internal control system; (ii) prolonged vacancy of the financial management specialist position from 2017 to project closing; (iii) a 2016 audit report that was not accepted by the Bank due to noncompliance with World Bank standards; (iv) poor planning and budget management that led to low budget execution throughout the first two years of the project and the need for a retroactive extension of the project in August 2019; (v) non-functioning of the steering committee throughout the project life that led to all decision-making resting solely with the project coordinator; and (vi) poor relationship between the fiduciary staff of the project and the line ministry, rendering exchanges of information and collaboration difficult to manage. Over a period of several months, designated accounts Page 13 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) were frozen and project activities were halted due to poor financial management. Procurement shortcomings reported by the ICR are discussed in Section 5. c. Unintended impacts (Positive or Negative) None reported. d. Other -- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome Unsatisfactory Unsatisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory This ICR Review rated the Quality of M&E as modest because of significant Quality of M&E Substantial Modest shortcomings, including unaccomplished verifications and impact evaluation. Quality of ICR --- Substantial 12. Lessons The ICR identified some lessons and recommendations, including the following lessons re-stated by IEG: The support system surrounding Performance Based Financing is key to achieving results. In this project, the support system included tools that were put in place during implementation such as training, coaching, supervision, verification mechanisms, statistical record keeping, treatment protocols, and scorecards. Facilities maintained records of accounts, performance, and rewards. PBF was further supported by a law enacted in 2019, paving the way for its institutionalization. Page 14 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) An imbalance in human resources distribution can be an adverse consequence of PBF. Contract structures attracted health workers from other health facilities that were not under the PBF program, creating an imbalance in staffing distribution. In addition, the task team identified a lesson during interactions with IEG: Setting up a budget for essential health services program based on a very volatile funding stream is likely to have negative consequences on the services provided. In this project, health services under PBF were discontinued due to the lack of funding that resulted from a sharp decline in oil prices and revenues, and the decline was consequential because the Borrower's counterpart financing constituted 80% of total project costs. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR was results-oriented, candid and tightly written. It provided a detailed critique of the project experience. The report illustrated a simplified theory of change underlying the project trajectory toward intended outcomes to increase utilization and quality of maternal and child health services in targeted areas of the country. The quality of analysis was adequate, and the quality of evidence was variable, but shortcomings were rooted in the project and not in the ICR that aptly pointed out related weaknesses. The ICR followed guidelines. It was internally consistent in general, but its reporting on the EA category in the safeguards section was inconsistent, and the recording of the EA category was missing in the ICR's Data Sheet. The ICR offered specific lessons derived from project experience. a. Quality of ICR Rating Substantial Page 15 of 16 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CG Health Sector Project (P143849) Page 16 of 16