Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) Report Number : ICRR0021290 1. Project Data Project ID Project Name P099930 MZ-Health Service Delivery SIL (FY09) Country Practice Area(Lead) Additional Financing Mozambique Health, Nutrition & Population P125477 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-45820,IDA-51940,TF- 28-Feb-2014 101,284,490.87 58155,TF-96375,TF-96399,TF- 96505 Bank Approval Date Closing Date (Actual) 16-Apr-2009 31-Dec-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 44,600,000.00 28,320,816.54 Revised Commitment 81,600,000.00 27,313,153.66 Actual 77,558,988.75 25,380,440.85 Prepared by Reviewed by ICR Review Coordinator Group Salim J. Habayeb Judyth L. Twigg Eduardo Fernandez IEGHC (Unit 2) Maldonado 2. Project Objectives and Components a. Objectives Original objectives: According to the Financing Agreement dated 7/15/09, the objectives were to: (a) reduce child mortality; (b) reduce maternal mortality; (c) reduce the burden of malaria; (d) reduce the prevalence of tuberculosis; and (e) reduce inequity in the access to health services in the Recipient’s territory. Page 1 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) The ICR and Project Appraisal Document's (PAD’s) statements of objectives were identical except for their substitution of “Recipient’s territory” with “Mozambique”. Revised objectives: Under an additional financing (AF) of 1/24/13, the "Amended and Restated Financing Agreement" of 3/25/13 revised the project objectives, and added nutrition aspects as follows: to increase access to, and utilization of, maternal and child health and nutrition services in target areas in the Recipient's territory. Hence, this ICR Review applies a split evaluation. Also, in the context of the project's one-year extension under a level 2 restructuring on 12/30/16, three associated outcome targets were revised upwards on 12/30/16, thus increasing the project ambition. The target for direct project beneficiaries was reduced, although its target concerning the percentage of female beneficiaries remained unchanged. The final number of beneficiaries exceeded both the original and revised targets. In view of the above, this ICR Review does not apply a second split evaluation for the 2016 revisions. Project scope: Under the original objectives, the scope was specified as "the Recipient’s territory". Under the 2013 AF, the project included only three Northern Provinces (Cabo Delgado, Niassa and Nampula). These three provinces had high poverty and unfavorable health indicators (see Section 3). b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 24-Jan-2013 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components Original components Note: The total actual cost of US$102.94 million was provided in the ICR, but not actual costs by component. 1. Improvement in service delivery (Appraisal US$42.6 million; Revised US$80 million). Page 2 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) • Provision of training and technical assistance to district health staff in the three Northern provinces of Cabo Delgado, Nampula and Niassa to enhance management and planning capacities, including organization and supervision of outreach and community-oriented activities. • Provision of training and technical assistance to health workers in the three provinces to improve health services provision. • Provision of goods and training to benefit public health facilities in the three provinces to enhance malaria control activities. • Rehabilitation of health facilities in the three provinces. • Construction/renovation of up to twenty-five health centers and two houses for personnel of the Ministry of Health (MOH). 2. Boosting of the National Malaria Control Program (Appraisal US$13.5 million; Revised US$15.5 million). • Provision of housing rental and training for provincial malaria focal points. • Training program for monitoring and evaluation on malaria control. • Provision of technical assistance and goods for vector control to support malaria treatment policy. • Provision of financing for essential drugs, long lasting treated nets, rapid diagnostic tests, microscopes, equipment and vehicles. • Construction of storage facilities. • Provision of training on public health operations. • Provision of assistance to develop risk maps through the supply of software and hardware to estimate population at risk using geographical information techniques. • Setting up sentinel sites for malaria surveillance in the three provinces. • Development of behavior change communication materials to sensitize the population to practices that help reduce malaria transmission. 3. Preparation of health sector investment plan (Appraisal US$0.5 million; Revised US$7.40 million). Provision of technical assistance to MOH staff for designing a 10-year health sector investment plan and for developing an implementation plan for the first five years of the investment plan. 4. Capacity building and operating costs (Appraisal US$15.80 million; Actual US$6.50 million). •Provision of training, technical assistance and goods to MOH staff to develop a health system observatory for the purposes of evaluating the evolution of health systems in the three provinces. Page 3 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) • Provision of administrative and operational support to three provinces, including the hiring of an operations assistant and a monitoring and evaluation (M&E) assistant for each of the provinces; provision of staff training at the provincial health departments for sampling techniques; provision of hardware, software and office equipment for M&E activities; and provision of training in financial management to district health staff. • Financing of operating costs. Component revisions At the 2013 AF, Component 1 was expanded with the addition of a new subcomponent on nutrition, which consisted of community-based activities targeted at pregnant and lactating women, adolescent girls and children up to 24 months of age, advocacy campaigns, and provision of nutrition commodities and equipment (ICR, p. 14). Nutrition services were also specified in the revised objectives (see Section 2a). Component 3 was renamed as Strategic Planning and Capacity Building to more accurately reflect the purpose of its activities, and the revision encompassed capacity building activities that were previously included under Component 4, which was also renamed as Project Management and Operating Costs. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Cost and Financing. The original project cost was estimated at US$72.4 million. It consisted of an International Development Association (IDA) Credit of US$44.6 million equivalent and cofinancing from the following development partners: US$15.6 million grant from the Canadian International Development Agency; US$7.9 million grant from the Russian government; and US$4.3 million from the Swiss Development Cooperation. No direct Borrower financing was planned. Additional financing on 1/24/13 brought an additional IDA Credit of US$37 million equivalent, raising the total estimated project cost to US$109.4 million. The final cost was US$102.94 million, partly explained by exchange rate fluctuations and grant revisions. Dates. The project was approved on 4/16/09 and became effective only on 6/11/10 (see Section 5). A mid-term review was carried out on 6/22/12, followed by restructuring with AF and project extension on 1/24/13. A second restructuring to further extend the project was undertaken on 12/30/16. The project closed on 12/31/17, about four years beyond the original closing date of 2/28/14. 3. Relevance of Objectives Rationale Page 4 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) At appraisal, Mozambique faced significant health challenges and geographic inequalities in health service coverage. The Northern provinces of Cabo Delgado, Niassa and Nampula suffered from higher rates of infant mortality, under-five mortality, fertility and undernutrition than the rest of the country. For example, under-five mortality rates in the three provinces were 242, 218, and 210 per 1,000 live births respectively, compared to less than 100 per 1,000 live births in Maputo (ICR, p. 6). Cabo Delgado and Nampula had the highest rate of chronic undernutrition in children under-5, exceeding 50%, as compared with 44% in all of Mozambique. Malaria was a major public health problem contributing to a large proportion of mortality in children and mothers (PAD, p. 1). The original objectives were aligned with the Country Partnership Framework (CPF 2017-2021) discussed below, but they were too ambitious for the planned time frame of the project (ICR, p. 16) and their scope misstated (see Section 2a), thus necessitating a project restructuring to bring the objectives in line with what the project actually intended to achieve. The revised objectives remained consistent with CPF 2017-2021, and specifically, they remained aligned with CPF’s Objective 6 for Improving Health Service Delivery under Focus Area 2 for Investing in Human Capital. According to the CPF, its program will have a strong focus on investing in a child's early years and will seek to contribute to the improvement of service delivery in the areas of reproductive, maternal, newborn, and child health while strengthening health systems performance. However, the ICR (p. 16) noted that, as the quality of services continues to be challenging, the development objectives could have also included a quality dimension to health services, in addition to increased access and utilization. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Reduce child mortality in the Recipient's territory. (Original objective, dropped in 2013) Rationale Training and technical assistance to district health staff and health workers to promote service delivery; provision of goods to health facilities; construction of health facilities; administrative and operational support; strengthening malaria control services and the provision of behavior change communication materials for malaria could reasonably be expected to increase access and utilization of primary health care by children and would plausibly contribute to reducing child mortality in the long run and in areas where such interventions are implemented. Page 5 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) Outcomes: No information was provided. According to the ICR, data required to monitor progress were difficult to obtain, as they relied upon infrequent household surveys such as the Demographic and Health Survey. The most recently available statistics on mortality rates, disaggregated by province, were from the Demographic and Health Survey of 2011, coinciding with the first year of project implementation. For subsequent years until project closing in 2017, data included only national modelled estimates (ICR, p. 16) which, without provincial disaggregation, would be difficult to attribute to project interventions. The ICR (p. 16) also stated that the effects of project interventions were unlikely to be realized during the project lifespan. Rating Negligible PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Reduce maternal mortality in the Recipient's territory. (Original objective, dropped in 2013) Rationale Training and technical assistance to district health staff and health workers to promote service delivery; provision of goods to health facilities; construction of health facilities; administrative and operational support; strengthening malaria control service delivery to pregnant women and the provision of behavior change communication materials for malaria could reasonably be expected to increase access and utilization of primary health care by women and may plausibly contribute to reducing maternal mortality in areas where such interventions are undertaken, although the reduction of maternal mortality requires an added focus on essential obstetric care. Outcomes No information was provided. As stated above under Objective 1, the ICR stated that data required to monitor progress were difficult to obtain, as they relied upon infrequent household surveys such as the Demographic and Health Survey. The most recently available statistics on mortality rates, disaggregated by province, were from the Demographic and Health Survey of 2011, coinciding with the first year of project implementation. For subsequent years until project closing in 2017, data included only national modelled estimates (ICR, p. 16) which, without Page 6 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) provincial disaggregation, would be difficult to attribute to project interventions. The ICR (p. 16) also stated that the effects of project interventions were unlikely to be realized during the project lifespan. Rating Negligible PHREVDELTBL PHEFFICACYTBL Objective 3 Objective Reduce the burden of malaria in the Recipient's territory. (Original objective, dropped in 2013) Rationale Training on malaria control activities; distribution of antimalarials, bed nets and chemicals for vector control; construction of storage facilities for malaria control; M&E training; provision of drugs, long-lasting nets, rapid tests and behavior change communication materials for malaria could reasonably be expected to facilitate intermittent preventive treatment by pregnant women and would plausibly reduce the burden of malaria. Outcomes No information was provided. Rating Negligible PHREVDELTBL PHEFFICACYTBL Objective 4 Objective Reduce the prevalence of tuberculosis in the Recipient's territory. (Original objective, dropped in 2013) Rationale A clear theory of change was not established. Outcomes Page 7 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) No information was provided. Rating Negligible PHREVDELTBL PHEFFICACYTBL Objective 5 Objective Reduce inequity in access to health services in the Recipient's territory. (Original objective, dropped in 2013) Rationale A clear theory of change was not established. Outcomes No information was provided. Rating Negligible PHREVDELTBL PHEFFICACYTBL Objective 6 Objective Increase access and utilization of maternal health services in three targeted provinces. (New objective added in 2013) Rationale Training and technical assistance provided to district health staff and health workers in the three Northern provinces to promote health services delivery; rehabilitation and construction of health facilities; provision of goods and supplies to health facilities; strengthening malaria control services, promoting intermittent Page 8 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) preventive treatment for pregnant women, and the provision of behavior change communication materials for malaria could plausibly contribute to increased access and utilization of maternal health services. Intermediate results • The percentage of pregnant women who received a complete dose of iron folic acid tablets increased from an unknown baseline to 48.2% in 2017, exceeding the original target of 20%, but short of the target of 63%. • Pregnant women receiving a first ante natal care visit decreased from a baseline of 497,750 in 2009 to 426,271 in 2017, exceeding the original target of 383,231, but short of the revised target of 593,000 (However, the ICR, p. 37, stated that the achievement in 2017 reflected only the first nine months of the final project year, whereas the target of 593,000 was an annual target. Hence, the reported number of 426,271 would have likely been higher if the remaining three months were also considered). The following inputs and intermediate results are also applicable to Objectives 6, 7, and 8 below: • The number of Agentes Polivalentes Elementares (multi-purpose workers) who were trained reached 1,732 workers in 2017, exceeding the original target of 1,100 and the revised target of 1,732. • The number of health personnel trained reached 796 in 2017, exceeding the original target of 300 and the revised target of 374 trainees. • Annual outreach visits increased from a zero baseline in 2009 to 4,460 in 2017, exceeding the target of 2,210. • 18 health facilities were constructed or renovated and equipped, attaining the target. Outcomes • The percentage of institutional deliveries in the three project provinces increased from a baseline of 50.8% in 2009 to 92.5% in 2017, exceeding both the original target of 58% and the revised target of 90%. • The percentage of pregnant women who received a second dose of intermittent preventive treatment for malaria at antenatal clinics increased from a baseline of 18.6% in 2009 to 70.3% in 2017, exceeding the original target of 50%, but short of the revised target of 78%. • The percentage of females (among all direct project beneficiaries) reached 76.1% in 2017, almost attaining the target of 80%. Page 9 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 7 Objective Increase access and utilization of child health services in three targeted provinces. (New objective added in 2013) Rationale Training and technical assistance provided to district health staff and health workers in the three Northern provinces to promote health services delivery; rehabilitation and construction of health facilities construction; provision of goods and supplies to health facilities; strengthening malaria control services and the provision of behavior change materials for malaria could plausibly contribute to increased access and utilization of child health services. Intermediate results Children receiving a dose of vitamin A increased from a baseline of 583,564 in 2009 to 1,225,930 in 2017, exceeding the original target of 1,066,594 and almost attaining the revised target of 1,385,200 (However, the ICR, p. 38, stated that the achievement in 2017 reflected only the first nine months of the final project year, whereas the target of 1,385,200 was an annual target. Hence, the reported result of 1,225,930 would have likely been higher if the remaining three months were also considered.) Outcomes Only one outcome-linked indicator on immunization was provided to reflect access and utilization of child health care services: the percentage of children vaccinated with Pentavalent Vaccine in the first year of life (DPTHepHib3, which includes meningitis-Haemophilus influenzae type B and Hepatitis B) increased from a baseline of 45% in 2009 to 97.3% in 2017, exceeding the target of 90%. Also, the ICR stated that the number of fully immunized children (including for polio, BCG-TB, and measles per TTL clarifications on 9/20/18) reached 235,201 in 2017, short of the revised target of 316,645 (However, the ICR, p. 38, stated that the achievement in 2017 reflected only the first nine months of the final project year, whereas the target of 316,645 was an annual target. Hence, the reported number of 235,201 would have likely been higher if the remaining three months were also considered). Rating Substantial PHREVDELTBL Page 10 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) PHEFFICACYTBL Objective 8 Objective Increase access and utilization of nutrition services in three targeted provinces. (New objective added in 2013) Rationale Implementing community-based nutrition activities, growth monitoring and promotion activities would plausibly result in increased access and utilization of nutrition services by women and children. Intermediate results • The percentage of children aged 0-24 months (% who are female) who attended at least one Growth Monitoring and Promotion service during the two preceding months increased from a baseline of 0 in 2013 to 35.4% in 2017, short of the target of 80% (the ICR, p. 39, stated that the percentage achieved for both sexes was 67.1%). • No results were provided for the percentage of female children aged 6-24 months who received at least 60 sachets of micronutrient powder during the two preceding months, as disaggregation by gender was not available (ICR, p. 40). The percentage attained in both sexes in 2017 was 34.6%, short of the target of 80%. Outcomes The ICR (p. 17) stated that it was not possible to conclusively assess the level of achievement due to the lack of routine data from all three target provinces and the lack of an endline survey. The ICR stated that these shortcomings translated into the "impossibility to quantitatively assess the impact" of project activities on nutrition (ICR, p. 31). Only partial routine data from two of the three provinces indicated improvement in coverage of infant and young child feeding (IYCF) practices consisting of food diversity, feeding frequency, and consumption of breast milk or milk. In a study conducted among 404 beneficiaries in Nampula, the percentage of children aged 0-24 months fed in accordance with all three IYCF practices increased from a baseline of 24.5% in 2013 to 80.8% in Nampula and 51% in Niassa in 2017, exceeding the target of 34%. The ICR also concluded that, since the collection of related data was challenging at the provincial level in the absence of a household-based survey, a more pragmatic PDO-level indicator should have been proposed to assess progress.(ICR, p. 18). Rating Modest PHREVDELTBL Page 11 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) PHOVRLEFFRATTBL Rationale • Under the original objectives, the project barely achieved its objectives, and therefore efficacy is rated Negligible. • Under the revised objectives, the ICR was able to demonstrate Substantial achievement for maternal and child health services. M&E quality issues were largely related to the nutrition objective, which was only partly achieved. Therefore, under the revised objectives, the aggregation of two almost fully achieved objectives with one partly achieved objective indicates an efficacy rating of Substantial. Overall Efficacy Rating Substantial PHREVISEDTBL 5. Efficiency The PAD (p. 22) referred to global experience suggesting that maternal and child health interventions are socially beneficial and that malaria control has significant externalities. The PAD also referred to a 2006 World Bank study, Mozambique: Better Health Spending to Reach the Millennium Development Goals, which presented modeling options for intervention packages. The study estimated the impact and additional cost per capita for four non-mutually exclusive service delivery options for reducing children under-five mortality and maternal mortality. The options were as follows: increased outreach services with mortality reductions of 7% and 2% respectively and with US$0.64 in additional cost per capita; increased community-based care with mortality reductions of 39% and 2% respectively and with US$0.74 in additional cost per capita; increased facility-based care with mortality reductions of 18% and 26% respectively and with US$2.83 in additional cost per capita; and increased curative outreach services with mortality reductions of 9% and 2% respectively and with US$1.00 in additional cost per capita. The PAD (p. 79) concluded that the proposed interventions were cost-effective. The ICR stated that framing such an epidemiological evaluation as an economic analysis was not the appropriate course of action, and that, given that there was no scope for comparability between the analysis conducted in the PAD and the ICR, the efficiency rating would be determined on the basis of operational efficiency (ICR, p. 21). The provinces targeted by the project had high poverty levels and unfavorable health indicators, reflecting an intention to use project resources efficiently. But there were significant shortcomings in the efficiency of implementation. The initial lack of ownership led to inadequate political support, which resulted in severe delays (ICR, p. 31). Effectiveness was held for 14 months (ICR, p. 21) due to delays in recruiting essential project staff and the development of the project implementation manual. Management capacities at MOH’s Page 12 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) Directorates were overstretched, and experience in managing large projects was lacking (ICR, p. 25). Staff turnover rate was high, including in procurement and M&E. In addition to civil works, the project experienced significant delays in community-based nutrition interventions. A project extension of one year, from December 2016 to December 2017, was necessary to enable completion of activities (ICR, p. 22). Weak capacities in M&E remained a major constraint during project implementation (ICR, pp. 30-31). Procurement shortcomings were noted throughout the entire project (ICR, p. 24). According to the ICR (p. 30), both procurement and contracting were "plagued" with obstacles rooted in legal constrains and regulations, all of which made the overall process very protracted. According to the ICR, the lack of accountability and the culture of long timelines put the project close to derailment at various points. The ICR (p. 30) concluded that procurement was a main limiting factor, which led to important delays, especially in the construction of health facilities. Procurement issues included delays in preparing bid evaluation reports due to the unavailability of evaluation committees and uncoordinated procurement responsibilities between the procurement management unit, the project manager and MOH technical staff. Financial management weaknesses also hindered project implementation (ICR, p. 24). Shortcomings included unavailability of funds and lack of earmarking at the provincial level; inadequate accounting for expenditures; delays in submitting interim financial reports and audit reports; delays in completing the treasury processes used for accounting and disbursement; and staffing shortages at central and provincial levels, including a lack of related expertise and slow administrative actions at the central level to manage project accounts. The TTL clarified (9/20/18) that there were no qualified audits with material deviations, but there were observations for rectifications and improvements. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is assessed across the entire project and is rated Substantial, as the objectives remained consistent with Bank and country strategies. Efficiency is also assessed across the entire project and Page 13 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) is rated Modest because of insufficient economic analysis and significant shortcomings in the efficiency of implementation. Efficacy is rated Negligible under the original objectives and Substantial under the revised objectives. According to IEG/OPCS guidelines, when a project’s objectives or its key associated outcome targets are revised, the final outcome is determined by the weight of Bank disbursements under each set of objectives (in this case, 16% under the original objectives and 84% under the revised objectives). • Under the original objectives, the outcome is rated Unsatisfactory (2) with a weight value of 0.32 (2 x 16%). • Under the revised objectives, the outcome is rated Moderately Satisfactory (4) with a weight value of 3.36 (4 x 84%). • These add up to a value of 3.68 (rounded to 4), which corresponds to a Moderately Satisfactory Outcome rating, indicative of moderate shortcomings in the project’s preparation, implementation and achievement. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Several factors raise the risk that expected development outcomes may not be maintained. In the context of current macroeconomic instability and growing debt service, cuts in domestic spending would be expected with shrinking scope to finance health services (ICR, p. 29). In the absence of effective donor coordination, fragmented donor financing could also be challenging. However, the newly approved Program for Results in the health sector may incentivize the government to increase the health budget and donors to continue their support to the sector. While weak technical capacities remain a risk, substantial capacity building was generated by the project (ICR, 24). Health care personnel, district staff, and community health workers benefited from training in the areas of maternal and child health, nutrition and malaria control. The three Northern Provinces benefited from the rehabilitation and construction of health facilities. The project supported several studies and operational research to inform policy and strategy, including support to the development of the Health Sector Strategic Plan and the strengthening of supply chain management. 8. Assessment of Bank Performance a. Quality-at-Entry Page 14 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) According to the PAD (p. 12), implementation arrangements were under the overall MOH responsibility with assistance from the National Directorate of Public Health for technical aspects, the Directorate of Administration and Management for financial management, the Directorate of Planning and Cooperation for monitoring and evaluation, and the National Malaria Control Program for malaria control activities. The PAD (p. 11) stated that the following lessons were considered in project design: building on solid economic and sector work, focus on M&E, consideration of government commitment and ownership, and simplicity of design. However, the lesson concerning adequate M&E was not adequately addressed. Identified risks included political and ownership changes, delays in procurement, recruitment and training, procurement delays, and fiduciary compliance, and mitigation measures were insufficient. At entry, the Bank team reviewed financial management arrangements, where risk was deemed to be substantial. Hence, the Bank required the updating of the Manual of Financial Management Procedures; agreement on the format of financial reports; and agreement on terms of reference for audits. These requirements were addressed, approved by the Bank, and adopted in the Operational Manual. An environmental assessment for indoor spraying operations, including a programmatic environmental assessment for malaria integrated vector management, was prepared and posted on MOH website (ICR, p. 27). For healthcare waste management, according to the PAD (p. 27), the project intended to use the plan developed under the HIV/AIDS Response project. A social assessment was undertaken to better understand local socio-cultural aspects that influence the demand side for health services (PAD, p. 24), particularly in the most disadvantaged communities. However, there were significant shortcomings. At entry, activities were not clarified for two objectives: the reduction of the prevalence of TB and the reduction of inequity in access to health services in Mozambique. There were inconsistencies in the scope of the project between the Financial Agreement and the planned interventions. M&E was not appropriately planned and prepared, neither in setting development objectives, nor in the adequacy of the results framework. Concerning capacities, the Bank’s initial assessment of the weaknesses in M&E capacities at the central MOH level and provincial levels and those of third-party organizations was inadequate (ICR, 29). Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision Implementation support missions were conducted regularly and often included fiduciary, environmental and social safeguards specialists based in the Maputo Country Office. There was continuity, as the TTL changed only once. During implementation, the Bank team reportedly was flexible and proactive in focusing on outcomes as demonstrated by the project restructurings and AF to support nutrition activities. Implementation Status and Results reports (ISRs) amounting to 20 reports were reportedly adequately prepared and used to monitor progress, discuss current bottlenecks, and raise questions to management. M&E challenges and data quality were identified through the ISRs and during the results Page 15 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) framework updates. Mission aides-memoire and accompanying management letters were reportedly used by the Bank and government as key documents to monitor progress and agreed actions to be taken following each mission. But the restructuring did not adequately address M&E arrangements for nutrition even though capacity issues had already been recognized, and it missed planning for a household survey, which was needed to assess progress toward the nutrition objective (ICR, p. 26). Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The original objectives were clear, and measurable in principle, but the results framework did not adequately reflect these objectives. The five objectives could not be adequately measured with only two PDO-level indicators on malaria and maternal health. Two of the original objectives did not establish a results chain. Also, the objectives, as stated, were not realistic (ICR, p. 24), including because they were unlikely to be achieved during the project lifespan (ICR, p. 16). M&E capacities were weak and not well assessed. Attribution was also mixed because of the multiplicity of similar interventions supported by other actors. The AF revised the objectives and re-designed the results framework. It addressed many of the above concerns, and strengthened the theory of change. However, some design gaps remained after AF revisions, notably for nutrition. Data related to the PDO-level indicator on nutrition were difficult to collect without conducting a household-based survey (ICR, p. 26). b. M&E Implementation M&E capacity constraints both at the ministerial (central and provincial) level, and at the level of third-party non-governmental organizations (NGOs) implementing community nutrition activities, hindered efforts to routinely collect data (ICR, p. 26) and remained a major constraint during project implementation (ICR, p. 30). Shortcomings included significant delays and quality issues in the nutrition data and the inability of NGOs to collect data regularly using a consistent methodology that would ensure comparability. Furthermore, the ICR (p. 26) stated that, at the central level, there was a lack of integration of different collection Page 16 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) methodologies and indicator definitions. According to the TTL (9/20/18), M&E shortcomings were also related to existing country demographic data and projections that were used in determining the denominators for project indicators. c. M&E Utilization According to the ICR (p. 27), M&E data were used during the entire implementation period to inform project management and decision making. Project objectives and targets were adjusted at the time of AF and when the project closing was extended in 2016. The ICR stated that M&E findings informed policy-making which contributed to the design of the follow-on project: Primary Health Care Strengthening - Program-for-Results (P163541), which was approved on 12/20/17. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as Category B since it triggered safeguard policies on Environmental Assessment (OP/BP 4.01) and Pest Management (OP 4.09), as there were potential risks in handling toxic chemical pesticides for vector control and from contaminated health care waste. An international firm was selected to develop an environmental assessment for all indoor residual spraying operations, including a programmatic environmental assessment for malaria integrated vector management, residual spraying, and the safe use of DDT and bendiocarb and lambda-cyhalothrin (ICR, p. 27). The draft was reviewed by the Bank and was posted on MOH’s website. A national workshop was organized in 2007 to provide an opportunity for stakeholders to comment on the draft document. Residual spraying supervisors, team leaders, and spray operators were trained according to World Health Organization guidelines and standards of the Ministry of Agriculture. Insecticide poisoning management training was provided to health workers. Antidotes and treatment medications were provided to trained health workers. Insecticide storage facility storekeepers were also trained in proper store management. As for health care waste management, and as stated in Section 8a, the project adopted the plan developed under an HIV/AIDS Response project, but no implementation information was provided. The ICR did not offer an explicit statement about compliance with environmental safeguards. A social assessment was undertaken to better understand local socio-cultural aspects that influence the demand side for health services (PAD, p. 24), particularly in the most disadvantaged communities. A group of women was selected to be trained as multi-purpose agents (Agentes Polivalentes Elementares) to provide traditional birth attendants with clean birth kits, and to promote institutional deliveries and community involvement. The training program for health staff and the operations manual included gender and reproductive health aspects. Page 17 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) b. Fiduciary Compliance Financial Management. As noted in Section 5, the ICR (p. 24) stated that financial management weaknesses and poor leadership hindered project implementation. Shortcomings included unavailability of funds and lack of earmarking at the provincial level; failure to properly account for expenditures; delays in submitting interim financial reports and audit reports; delays in completing treasury processes used for accounting and disbursement; staffing issues at central and provincial levels; and a lack of financial management and procurement specialists who were familiar with Bank guidelines. The ICR did not offer information about audits and qualifications. The TTL (9/20/18) stated that there were no qualified audits with material deviations, but there were observations and recommendations for improvement. Procurement. Procurement activities were carried out in accordance with Bank guidelines (ICR, p. 28), but there were extended delays and shortcomings throughout project implementation (see Section 5). At the mid-term review, measures were taken to strengthen MOH procurement, including the recruitment of a senior procurement specialist. c. Unintended impacts (Positive or Negative) According to the TTL (2/20/19), the project was instrumental in reviving the community health workers scheme, which facilitated the provision of health services at the peripheral level. d. Other -- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory --- Satisfactory Moderately Bank Performance Moderately Satisfactory --- Satisfactory Note: M&E issues that Quality of M&E Modest Modest persisted were largely related Page 18 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) to the nutrition objective, which was only partly achieved. Quality of ICR Substantial --- 12. Lessons The ICR (pp. 30-31) provided several lessons and recommendations restated by IEG: The lack of ownership and political support delay project implementation. The project was perceived as an exogenous investment within the health sector. The follow-on project opted for performance-based financing to support the government’s own programs under its established institutional arrangements and procedures. Adequate capacity for monitoring and evaluation can potentially facilitate the provision of reliable evidence on project performance. Under the project, M&E deficiencies were largely due to capacity constraints at all levels, and notably at the community level, throughout the implementation period. A multi-sectoral approach to nutrition interventions with sufficiently long implementation periods can potentially enhance the impact of interventions. The project indicated that the inclusion of educational activities at schools was insufficient, and that behavioral changes at the community level required more exposure time for effective community penetration. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR analyzed the project's experience in a candid and clear manner. It was aligned to development objectives. The ICR established the theories of change underlying the revised project objectives and explained related issues under the original objectives. There was no evidence provided under the original objectives as noted by the ICR. Under the revised objectives, the quality of evidence was adequate overall for maternal and child health services, and inadequate for nutrition. The narrative and evidence supported the ICR's conclusions on efficacy, except for nutrition. Evidence gaps were related to project shortcomings and not to the ICR’s sound analysis. Lessons and recommendations were related to project experience. Actual project costs by component were not provided. The ICR was consistent with guidelines, Page 19 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MZ-Health Service Delivery SIL (FY09) (P099930) except for insufficient reporting on financial compliance and audits, and the lack of an explicit statement about compliance with environmental safeguards. a. Quality of ICR Rating Substantial Page 20 of 20