Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) Report Number : ICRR0021133 1. Project Data Project ID Project Name P125018 3A-West Afr. Disease Surveillance (FY12) Country Practice Area(Lead) Western Africa Health, Nutrition & Population L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-14785 30-Jun-2017 10,750,000.00 Bank Approval Date Closing Date (Actual) 22-Oct-2013 30-Jun-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 10,000,000.00 10,000,000.00 Revised Commitment 10,000,000.00 10,000,000.00 Actual 10,000,000.00 10,000,000.00 Prepared by Reviewed by ICR Review Coordinator Group Anahit Aghumian Judyth L. Twigg Joy Behrens IEGHC (Unit 2) 2. Project Objectives and Components a. Objectives The project development objective (PDO) was to strengthen the regional disease surveillance and response system of Economic Community of West African States (ECOWAS) member countries. The PDO was defined the same way in the Grant Agreement between the Africa Catalytic Growth Fund and ECOWAS. Both the PAD and ICR refer to this project as “catalytic” in that it aimed “to fill a significant immediate funding gap in the areas of intervention and support catalytic growth for disease control in the sub-region and, in this way, to facilitate the development of additional larger operations by showing the commitment of a key development partner in this area, laying the foundation for the surveillance network, reinforcing partnerships Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) and building key capacities (PAD p.5) The PAD did not unpack the PDO into separate expected outcomes. The ICR assessed the achievement of the project's development objective under two sub-objectives: (i) strengthened regional disease surveillance and response systems; and (ii) strengthened disease surveillance and response systems of ECOWAS member countries. It is unclear why the ICR defined these two sub-objectives for assessment. The key activities under the first sub-objective were aimed to strengthen disease surveillance and coordination capacity of the regional health organization, West Africa Health Organization (WAHO). The key activities of the second sub-objective were aimed to strengthen country level capacity for disease surveillance and reporting. Neither of these sub-objectives had activities directly aimed to strengthen disease response. This ICR Review assesses the achievement of the PDO as stated in the grant agreement. b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components Component 1 – Regional capacity strengthening (appraisal cost: US$4.81 million, actual cost US$4.48 million) by promoting the use of World Health Organization (WHO)-sanctioned protocols and tools for integrated disease surveillance and response (IDSR), by fostering the development of harmonized policies, data collection and reporting tools and skills for disease surveillance, and by improving the sharing of data, best practices, and technical expertise in ECOWAS member countries. Activities to be supported were: a) establishment of functional centers for epidemiological surveillance (CES) in targeted districts in all ECOWAS member countries; b) assessment of the core capacities to implement the international health regulations in two remaining ECOWAS countries where such an assessment had not been carried out at the time of appraisal (Guinea Bissau and Senegal); c) facilitation of the revision of national disease surveillance and response strategies and plans in line with the 2010 WHO Technical Guidance for IDSR in all ECOWAS countries; d) organization of regional simulation exercises on outbreak preparedness and response; e) documentation and sharing of lessons learned from inter-country/cross-border collaboration initiatives; f) regular publication and dissemination of an epidemiological bulletin for ECOWAS; g) harmonization of tools for data collection and reporting; h) creation of a regional platform for electronic information sharing; i) harmonization of training modules on disease surveillance and laboratory techniques for CES personnel surveillance at district level; j) facilitation of exchanges among epidemiologists and other disease surveillance personnel in the ECOWAS sub-region; and k) enhancing the institutional capacity of the West Africa Health Organization (WAHO) for regional coordination and supervision. Component 2: Strengthening human resources (appraisal: US$4.93 million, actual: US$4.71million) was to support: a) long-term specialized training at the masters level (two-year) in disease surveillance and Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) response building on the experience of the Field Epidemiology Training Program (FETP) and the Field Epidemiology and Laboratory Training Program (FELTP); b) short-term and on-the-job training of frontline health workers at district level on disease surveillance using harmonized curricula based the Projet d’Appui en Surveillance Epidémiologique Intégrée approach to the development of CES; and c) short-term and on- the-job training of frontline health workers at district level on laboratory techniques using harmonized curricula based the Veterinary Laboratory Network for Avian Influenza and Other Transboundary Animal Diseases in West Africa initiative of laboratory training of personnel at district level and the strengthening of national and regional laboratories. Component 3: Management support (appraisal: US$1.01 million; actual US$2.31 million) was to strengthen the capacity of WAHO in project management through the following activities: a) recruitment of a project manager, financial specialist, procurement specialist, and monitoring and evaluation (M&E) specialist for a limited period of time until these functions could be taken over by WAHO regular staff; b) provision of technical and financial support to enhance M&E strategies and implement project monitoring, including field supervision, and evaluation activities; and c) participation of key staff in regional simulation exercises and other regional information sharing and learning opportunities. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project financing, costs, and borrower contribution The project was funded by the World Bank through the Africa Catalytic Growth Fund regional integration window with a grant in the amount of US$10 million over a four-year period. The direct recipient of the project funds was ECOWAS, and WAHO, which is ECOWAS's specialized agency, was the main implementing agency. In turn, WAHO entered into sub-contractual agreements with six implementing partners to carry out activities in 15 ECOWAS countries. The implementing partners were: 1) Center for International Cooperation in Health and Development, 2) Fondation Mérieux, 3) University of Oslo, 4) University of Ghana, 5) Centers for Disease Control and Prevention (CDC)/ University of Ouagadougou, 6) World Health Organization. The grant was fully disbursed. WAHO, the implementing agency, contributed US$ 1.5million to the project, double the initially planned $750,000. There was a discrepancy in the costs per components between the ICR and the PAD as shown in the following table. This discrepancy was not explained in the ICR. According to the project team, the ICR appraisal estimate reflected the additional US$1.5 million contributed by the WAHO. Components PAD appraisal ICR appraisal Actual cost (ICR) (US$ estimate (US$ estimate (US$ millions) millions) millions) 1) Regional Capacity Building 4.81 4.96 4.48 2) Strengthening Human 4.93 (0.75 WAHO 4.76 4.71 Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) Resources contribution) 3) Management Support 1.01 1.78 2.31 Total cost 10.75 11.5 11.5 Dates Project preparation started in 2011 and took about two years. This was due to the uncertainty in the extension and replenishment of the Africa Catalytic Growth Fund. The project was appraised on May 3, 2013, approved on October 22, 2013, and became effective on December 11, 2013. The first disbursement of the project, however, was carried out on May 22, 2014. In the meantime, the WAHO made the first installment to the project from its funds in the amount of $300,000 on March 7, 2014 in order to start implementation. Despite delays in procurement, the activities of the project were implemented as scheduled and closed as planned on June 30, 2017. A mid-term review of the project was carried out on April 25, 2016, instead of the initially scheduled December 2015, that is, slightly more than one year before project closing. According to the ICR, the mid- term review was delayed due to political issues affecting Burkina Faso, which hosted WAHO. 3. Relevance of Objectives & Design a. Relevance of Objectives The objective of this regional project was highly relevant at the time of its appraisal. The project would benefit 15 countries of ECOWAS to steer collective action for a critical global public good in the sub-region of West Africa. The objective was fully aligned with the WB's Regional Integration Assistance Strategy for Sub- Saharan Africa (RIAS) Strategic Pillar III – Coordinated Interventions to Provide Regional Public Goods as well as the Bank's 2009 Regional Framework for Communicable Diseases Control and Preparedness (RFCDCP), which aimed to: (a) develop regional integrated multi-disease surveillance and response capacity; (b) strengthen regional capacity for laboratory diagnosis of infectious diseases; and (c) strengthen regional institutions and networks for inter-country collaboration. With its focus on disease surveillance, the project also was aligned with the WHO’s Regional Office for Africa (WHO-AFRO) Regional Strategy on integrated Disease Surveillance and Response (IDSR) which was adopted by all 15 ECOWAS countries. The project targeted a clear need to build core technical capacities to meet legally binding requirements of the 2005 International Health regulations (IHR) through implementation of the IDSR system strategy. (PAD, p.6) The relevance of the project objective remained High at the time of project closing as the Ebola epidemic in West Africa confirmed the need to strengthen regional disease surveillance and response systems and inter- country collaboration among ECOWAS countries to detect disease outbreaks earlier and provide rapid and Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) effective response. Rating High b. Relevance of Design A regional approach to controlling communicable diseases in the sub-region of West Africa, adopted by the project, was an effective strategy to address global/regional public bads. The underlying theory of change of the project was that by providing support to improve the capacity of regional institutions (WAHO), to improve disease surveillance data sharing and to strengthen human resources in epidemiology and laboratory across ECOWAS member states, the project would strengthen the regional disease surveillance and response system in the sub-region. The project design was also in line with the rationale of IDSR strategy, which promotes rational use of resources by integrating and streamlining common surveillance activities (IDSR 2010, p. 6). However, the PDO of the project was very broad, and neither its key activities nor the available funding were adequate for achieving the PDO. The key activities that the project intended to carry out would have contributed to strengthening of regional and country capacity for disease surveillance and reporting, but would have not contributed to strengthening regional and country capacity for response to disease outbreaks. In that sense, there is a disconnect between the project's development objective and its key activities. At design, the results framework (PAD, p. 19) did not make explicit how the links were envisioned between specific expected outcomes, intermediate outcomes, outputs, and activities. By not mapping the proposed activities or expected outputs to the intermediate and final outcomes, the PAD left unclear how and where the proposed activities would contribute to specific expected outcomes (e.g. which activities and how they would have contributed to the strengthening of the disease response system). Rating Modest 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To strengthen the regional disease surveillance and response system of ECOWAS member countries. Rationale The PAD did not unpack the PDO into separate expected outcomes. This validation infers two outcomes to assess the achievements of the project, while recognizing that some of the key outputs of the project Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) contributed to these two outcomes equally. Outcome 1) Strengthened regional disease surveillance and reporting capacity in the ECOWAS sub-region; Outcome 2) Strengthened national level disease surveillance and reporting capacity of ECOWAS member countries. Outcome 1: Strengthened regional disease surveillance and reporting capacity in the ECOWAS sub- region Outputs • Regional M&E Platform (DHIS2) for electronic information sharing is established. The ICR (p. 12) indicated that the harmonization of tools for data collection and reporting, another activity the project aimed to support, are in progress and this is activity is supported with additional funding by the United States Agency for International Development (USAID). • The WAHO Project Management Unit (PMU) was staffed as planned with 11 functioning members, including financial management and procurement specialists. Intermediate Outcomes Regional simulation exercises for outbreak response were not carried out by WAHO as planned. The original target was 7 exercises. Outcomes Outcome indicator (OI) 5: 80 weekly and 3 quarterly regional disease surveillance bulletins were published and disseminated by WAHO to ECOWAS member states, based on data received through the regional platform. According to the ICR, this target was achieved substantially. Outcome 2: Strengthened disease surveillance and response system of ECOWAS member countries Outputs • Assessments of the core capacities to implement the international health regulations (IHR) were carried out in Cape Verde, Mali and Senegal. • Support was provided to revising national disease surveillance and response strategies and plans in line with the 2010 WHO’s Technical Guidance for IDSR in 6 countries as opposed to all 15 ECOWAS countries; • Drafting and implementation of five-year IDSRS strategic plans were done in 5 countries, Ghana, Liberia, Mali, Niger, Nigeria and Senegal. In one country, Burkina Faso the project supported the revision and printing of 1,000 copies of the IDSRS. Intermediate outcomes • 60 Centers for Epidemiologic Surveillance(CES) established in targeted districts in 10 ECOWAS Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) member countries, meeting the original target. • 48 health workers completed long- term (masters level) FELTP training (from 0 at baseline), not reaching the target of 54. According to the ICR, this target was achieved substantially. The distribution of the total graduated candidates by gender and country of sponsorship is provided in the ICR. • 420 health workers (of which 99 women) received short-term and on the job training of frontline health workers at district level on disease surveillance and on laboratory techniques using specified harmonized curricula. This surpassed the original target of 350 (baseline 0). • In total, 708 health personnel directly benefited from the trainings, of which 168 were female, surpassing the 410 target. There was no baseline and target set for female participation for this indicator. Outcomes OI1. 95.2 percent of reports for selected priority diseases submitted by the CES were complete and on-time and used standard formatting. The reporting on this outcome was different from the initially selected indicator, so the original target is not applicable. OI2. 543 health personnel received training, of whom 23 percent were female, surpassing the target of 404 persons, but slightly lagging on the target for female participation, originally set 25 percent of the total trainees. OI 3. 82 percent of short term trainees in field epidemiology or laboratory skills were working at district 12 months after completing training. This surpassed the original target of 80 percent. OI 4. 96 percent of FELTP trainees graduated their training, surpassing the original target of 85 percent. Overall, the ICR reported that 4 of 5 outcome indicators were surpassed and 1 was achieved substantially. However, there were discrepancies in the way outcome data were collected and reported in 4 of these 5 outcomes, leaving gaps in the evidence on the achievement of some project outcomes: • The original outcome indicator 1 in the PAD was "percentage targeted districts which submit complete and on-time reports for selected priority diseases using the standard reporting format". However, the data reported were on "the percentage of reports received on time..." which did not tell how many of those 60 CES were able to submit "complete and on time reports using the standard format." According to the project team, the reason for this difference in computation methodology was the significant delay in schedule of CES trainings (due to procurement delay and customs issue), which were necessary for the CESs to comply with the new reporting requirements. • For the outcome indicator 3 "percentage of short-term trainees in field epidemiology or laboratory skills who are working at district level 12 months after completing training, by gender and country of sponsorship," the reporting on the achievement was based on the tracking of only 17 trained health workers out of a total of 420, because the rest of health workers who benefited from the short-term training had not worked for a full 12 months at district level at the time of project closure. Thus, the data Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) collected for the ICR in December 2017 represents about 4 percent of those who benefited from the short- term training, which suggests that caution is warranted in drawing a conclusion about whether the 80 percent target was surpassed. • The ICR also does not provide an explanation why such a limited number of short-term trainees fully completed the 12-month requirement by the project closure in December 2017. The PAD envisaged the following targets for the short-term trainings (PAD p. 19): year 1—0 trainees; year 2—50 trainees; year 3— 200 trainees; year 4-100 trainees. By December 2017, the majority of the graduates from the year-2 and year-3 cohorts were supposed to be working more than 12 months. According to the project team, however, the short-term trainings were significantly delayed due to the initial delays in procurement of equipment for the CES, and due to customs issues the implementing agency faced in those 15 countries. • The target for the outcome indicator 4 "completion rate of FELTP trainees by gender and country of sponsorship" was lowered from the original 60 to 50 and has been surpassed. According to the ICR, at the beginning of the training program the CDC, one of the implementing agencies, agreed to train 30 people, as opposed to the originally set target of 40 (ICR data sheet), resulting in lowering the target. The new target, however, was not formally revised in the results framework. • The target for the outcome indicator 5 "semi-annual regional disease surveillance bulletins published by WAHO and disseminated to ECOWAS member states" had become less relevant, according to the ICR, because of the WAHO's request to publish weekly bulletins by ECOWAS countries as a more practical tool. Based on these weekly bulletins, the ICR calculated the possible number of semi-annual bulletins, and considered the target as substantially achieved. In other words, the targets for outcome indicators OI 4 and OI5 required revisions, which have not been done formally. Thus, overall, the project provided technical assistance and implemented large number of trainings to strengthen human resources and institutional capacity of ECOWAS countries for disease surveillance and reporting. It contributed to strengthening of WAHO's institutional capacity for project management and coordination. However, all activities were aimed at strengthening surveillance and reporting capacity at regional and country level and, none of the project activities were directly aimed at strengthening the "response system" as stated in the PDO. In addition, the gaps in evidence mentioned above on achieving two outcomes— capacity building of 60 CESs, measured by the “percentage targeted districts which submit complete and on-time reports for selected priority diseases using the standard reporting format” (OI 1), as well as the job retention rate of those who received short-term trainings in the field of epidemiology or laboratory skills (OI 3)— weaken the Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) claims of achievement of those outcomes. Rating Substantial PHREVDELTBL PHREVISEDTBL 5. Efficiency The PAD provided a strong rationale for the benefits of increasing regional and country level institutional capacity for disease surveillance and response as a global public good, but no detailed economic and financial analysis was provided. The ICR used a Markov Chain Monte Carlo simulation model to estimate the health and economic impacts of reducing the likelihood of disease outbreaks in West Africa due to the implementation of the project. The model used parameters such as an annual probability of outbreaks (0.03), annual real growth rates in West Africa, loss of output associated with an outbreak (about 4.8 percent of regional gross domestic product), and an estimated contribution of the project to overall effectiveness of surveillance and response systems of five percent. The net present value of expected health benefits of controlling an outbreak in West Africa was calculated to be US$ 283.6 million, covering estimated reduced losses from mortality, productivity losses due to morbidity and absenteeism, and losses resulting from behavior changes to avoid infection. The ICR calculated benefit-cost ratios under different scenarios, based on a combination of different probabilities of pandemic and severity of outbreak. According to the model used, the net present value of expected annual health benefits of controlling an outbreak in West Africa was estimated at US$ 10.8 million, thus making the benefit equal to US$ 6.6 for each US$ 1 invested through the project (in the case of severe outbreak with 3% probability). As in the PAD, the benefits covered reduced losses from mortality, productivity losses due to morbidity and absenteeism, and losses resulting from behavior changes to avoid infection. The benefits would be reduced to US$ 0.3 for each US$ 1 invested in the case of a mild outbreak with 1% probability. It should be noted that this Monte Carlo simulation model has been replicated in the PADs of the three Regional Disease Surveillance Systems Enhancement Program in West Africa projects (REDISEE) that followed this one. Overall, as a catalytic project, a reasonable case can be made that the project's funds were used efficiently. The strengthening of WAHO, for example, has allowed use of the implementing agency by five Bank- financed larger regional projects in the ECOWAS countries. Operational efficiency: There were initial delays in the implementation of the project due to the challenges in the process of obtaining memoranda of understanding (MOUs) between ECOWAS member countries (Ministries of Finance) and WAHO. Lack of MOUs with country governments impeded the duty-free importing of goods and equipment procured under the project. The ICR (p. 9) noted that issues surrounding duty-free privileges persisted even after MOU signatures were received. Despite these implementation challenges, the Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) project was completed on time and disbursed fully. The project team, however, noted the negative impact of the significant delay (1.5 years after the project became effective) due to customs issues on the ability of the project to equip the CES and to carry out the short-term trainings on time. This eventually resulted in the shortage of data to report on two (OI 2 and OI 4) of the project's outcome achievements in the ICR. The Implementation Status Reports also noted that implementation was further delayed by the participation of the WAHO team in the Ebola response, which took most of WAHO management and technical staff away from the project for a significant period. The project's management cost exceeded the estimated budget by 130 percent, from the initial estimation of US$ 1.01million to US$ 2.31 million. No explanation was provided in the ICR on the reasons for this discrepancy. The project team confirmed that additional management costs were covered by the implementing agency, WAHO, to retain key staff to be able to implement the project as well as the other new regional projects they committed to implement. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project development objective to strengthen regional capacity for disease surveillance and response for the West Africa region was and remains highly relevant, given the high frequency of infectious disease outbreaks in the region and the Ebola pandemics in 2014-2015. The relevance of the project's design is rated Modest because the project's scope and budget were not adequate for achieving such a broad development objective. Achievement of the objective to strengthen the regional disease surveillance and response system of ECOWAS member countries is rated Substantial. The project provided technical assistance and implemented a large number of trainings to strengthen human resources and institutional capacity of ECOWAS countries for disease surveillance and reporting. The project also contributed to enhancing the capacity of WAHO as an implementing agency of regional projects. The gaps in evidence on achieving two outcomes—the capacity building of 60 CESs and the job retention rate of health personnel receiving short-term trainings — were due to delays in procurement and customs issues that were particularly challenging in the context of the ECOWAS Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) region. In addition, all the project activities were oriented toward strengthening surveillance and reporting systems, and none of the project activities were directly aimed at strengthening the "response system," as stated in the PDO. Project efficiency is rated Substantial. Despite some shortcomings in operational efficiency, the project led to larger follow-on operations and mobilization of resources. These ratings are indicative of moderate shortcomings in the project's preparation and implementation, leading to an Outcome rating of Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating Several risks to development outcome can be distinguished. These include a deterioration of the benefits that the project achieved: a weakening of the institutional capacity of WAHO and of district-level CES due to inadequate resources; the failure to retain those trained in the field of epidemiology and laboratory skills at district level after they have completed their training, and the likelihood that the focus of country governments in the sub-region will shift to other competing and immediate health priorities. At this stage, some of these risks, such as the weakening of WAHO's capacity, appear very low. The project significantly contributed to WAHO's institutional capacity strengthening, as attested by the large number of new regional projects that are now managed by WAHO. WAHO has become the implementing agency for five Bank-financed regional projects and 12 projects financed by development partners, and this experience is expected to strengthen its institutional capacity further. The Bank-financed Regional Disease Surveillance Systems Enhancement (REDISSE) program (a series of three projects) is designed to create incentives to retain trained health personnel, and will help participating country governments to maintain their focus on strengthening of disease surveillance and response. The risk of deterioration of district-level CES, a concern also raised in the ICR of the grantee, remains. The ICR stated (p. 14) that REDISSE I, which became effective before this project was closed, will ensure the continuity of funding to existing CES and establish new CES. The REDISSE I project PAD, however, does not envisage such activity. Yet, the project team noted that after CES have been equipped and staff trained under the project, the costs of maintaining those CES are small, and the country governments should be able to maintain them without donor support. a. Risk to Development Outcome Rating Modest 8. Assessment of Bank Performance Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) a. Quality-at-Entry The regional approach and reliance on grant financing were appropriate for this project, which aimed to provide regional and global public goods. The project team consulted with other key bilateral and regional actors in disaster surveillance and reporting, including the WHO regional office, USAID, the Canadian International Development Agency, the United States Centers for Disease Control and Prevention, and WAHO, about the division of labor. Key risks were identified correctly. Some of those risks are common in the design and implementation of regional, multi-country projects: the lack of ownership of regional projects by countries and managing the expectations of the countries from this catalytic project, and weak institutional capacity of the main regional implementing agency. Some of these risks, such as the capacity of WAHO, were mitigated effectively. However, weak ownership of countries persisted. Implementation arrangements, overall, took into consideration the existing institutional capacity of the implementing agency. The incongruity between - on the one hand - the broad nature of the PDO (which also included disease response) and - on the other hand - the planned activities (which were focused only on disease surveillance capacity building) as well as available funding, was not resolved during project design. M&E arrangements, while designed appropriately to mitigate the weak capacity of the implementing partner agency, were not implemented as planned, thus contributing to the inability to mitigate the weaknesses identified in the indicators. Having an M&E specialist only for a short time period prevented appropriate guidance on the M&E aspects of the project (ICR, p. 27). Although the project was not designed to have gender-sensitive activities, three out of five outcome-level indicators were to be disaggregated by gender and country of origin. However, collecting data segregated by gender and country of origin was very useful for establishing baselines for subsequent Bank-financed projects in the region. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Despite some of the design shortcomings were identified early, the team did not restructure the project. The project's mid-term review, which was carried out in February 2016, recommended restructuring "to change the wording of the PDO in order to better reflect what exactly the project is and can contribute to (to increase the human and technical capacity for disease surveillance at country and regional level in ECOWAS region)" (ISR No. 7); some of the indicators also were to be revised. The formal restructuring of the project, however, was not done for various reasons. According to project documents, the Country Management Unit rejected the request for restructuring, citing the nearing date of project closing at the time the restructuring package was presented. Other supervision processes were adequate, and Implementation Status Reports were frank, pointing to the problems in design and bottlenecks in implementation, such as fiduciary capacity constraints and procurement delays. The mid-term review was carried out with a delay, when 59 percent of project funds were already disbursed. The mid-term review's conclusions were consistent with the findings of the Implementation Status Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) Reports. The failure to collect and report the required data some PDO indicators had a negative impact on the quality of supervision. In sum, while the identification of issues was done proactively through supervision missions and reports, the Bank did not react adequately to resolve those problems. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. Assessment of Borrower Performance a. Government Performance A perceived lack of project ownership by some of the ECOWAS countries partially can be considered as a shared responsibility of the Bank's task team and WAHO. Despite ensuring the endorsement of the project by the Ministers of Health of the ECOWAS countries to ensure ownership of regional capacity building for IDSR, there was low demand for support across countries during the annual work planning process. To address this issue, a communication strategy was developed in the second half of the project to increase awareness and visibility of the project among the stakeholders and development partners (ICR, p. 9). Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance The implementing agency, WAHO, did not have prior experience in managing Bank projects and had weak fiduciary capacity that led to initial delays in implementation. Weak institutional capacity was anticipated in the project's design, and the first activity of the project was the appropriate staffing of the project PMU with procurement, financial management, and M&E specialists (under the third component of the project aimed to strengthen WAHO's capacity in project management). One of the key improvements noted in the ICR was the establishment of streamlined coordination of project activities by different implementing partners. The ICR (p. 7) also noted the high level of dedication of WAHO to implement the project by, for example, transferring the first installment of counterpart funds to the designated project account before Bank funds were available, in order not to delay the implementation. Four of six implementing partners (listed in Section 2e) performed well. Some had frequent delays and poor reporting, primarily during the period of the Ebola crisis, when WHO Africa Region staff were redeployed. Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The project's objective was broad. Some of the outcome and intermediate outcome indicators either were not formulated properly (e.g. semi-annual regional disease surveillance bulletins published by WAHO) or were overly ambitious (e.g. regional simulation exercises for outbreak response by WAHO). At design, the results framework (PAD, p. 19) did not make explicit how the links were envisioned between specific expected outcomes, intermediate outcomes, outputs, and activities. By not mapping the proposed activities or expected outputs to the intermediate and final outcomes, the PAD left unclear how and where some of the proposed activities would contribute to specific expected outcomes. The design of M&E implementation detailed in the PAD was reasonable. The plan was to have two WAHO M&E specialists to lead M&E implementation, and to strengthen WAHO's M&E capacity by hiring an additional M&E specialist with experience in disease surveillance. The project was designed to have WAHO responsible for the collection of progress data from the districts receiving project support. Additional data on the training activities was to come from implementing partners, as well as from ministries of health that were to produce semi-annual epidemiological bulletins. Additional measures were envisaged to triangulate the received data though pre-developed checklists and surveys. This progress information was supposed to feed into an annual planning exercise carried out with the WAHO Program Committee. Overall, M&E design was anchored on WAHO's coordinating role and based on an assumption that participating countries would submit relevant data collected through standardized methods and tools that would facilitate consistent activity. The electronic platform at regional level would have helped to integrate such data. The plan to have strong M&E capacity in WAHO, however, did not materialize, and this gap had a negative impact on M&E implementation. b. M&E Implementation The ICR primarily discussed the project's support for the regional M&E platform (DHIS2) used to gather and monitor data at regional and country levels. This validation, however, assesses the project's M&E implementation more broadly as it was designed in the PAD, to collect data on project outcomes. The implementation of this M&E plan had shortcomings. According to the project team, the M&E specialist hired by the PMU was on staff for only two years, and the two WAHO M&E specialists were very busy with other projects. In other words, the M&E capacity of the WAHO PMU remained weak. Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) The absence of an M&E specialist(s) in the team throughout the lifecycle of the project had a negative impact on the collection of data, as already noted. As stated in the Efficacy section, some of the outcome indicators, such as OI4 and OI5 required revisions as the project progressed. There are also gaps in the evidence in the reporting of outcomes on CES(OI1) and on job retention of short-term trainees at district level 12 months after completing training (OI3). c. M&E Utilization The ICR did not provide evidence that the data collected on implementation progress were used to make course corrections in the project. The ICR recognized the value of DHIS2 as an important step in establishing a regional platform for data collection and information sharing, and of the submission of 80 weekly epidemiological bulletins over the lifetime of the project (ICR pp 11-12). According to the ICR, 14 out of 15 ECOWAS countries now regularly send data through the platform. The harmonization of automatic data collection and transmission tools to the WAHO is in progress with support from USAID. M&E Quality Rating Modest 11. Other Issues a. Safeguards No environmental safeguard was triggered under this project, and it was rated Environmental Assessment category C. However, the PAD indicated the strong correlation between climate change, which increases the risk of extreme weather events, food and water security, and exacerbation of susceptibility to communicable and non-communicable diseases. b. Fiduciary Compliance Procurement There were delays in procurement at the beginning of the project stemming from impediments in the duty-free importing of goods and equipment to the recipient countries due to differences in custom procedures among these countries. The ICR noted that issues surrounding duty-free privileges persisted even after MOUs between WAHO and ECOWAS countries were signed. Another constraint to procurement was the low disbursement ceiling for the project, set at $300,000 in the disbursement letter. This low disbursement ceiling was set as a risk mitigation measure, due to the inexperience of WAHO in managing Bank-financed projects (ICR, p. 12). Despite these delays with procurement in the first year, the project was implemented within set dates and disbursed fully. Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) Financial management WAHO, the implementing agency, carried out the financial management and reporting of the project as required. WAHO submitted interim financial reports periodically to the Bank. The first external audit was completed in June 2015. All financial audits were submitted to the Bank, published on WAHO's website, and certified by the Bank (ICR, p..13). c. Unintended impacts (Positive or Negative) The presence of gender-segregated indicators under the second component allowed the detection of important gender disparities in participation in the FETP/FELTP programs, reflecting the gender disparity in the workforce. Of the 48 selected participants in the long-term FELTP program, only seven were women. This finding was used in the design of the REDISSE 2 (2017) project, which envisaged carrying out assessments of the current workforce in terms of quantity, geographical and gender distribution, and capacity, and implementing activities to create incentives that not only draw those with relevant skills to the public sector, but also improve staff motivation and retention, taking into account gender differences within the health workforce. The project served as a catalyst to mobilize additional resources for strengthening disaster surveillance and response capacity in ECOWAS countries. According to the project team, the project attracted CAD 20 million from the Canadian International Development Agency. These funds are being used to improve gender- targeted activities under the REDISSE program. The project also contributed to strengthening the institutional capacity of WAHO as a PMU to manage larger region-wide operations. Following the assessment of the performance of WAHO as a PMU, the agency has received a new mandate from the ECOWAS (in 2016) to receive funding directly from the Bank (ICR, p. 25). d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Relevance of the project's design is Modest because the project's scope and budget Moderately were not adequate for Outcome Satisfactory Satisfactory achieving such broad PDO. Efficacy is rated Substantial. The project contributed to strengthening of human Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) resources and institutional capacity of ECOWAS countries for disease surveillance and reporting and enhancing the capacity of WAHO. There are, however, gaps in evidence on achieving some of the stated outcomes. More importantly, no project activities were directly aimed at strengthening the "disease response system", as stated in the PDO. Efficiency of the project is rated Substantial. Despite some shortcomings in operational efficiency, the project led to mobilization of resources and larger follow-on operations critical for further strengthening of the regional disease surveillance and response capacity in the ECOWAS sub-region. Risk to Development Modest Modest --- Outcome Shortcomings in project design, while identified at an early stage, were not Moderately Moderately Bank Performance addressed. The shortcoming Satisfactory Unsatisfactory in the implementation of M&E led to missed opportunities in data reporting. Moderately Moderately Borrower Performance --- Satisfactory Satisfactory Quality of ICR Modest --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons Lessons drawn from the ICR, adapted by IEG: When designing and implementing regional projects with wide geographic coverage, some steps can strengthen country ownership, which is important for the project's effectiveness and sustainability of results. Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) • Country ownership can be strengthened by giving the participating countries some decision-making power over how the funds will be spent in their respective countries through financial agreements; The experience of this project indicates that having MOUs between ECOWAS member countries and WAHO was clearly not sufficient to ensure country ownership, and the design should have allowed some flexibility for the participating countries to design activities and have access to funds. • Developing an effective communication strategy at the outset to improve the awareness of all the stakeholders in participating countries throughout the lifecycle of the project can significantly improve the country ownership and contribute to sustaining the results after the closure of the project. It can also be effective in managing the participating country expectations when the project has a catalytic nature and small budget. In this case, lack of an effective communication strategy on how best to promote the project across ECOWAS member countries and to advocate for disease surveillance as a priority agenda contributed to the perceived indifference displayed by some countries to the project. The project's visibility among the stakeholders and DPs was significantly increased after such communication strategy was put in place. • The effectiveness of a regional project can be improved by making sure that there are focal points assigned from each implementing partner agency. This can improve the PMU's coordination efforts when the implementation structure is complex and includes many sub-contracted partners working in different countries. In the project, the reporting difficulties experienced with some implementing partners were due to the lack of dedicated individuals assigned to the project. Lessons drawn by IEG: • In projects of high complexity and of a catalytic nature (i.e. intended to demonstrate the commitment of a key development partner in this area and to stimulate further resource mobilization), a strong results framework and rigorous M&E design and implementation can help distill credible data and evidence on what works and what doesn't to improve the design and implementation of scale-up operations. • Collecting gender-segregated data in projects aimed to strengthen human resources, even if the project is not designed to have gender-sensitive activities, can yield very useful baseline information. In this case, the collected data disaggregated by gender and country of origin has revealed significant gender disparities. This information has been effectively used by the subsequent larger projects (REDISSE program) to design gender-sensitive activities targeting gender disparities detected by this initial project. Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review 3A-West Afr. Disease Surveillance (FY12) (P125018) 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR presented evidence on the project's achievements well by putting together data reported by different implementing partners and WAHO, which in some cases were not consistent. In some cases, the ICR went out of its way to find data to assess achievements of outcomes when reporting from the formal results framework was inadequate or incomplete (e.g. data on short-term trainees retaining their jobs 12 months after the trainings were completed). The broad lessons provided in the ICR as well as the lessons derived under each of the components were valuable and already seem to be taken into consideration by the REDISSE program. The ICR was candid in pointing out to shortcomings in the collection and reporting of data on some of the key outcome indicators. There are some caveats to the reporting in the ICR. The results chain of the project, presented in the ICR's Figure 2 (p 19), while logical, included inputs that were not defined in the PAD and introduced new outcomes, (e.g. increase investment in HNP projects supporting regional collaboration, scientific information routinely shared in the region and globally, etc.) not supported by the selected outcome indicators of the project and not substantiated in the main text. There were also some inconsistencies in data reported by the implementing partners, lack of explanation of the differences between the estimated and actual costs. The ICR also could have candidly reported on some of the issues in M&E design, implementation and use, other than focusing on just one aspect (the regional IT platform). a. Quality of ICR Rating Modest Page 19 of 19