ICRR 13581 Report Number : ICRR13581 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 09/19/2011 PROJ ID : P094694 Appraisal Actual Project Name : Eritrea Hiv/aids/sti, US$M ): Project Costs (US$M): 26.5 27.3 Tb, Malaria And Reproductive Health Project (hamset Ii) Country : Eritrea Loan /Credit (US$M): Loan/ US$M ): 24.0 24.0 Sector Board : US$M): Cofinancing (US$M ): Sector (s): Health (50%) Central government administration (30%) Other social services (13%) Sub-national government administration (6%) General education sector (1%) Theme (s): Tuberculosis (25% - P) HIV/AIDS (25% - P) Population and reproductive health (24% - P) Malaria (13% - S) Participation and civic engagement (13% - S) L/C Number : CH175 Board Approval Date : 06/30/2005 Partners involved : Closing Date : 06/30/2010 06/30/2010 Evaluator : Panel Reviewer : Group Manager : Group : Judyth L. Twigg John R. Eriksson IEG ICR Review 1 IEGPS1 2. Project Objectives and Components: a. Objectives: According to the Development Grant Agreement (DGA, p. 19), the project’s objectives were “to assist the Recipient in: (a) containing the spread of HIV/AIDS/STI in vulnerable groups as well as the general population through a multi-sectoral approach which aims at scaling up prevention, diagnosis, care, and support services for HIV /AIDS/STI, with a renewed focus on the most vulnerable populations; (b) expanding DOTS coverage, improving case detection and treatment outcomes for tuberculosis; (c) further reducing or at least maintaining malaria mortality and morbidity at current low levels; (d) improving the coverage of reproductive health interventions; and (e) strengthening the overall health system’s capacity to address the HAMSET diseases .� The HAMSET diseases are defined (DGA, p. 2) as HIV/AIDS, malaria, sexually transmitted infections (STI), tuberculosis, and reproductive health . The Project Appraisal Document (PAD, p. 9) states the objectives slightly differently : “to: (i) contain the spread of HIV/AIDS/STI in vulnerable groups as well as the general population through a focused multi -sectoral approach, with renewed attention to the most vulnerable populations; (ii) expand the coverage of Directly Observed Treatment (DOT), improve case detection and treatment outcomes for malaria; (iii) reduce or at least maintain malaria mortality and morbidity at the current low levels; (iv) improve the coverage of effective Reproductive Health (RH) interventions; and (v) to strengthen the overall health system, including human resources for health, to enable the country to better address HAMSET diseases.� DOT is the standard therapy for tuberculosis (TB), where a trained health worker directly observes the patient taking each dose of anti -TB medication. This review uses the DGA statement as it is more evaluable . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): The project contained four components : Multi -sectoral response (appraisal, US$ 3 million; actual, US$ 2.34 million, or 78 1. Multi- 78%% of appraisal ). Support for participating agencies other than the Ministry of Health, including the ministries of education, defense, and labor and human welfare, and civil society organizations at both the national and zoba levels, to scale up prevention, care, and support interventions for HIV/AIDS/STIs, malaria, TB, and reproductive health . Promotion of healthy behaviors and lifestyles through various multi-level communication methods, including information dissemination, education and communication campaigns, and peer -led behavior change communication programs . US$ 14 million; actual, US$ 16 million, or 114% 2. Health sector response (appraisal : US$14 114 % of appraisal ). Subdivided into: HIV/AIDS/STI: appraisal, US$3 million; actual, US$1.95 million, or 65% of appraisal TB: appraisal, US$2 million; actual, US$1.41 million, or 70.5% of appraisal Malaria: appraisal, US$2 million; actual, US$1.26 million, or 63% of appraisal Reproductive health: appraisal, US$4 million; actual, US$5.58 million, or 140% of appraisal Human resources for HAMSET diseases : appraisal, US$3 million; actual, US$5.81 million, or 194% of appraisal Each of the four disease sub -components comprised three activity categories : (a) improving the information base for decision making, through supporting disease surveillance and reporting, carrying out of surveys and operational research in program management, epidemiology, drug resistance, and disease behavior; (b) scaling up and expanding preventive interventions; and (c) scaling up diagnostic, treatment, care, and support services, all through technical advisory services, in -service training, financing of operating costs, acquisition of drugs, medical supplies, and equipment, and minor renovation of reproductive health facilities . The human resources subcomponent supported the development and implementation of an overall human resources policy in the health sector and a strategic plan to strengthen human resources for HAMSET diseases, through : (a) consolidating existing health policies and developing a five -year national strategic plan; (b) developing and implementing annual work plans, providing technical advisory services, training, and workshops, and developing and disseminating training materials . 3. Community -managed HAMSET response program (appraisal : US$ US$4 84% 4 million; actual, US$ 3.35 million, or 84 % of appraisal ). Scaling up and consolidating community -managed response initiatives to address HAMSET diseases through: (i) financing of sub-grants for small-scale community subprojects aimed at controlling HAMSET diseases; and (ii) developing the capacity of communities to mobilize and identify HAMSET problems and to development, implement, monitor, and evaluate subprojects . 4. Project management and coordination, capacity building, M&E, and innovation and new policy development (appraisal, US$ 3 million; actual, US$ 3.71 million, or 124%124 % of appraisal ). Strengthening project management and evaluation capacity, through : (i) provision of support to project management units at the national and zoba levels, and coordination of project activities at the central level; (ii) training in planning, management, and implementation of project activities for key staff in non -health sectors and local governments; (iii) developing and carrying out a health sector monitoring and evaluation system; (iv) developing, piloting, and evaluating innovative approaches to service delivery; and (v) strengthening health policy related to HAMSET . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost : Two reallocations took place during implementation . In May 2008, at 61% disbursement, in response to the availability of substantial resources from the Global Fund to Fight AIDS, Tuberculosis, and Malaria, there was a reallocation across health sector response subcomponents toward priorities not supported by the Global Fund, in particular human resources and reproductive health . This reallocation followed recommendations made at the Mid-Term Review. In July 2009, at 94% disbursement, an additional reallocation was made in similar response to Global Fund priorities. These reallocations did not affect the development objectives . The ICR does not explain why more was spent than planned on the project management component . The project team clarified that additional spending went toward monitoring and evaluation activities, including funding a Lot Quality Assurance Survey (LQAS). Financing : During the lifetime of the project, there were exchange rate gains of approximately US$ 1.5 million, but these were used to compensate for price fluctuations . The ICR does not indicate for what activities, or for what component (s), this spending was directed. The project team clarified that the gains were used under Component II primarily to finance equipment used for rehabilitation of facilities . Borrower Contribution : The ICR provides contradictory information on the Borrower ’s contribution. Page 18 reports that the Borrower contributed US$2.8 million of a planned US$2.0 million contribution, while the Project Costs and Financing table on page 36 reports that the Borrower contributed US$ 4.8 million. The project team confirmed that the correct Borrower contribution was US$2.8 million. Dates : The project closed as scheduled on June 30, 2010. HAMSET II was a repeater project of the Bank -financed Eritrea HIV/AIDS, Malaria, STD, and TB Project (HAMSET), financed by a US$40 million credit from December 19, 2000 (approval) to March 31, 2006 (closing). 3. Relevance of Objectives & Design: Relevance of Objectives is rated High . The project’s objectives are relevant to Eritrea ’s health situation at the time of appraisal, to current Eritrean government strategy, and to current Bank strategy . At the time of appraisal, Eritrea’s adult HIV prevalence rate was relatively low at 2.4% (2003), but much higher in some geographic areas and sub -populations, pointing to the threat of a more generalized epidemic. The country’s malaria control program had achieved an 80% reduction in morbidity and mortality from 2000 to 2004, but continual control efforts were judged necessary because of the significant risk of renewed outbreaks. TB control was thought to suffer from inadequate implementation capacity, weak supervision, lack of quality assurance, and a limited information base for decision -making. Reproductive health indicators were among the worst in sub-Saharan Africa (SSA), with maternal mortality rates at 998/100,000, 30% of deliveries with a skilled attendant, 8% use of modern contraception, and 85% prevalence of female genital mutilation (FGM). Skilled human resources for health were in critical shortage, with health staff -to-population ratios among the lowest in SSA . As the Project Appraisal Document (PAD, p. 4) notes, the burden of these diseases remained an unfinished agenda in Eritrea. The Government’s health sector strategy at appraisal contained well -defined, comprehensive five-year plans for HIV/AIDS/STI (2003-2007), TB (2004-2008), and malaria (2001-2005, and 2006-2010) that provided the framework for the HAMSET II project’s support. Eritrea’s most recent Development Strategy (2005-2007) continues to focus on reproductive health care, treatment and control of common infectious diseases, control of malaria, containing and controlling HIV/AIDS, and acquiring manpower and equipment for the health sector . The most recent Bank strategy document for Eritrea, the June 2008-2010 Interim Strategy Note, supports the government in continued improvement of human development through reductions in incidence of communicable disease, improved reproductive health care, and improved child health outcomes . Relevance of Design is rated Substantial . The PAD (pp. 9-14) contains a discussion of objectives, components, and outcome indicators from which it is possible to infer the connection of planned activities under the project ’s components to its expected outcomes and achievement of objectives. It also contains a results framework (pp. 49-52) clearly specifying the project ’s overall goal, development objectives, impact indicators, and monitoring plans . However, the PAD does not include an explicit description of components /activities that demonstrates the plausible attribution of planned inputs and outputs to outcomes. 4. Achievement of Objectives (Efficacy): AIDS /STI in vulnerable groups as well as the general population through a Contain the spread of HIV /AIDS/ multi -sectoral approach which aims at scaling up prevention, diagnosis, care, and support services for multi- HIV /AIDS/ HIV/ AIDS /STI, with a renewed focus on the most vulnerable populations is rated Substantial . Outputs : The number of voluntary counseling and testing (VCT) sites increased from 74 in 2005 to 135 in the second quarter of 2010. The annual number of VCT attendees increased from 69,121 in 2005 to 96,285 in 2009. The percentage of adults who were both tested and received their results increased from 17% in 2007 to 37.4% in 2009. Provision of prevention of mother -to-child transmission (PMTCT) services increased from 39 sites in 2005 to 93 sites across all six geographic zones in 2009. The number of pregnant women tested for HIV increased from 8,144 in 2005 to 50,000 in 2009, and 42,809 pregnant women received PMTCT services . The ICR does not indicate what coverage this represents . The project team later added that this represented coverage of approximately 38%. Approximately 27 million male condoms and 215,000 female condoms were distributed. 58.891 persons accessed male and/or female condoms. The number of blood transfusions screened for HIV, Hepatitis -B, Hepatitis-C, and syphilis increased from 4,800 in 2005 to 9,300 in 2009. The number of health facilities providing ART increased from 14 in 2007 to 17 in 2009. The number of home-based care clients increased from 119 in 2005 to 349 in 2009, and the number of home-based care volunteers increased from 604 to 2,186 over this same period. 16,322 infected or affected persons were provided with home -based care/support, and 2,665 HIV-infected people were referred for specialized care and /or income generation. 10,815 orphans were provided with care /support. The ICR does not state what coverage this represents; the project team later added that there are no household survey data to provide estimates of the prevalence of OVCs or their characteristics. The ICR presents limited evidence on outputs that would have been expected to have an impact on incidence of HIV specifically among vulnerable populations . Of the community subprojects, 152 were HIV/AIDS-specific, and most of remaining subprojects that integrated diseases contained an HIV /AIDS component; of all beneficiaries reached by the project, 93% were reached by subprojects that included HIV interventions . 175,342 beneficiaries were given information on HIV/AIDS. According to Table 5 (ICR, p. 42), 27,361 persons classified as "mobile population" were reached with unspecified prevention activities . The ICR does not specify what coverage of the mobile population this represents, and data are not provided on other interventions specifically targeting key high -risk groups. The project team later added that these activities included information, education and communication (IEC) and behavior change communication (BCC) by a variety of institutions, and that there are no data on the numbers of members of mobile population groups that would permit an estimation of coverage . Outcomes : HIV prevalence among pregnant women aged 15-24 tested at antenatal clinics decreased from 2.38% in 2005 to 0.88% in 2007 (the target was to maintain prevalence at 2005 levels). The syphilis seroprevalence rate among pregnant women fell from 1.6% in 2005 to 1.1% in 2007, against a target of 1.0%. Data are not provided for the latter half of the project period; the project team later explained that data from a 2009 syphilis seroprevalence survey are not yet available. The percentage of persons testing positive for HIV during VCT decreased from 3.38% in 2005 to 2.06% in early 2010. The percentage of pregnant women testing positive for HIV at sites for prevention of PMTCT decreased from 1.95% in 2005 to 0.66% in 2010. The percentage of infants born to HIV -positive mothers who are HIV-negative increased from 87% in 2007 to 95% in 2009. Data were not available at project closing for HIV prevalence among commercial sex workers (CSWs), which was a key project indicator. The ICR does not explain why these data were not available; the project team later explained that 2009 survey data have not been made available by the Government . However, data were provided for one "hot spot," Asmara, where HIV prevalence among CSWs declined slightly from 8.01% in 2006 to 7.75% in 2009. The percentage of CSWs reporting condom use at last sex with a non -regular partner increased from 76% in 2006 to 87.2% in 2008, effectively meeting the target of 90%. The percentage of truck drivers reporting condom use at last sex with a non-regular partner increased from 78% in 2006 to 96.3% in 2008, exceeding the target of 90%. The ICR (p. 42) reports that 73,458 youth "adopted safe sexual practices, " but it does not specify what this means . The project team later clarified that this refers to youth who used a condom during their last sexual encounter with a non-regular partner. The cumulative number of patients enrolled in anti -retroviral therapy (ART) increased from 709 in 2005 to 5,557 in April 2010. The percentage of adults and children with advanced HIV infections (CD4<250) receiving ART increased from 7% in 2005 to 77% in 2010, against a target of 40%. Expand DOTS coverage, improving case detection and treatment outcomes for tuberculosis is rated Substantial . Outputs : 1,431 health workers were trained on the DOTS strategy, against a target of 700. 12 microscopes were procured and distributed to DOTS sites. 5000 units of anti-TB medicines were procured. The percentage of health facilities with skilled staff trained in detection and treatment of TB remained stable at 100% throughout the project period . The target was set at 95%, below baseline. 240,998 suspected TB cases were screened/tested; 122,747 suspected TB cases were referred; and 86,700 TB cases were given care/support The percentage of health facilities with acid -fast bacilli (AFB) microscopy increased from 79% in 2005 to 100% in 2009, against a target of 98%. The percentage of health facilities with adequate stock of TB medicines over the previous three months remained stable at 100%. The ICR (p. 40) reports that information systems were poor, supervision was weak in health facilities, and there were gaps in the availability of testing equipment . 37,787 HIV-infected people were screened for TB . 32,039 beneficiaries accessed information on TB . A low percentage of the community subprojects focused on TB . Outcomes : The percentage of new smear positive TB cases successfully treated increased from 79.4% in 2004 to 88% in 2009, exceeding the target of 85%. The percentage of infectious TB cases in the population that were detected increased from 41% in 2005 to 49% in 2008, against a target of 70%. The percentage in 2010 when the project closed is not available . 100% of diagnosed TB cases were treated with DOTS, meeting the target . The ICR (p. v) indicates that 100% were treated with DOTS at baseline, but the target was set at 80%. Further reduce or at least maintain malaria morbidity and mortality at current low levels is rated Modest . Although key project outcome targets were met, key output and intermediate outcome targets were not met, and no marginal impact analysis was conducted to identify the impact of the Bank -financed interventions. Under these circumstances, it is very difficult construct a logical chain allowing attribution of observed outcomes to the project-financed interventions. This is in contrast to the earlier HAMSET I project, where project activities (distribution of bed nets, indoor residual spraying, larviciding, and health system strengthening measures ) plausibly led directly to observed intermediate outcomes, such as an increase in bed net ownership from 34% in 2002 to 78% in 2006, that would account for observed declines in malaria -related morbidity and mortality. Under this repeater HAMSET II project, as described below, there was very little progress, or even a reversal, in similar key intermediate outcome indicators (related to bed net ownership and use ). The ICR does not indicate that HAMSET II's activities included indoor residual spraying or larviciding; although the project team added the information that HAMSET II was involved in indoor residual spraying and larviciding, no information is offered on how, where, and to what extent this activity was undertaken. The ICR (p. 21) for this project cites a study arguing that the observed fall in malaria morbidity and mortality is likely due to program interventions outside the project that were implemented on a massive scale in Eritrea, including larviciding, indoor residual spraying, and community -based management of fevers. Outputs : 269,243 persons were trained or informed on malaria control and /or were cleared of mosquito breeding sites . 151,559 persons participated in malaria environmental management . The ICR does not give data on the coverage of the population these figures represent . 748,885 long-lasting insecticide-treated malaria nets (ITNs) were purchased and/or distributed, exceeding the target of 150,000. Approximately 1.1 million rapid diagnostic tests were procured, exceeding the target of 212,520. The percentage of households having at least one long -lasting net or ITN retreated with insecticide within the last 12 months was 73% in 2004 and 71% in 2008, against a target of 80%. The percentage of households having two such nets was 47.7% in 2004 and 40% in 2008, against a target of 80%. The percentage of households aware of at least one environmental management preventive method increased from 68.4% in 2005 to 75.3% in 2008, against a target of 85%. The ICR does not cite studies indicating whether households with fewer ITNs experienced higher incidence of malaria. 84,275 cases of malaria were referred. The percentage of children under five with fever /malaria who received antimalarial treatment according to national policy within 24 hours of onset of fever was 7.5% in 2004 and 4.5% in 2008, against a target of 50%. The percentage of health facilities reporting no stock -out of first-line antimalarial drugs during the previous three months increased from 82% in 2004 to 90% in 2008, against a target of 100%. A low percentage of the community subprojects focused on malaria; however, the project team later added that over 55% of beneficiaries participated in subprojects that involved some form of malaria control activities . Outcomes : Malaria morbidity (confirmed cases) decreased from 35,215 cases in 2004 to 6,785 cases in 2009. Malaria mortality decreased from 24 cases in 2004 to 3 cases in 2009. The target was to maintain 2004 levels. The mortality rate per 100, attributed to clinical and confirmed malaria (all ages), decreased from 1.2% in 2005 to 0.69% in 2009, against a target of 1.08%. The malaria case fatality rate per 100 (under five) decreased from 1% in 2007 to 0.4% in 2009, against a target of 0.99%. The ICR (p. 9) reports that these outcomes were achieved despite high levels of rainfall . The percentage of children under five years of age sleeping under an ITN the previous night was 48.3% in 2004 and 48.9% in 2008, against a target of 75%. The percentage of pregnant women sleeping under an ITN the previous night was 50.4% in 2004 and 54% in 2008, against a target of 75%. Improve the coverage of effective Reproductive Health RH) interventions is rated Modest . (RH) Outputs : 11,709 people were reached with advocacy on gender violence and women's rights . 69,427 youth/adolescents accessed sexual and reproductive health information or services . 121,528 persons in total accessed information on a variety of sexual and reproductive health issues . 198.811 persons were reached with integrated and high -quality sexual and reproductive health services . The ICR does not state what coverage of the population (s) these figures represent. The percentage of public hospitals offering basic emergency obstetric care increased from 55% in 2004 to 100% in 2009, exceeding the target of 75%. The percentage of health centers offering such care increased from 47% in 2004 to 80% in 2009, meeting the target of 80%. The project team added the information that coverage of post -partum care services increased from 5% in 2004 to 29.7% in 2009. The percentage of women who delivered with skilled birth attendance increased from 30% in 2005 to 34% in 2009, against a target of 60%. The percentage of institutional deliveries in rural target areas increased from 26.3% in 2005 to 27.2% in 2008, against a target of 29.9%. The percentage of pregnant women receiving focused antenatal care (four visits) increased from 28% in 2005 to 37.5% in 2008, against a target of 50%. Antenatal clinic utilization (at least one visit) increased from 70.4% in 2002 to 88.5% in 2010. The project team added the information that the number of facilities providing antenatal care services increased from 235 in 2003 to 247 in 2009. The contraceptive prevalence rate (modern methods) increased from 4% in 2005 to 8.4% in 2010, against a target of 10%, according to project data. Demographic and Health Survey (DHS) data show contraceptive prevalence increased from 4% in 2002 (ICR, p. 71, gives a figure of 8% for 2002, DHS data) to 8.4% in 2010. The project team added the information that lab -in-a-suitcase was introduced in 2008, for use in outreach and in facilities where lab services are not accessible . A low percentage of the community subprojects focused on reproductive health . The project team added the information that, while a relatively small number of the community subprojects focused exclusively on reproductive health, over 56.3% of beneficiaries participated in subprojects that had come reproductive health component . Outcomes : The perinatal mortality rate decreased from 48.8/1,000 in 2005 to 44.3/1,000 in 2008, against a target of 39.9%. The project team added the information that the maternal mortality rate decreased from 998/100,000 in 2002 to 486 in 2010, but no information was corresponding to the years of this project . The presence of female genital mutilation (FGM) among girls under ten years of age decreased from 89% in 2002 to 83% in 2010, against a target of 80%. Strengthening the overall health system ’s capacity to address the HAMSET diseases is rated Substantial . Outputs : A database for MOH staff was created, and a functioning staff appraisal system was put in place . All hospitals and health stations conducted staff appraisals on a regular basis . A health worker productivity study was conducted . However, a “human resources for health� strategy and costed plan were not produced as planned . The project team later added that the strategy and costed workplan were produced within six months of project closing . A total of 2,423 health personnel were trained. 440 nurse midwives were trained, against a target of 200. 133 public health technicians were trained, against a target of 200. 300 laboratory technicians were trained, against a target of 300. The project team later explained that the failure to meet the target for training of public health technicians was due to an initial over-estimation of the number of technicians needed; for this reason, training of these technicians was discontinued in 2008. The first class of medical students graduated from the Eritrea School of Medicine . Ten students received a Masters in Anesthesiology from Moi University, Kenya . Institutional arrangements for health training institutions were improved so that they fall under the Board of Higher Education . Health training facilities received unspecified renovations. An unspecified number of computers and textbooks were procured and distributed . The instructor-to-student ratio decreased from 1:25 in 2005 to 1:20 in 2010 against a target of 1:15. Outcomes : The percentage of health stations with at least one nurse was 28% in 2005, 45% in 2007, but fell to 35% in 2009, against a target of 50%. The ICR does not explain the decline from 2007 to 2009. The project team later explained that these data in the ICR are incorrect, and that the percentage of health stations with at least one nurse increased from 28% in 2005 to 35% in 2010. The percentage of health stations able to prepare microscope slides increased from zero in 2000 to 100% in 2010, against a target of 100%. The ICR does not provide data for the specific dates of the project . The project team later added that this indicator increased from zero in 2005 to 100% in 2010. 5. Efficiency (not applicable to DPLs): Efficiency is rated Substantial . The PAD (pp. 120-123) contains an economic analysis arguing that the project would have positive effects on Eritrea ’ s economic growth, poverty alleviation, and cost -effective use of resources, but it did not perform calculations specific to the contributions of the HAMSET II project . The ICR (pp. 47-52) contains a similar discussion . The ICR’s cost-effectiveness analysis (pp. 23-24, 47-52) cites studies on the cost -effectiveness of various kinds of interventions to combat HAMSET diseases, indicating that high cost -effectiveness ratios are observed for activities that require integrated management of multiple diseases . A majority (77.3%) of the community-based subprojects addressed more than one disease, and more than 90% of beneficiaries were impacted by subprojects that integrated at least two HAMSET diseases. It is therefore plausible that integrated management resulted in increased efficiency, although these purported efficiencies are not demonstrated by the ICR . There may also have been synergies among interventions and efficient integration of health system strengthening with vertical disease programs . In addition, the project adopted an implementer-based approach, with project components defined by their implementers rather than by the diseases with which they were concerned, improving coordination and efficiency . Ministries, departments, and sectors selected under the multisectoral component were based on an assessment of their comparative advantage . Also, technical and supervision support to the CMHRP subprojects was provided by the technical MOH staff, strengthening the quality of input and efficiently integrating CMHRP into the broader health sector response . Careful attention was paid to improving the efficiency of the community -based subprojects over the lifetime of the project. Over time, subprojects were chosen more selectively, the percentage of activities that were integrated rose, the size and duration of subprojects increased, and the number of beneficiaries per subproject was increased in order to drive down average subproject costs . The ICR reports, however, that by 2009, average beneficiary cost was US$3 for information/education and behavior change subprojects, and US$ 15 for care and support subprojects; this could be considered high in a country where total expenditure on health is approximately US$ 10 per capita. However, the project team later added that the per capita cost for all subprojects decreased from US$ 31 during the project pilot phase to US$ 5.11 in the first quarter of 2008, indicative of significant improvement in cost -effectiveness over the lifespan of the project . The ICR (pp. 25, 48) claims that the project realized efficiencies through focused HIV prevention interventions involving high-risk groups (sex workers, truck drivers ), but it does not provide specific data or evidence on these interventions anywhere in the ICR . The project team later provided information on activities involving CSWs, including empowering them to form groups, mobilize funding, implement programs, and take action to protect themselves against HIV and violence; activities involving mobile populations focused on IEC /BCC. The ICR does not explain why significantly more was spent than planned on project management . The project team confirmed that the additional spending was primarily on monitoring and evaluation activities . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The project’s objectives were Highly relevant and its design was Substantially relevant, and its efficiency was Substantial. Although three of the project's objectives were Substantially achieved (HIV/AIDS/STI, TB, and health systems strengthening), two objectives (malaria control and reproductive health ) achieved efficacy ratings of Modest . Key output and intermediate outcome targets for malaria control were not reached, and therefore it is difficult to attribute observed results on malaria control to the project's interventions . Very few key targets were met for reproductive health. a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: Institutional strengthening appears likely to be sustained . The Government intends to continue the Joint Annual Review process, steering committees and task forces connecting the national and zoba level remain in place, and the Project Management Unit (PMU) is now being used by other donors, including the Global Fund . The ICR (p. 45) expresses concern, however, that the short average length of community -based subprojects (six months) was insufficient to build capacities and ensure lasting behavior change . Financial risk, however, is substantial . While GDP growth was sustained at an average of 2% over the project period, external and domestic debt are at extremely high levels, and the country ’s currency is estimated to be about six times overvalued. At project preparation, it was estimated that US$ 1.4 million in recurrent annual costs would be needed after the project for maintenance, supervision, and continuance of training activities; it is not clear that those resources will be available. Technical risk is also substantial . While the project invested in human resources for health, resulting in increases in capacity and skills, Eritrea’s continued isolation limits opportunities for future learning from external sources . According to the ICR (p. 29), there does not appear to be an alternative development partner who is able to provide levels of technical assistance comparable to the Bank ’s. The persistence of fuel shortages and suspension of private contractors, both of which contributed to procurement, construction, and supervision difficulties during project implementation, show no signs of being resolved and are likely to continue to risk sustainability of achievement of development objectives . The current leadership of the Eritrean health sector appears to have limited capacity to overcome these operational barriers and mobilize resources. Substantial external risks include variations in rainfall that could impact achievement in malaria control, the possibility that border disputes could escalate into full -scale conflict, and food insecurity . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: at-Entry is rated Moderately Satisfactory . Ensuring Quality -at- Overall Quality-at-Entry was rated “highly satisfactory� by the Quality Assurance Group (QAG). Preparation was highly participatory and consultative, with meetings including a broad array of stakeholders . The project learned key lessons from its predecessor HAMSET project, as well as related project in the region : the importance of renewed focus on targeted interventions for vulnerable groups and their linkages to the general population; the need for further strengthening of the M&E system; the need for community -driven efforts on disease control and behavior change; the importance of ensuring that lower -level implementers prioritize the most effective interventions; and the importance of addressing reproductive health issues simultaneously with the HAMSET diseases. Lessons were also learned from the Eritrea Health Project (1998-2004), especially with respect to developing a focused results chain, favoring training and policy development activities over construction, and including malaria control interventions . The Interim Review of Multi-Country HIV/AIDS Program (MAP) projects (2004) also offered useful lessons, including moving toward a performance -based disbursement system, involving civil society in the design of the application process for subgrants, and ensuring adequate resources for development of operational M&E systems . The ICR (p. 8) points out that the project could have benefited from additional analysis of government capacity for the implementation of the health care waste management safeguard . Risks were candidly identified, and reasonable mitigation strategies identified and adopted . Two identified risks, relatively weak M&E and underlying national cross -cutting challenges and macroeconomic constraints, eventually posed implementation challenges . The ICR (p. 9) states that it is unlikely the project team could have done more during preparation to further mitigate these risks . The project was complex in design, including four different disease interventions and human resource strengthening. The project’s organization by implementing agency, rather than by disease, was a departure from the first HAMSET project. This innovative design element facilitated implementation by following existing organizational structures and allowing multi -sectoral and community-based actors to implement multi-disease components more easily. The inclusion of human resources for health was a very strong element of design, not financed by other donors; it signaled recognition that progress on all disease programs would be limited without this essential capacity-building. Some of the key performance indicators were flawed, in particular the use of HIV prevalence, which is known to conflate incidence with mortality and migration . Original targets for receipt of focused antenatal care and delivery with skilled birth attendance were later judged by the project team to be too ambitious (ICR, p. 11). Revisions to these indicators were proposed at the Mid -Term Review (MTR), but ultimately these proposed revisions were not incorporated into the results framework; the ICR (p. 10) does not provide a complete explanation for this decision . The project team clarified that the failure formally to revise the indicators was influenced by the transaction costs . Quality of Supervision is rated Moderately Satisfactory . The Bank put together large teams to provide technical assistance, with missions typically including experts across all the HAMSET diseases and other areas . Supervision missions were used as effective monitoring and planning tools. Conducting supervision visits and holding national -zoba and inter-zoba meetings was difficult because of limited availability of fuel and vehicles, but these constraints were overcome, suggesting a high level of commitment to implementation success . There were no missions during the last 18 months of the project, due to uncertainty over whether the Government would sign a supplemental letter, but the Bank team continued to hold semi-annual “missions� via videoconference. However, there were shortcomings . For most of the project period, the Implementation Status Reports (ISRs) did not include all of the key project indicators, meaning that the results framework did not effectively fulfill its function as a monitoring tool (the ICR does not specify the extent of this problem, for example, whether some or a majority of indicators were missing from the ISRs ). This problem was not rectified until very late in the project period (late 2009). Also, although the coordinating unit for M&E had prepared an integrated work plan for the project, the Bank (and the PMU) often bypassed it, going directly to technical managers for data and therefore undermining the M&E Division’s core functions. At the MTR, staffing for the Environmental Health Unit (EHU) was strengthened, contributing to improvements in the implementation of the Health Care Waste Management Plan (HCWMP). The project entered potential problem project status in early 2009, however, when the failure to construct a safe storage facility for DDT and delays in preparing the HCWMP triggered the pest management and environmental safeguards . According to the ICR, "It is not surprising, then, that overall progress in HCWM was weak . By June 2009, only moderate progress had been made in following the recommendations set out in the 2005-2009 Health Care Waste Management Plan. Also, the development, distribution and implementation of HCWM policy, planned for 2007, was only developed and disseminated within the last six months of 2009, and was not yet fully implemented at project close" (p. 16). The project "emerged from potential problem status in January 2010, after significant progress on these safeguards had been made " (ICR, p. 12), with no operational implications. A safe storage facility for DDT is expected to be completed in 2011. According to the ICR, the team took many proactive steps to help in the resolution of health care waste management, including reallocation of funding across categories and providing expert consultant advice (ICR, page 31). The ICR does not indicate if this issue was resolved . The project team later explained that the issue was largely resolved, with health care waste management rated Satisfactory at closing. The necessary regulatory framework was developed and adopted, health care waste management committees at the national and zoba levels were established and staffed, and progress was made in the nation-wide distribution of segregation material for storing and disposing of health care waste . Key staff have been trained and are in place to oversee key aspects of health care waste management across the country . See Section 11. at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Government Performance is rated Moderately Satisfactory . The ICR (p. 7) states that the high quality of Government inputs was a “critical contributing factor� to the quality of project preparation. Counterpart funds were made available in a timely fashion . The Government exercised leadership in convening multisectoral partners and various stakeholders . However, Government policies resulted in uneven availability of foreign exchange and private contractors, fuel shortages, and travel restrictions, with particular negative impact on civil works and project supervision, especially in more remote areas . Implementing Agency Performance is rated Moderately Satisfactory . The PMU is credited with high levels of staff continuity, cohesiveness, familiarity with Bank procedures and processes (from the first HAMSET project), and a constructively self -critical disposition. The PMU possessed very strong project management and implementation skills that enabled them to overcome many obstacles and constraints. However, the Borrower’s ICR (p. 59) states that the PMU, although extraordinary in its commitment throughout the project period, was sometimes constrained by lack of adequate focus and technical expertise in several program areas. M&E was also a relative weakness, with focus across implementing agencies on inputs and process indicators rather than outcomes . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: M&E Design : M&E was viewed as a Substantial project risk during preparation . Mitigation measures included applying a Lot Quality Assurance Survey (LQAS) methodology in M&E; making the operational plan for M&E of HAMSET disease a project condition; establishing a computerized Management Information System (MIS) in the PMU; recruiting M&E staff for the PMU; and ring-fencing the budget for M&E. Overall, the results framework and M&E arrangements for the project were strong . Indicators from the National Strategic Plan and the project were well harmonized . Outcome indicators were specified for a wide spectrum of beneficiaries, including commercial sex workers, bar /hotel/tea shop workers, truck drivers, pregnant women, children, TB patients, and the general population . However, there are inconsistencies in the PAD on the indicators (Annex 3, Results Framework with Illustrative Indicators, pp . 49-52; versus the key indicators discussion in the main text, pp. 9-10), intermediate indicators were lacking, and baseline data for many indicators were not available until well into project implementation. M&E Implementation : The institutional structure of M&E was fragmented, with key units spread across three departments . Also, responsibility for the Health Management Information System (HMIS) and for overall M&E supervision and direction lay with two different entities, severing the link between information collection and analysis /use of those data for decision making. The MOH (Department of Research and Human Resources ) performed routine data collection for the technical programs well, using a sophisticated electronic HMIS . Reporting by line ministries and civil society implementors, however, was weak, with reporting focusing overwhelmingly on inputs, processes, and activities rather than outcomes. As a result, the results framework was limited as a tool for project performance measuring and monitoring. The Community-Managed HAMSET Response Program (CMHRP) successfully established a management information system in all zones . The system tracks data on planned and actual activities, target groups, expenditures, and some results, with disaggregated data down to the subproject level . However, this system was not successfully integrated with the MOH HMIS, did not include baseline values and targets for subprojects, and was not modified to include outcome indicators . Lack of human resource capacity was a consistent problem . The MTR recommended that additional staff be recruited for M&E, but these vacancies took a long time to fill, and at project closing the M&E Division still lacked capacity. M&E Utilization : Progress toward objectives and targets was discussed at the Joint Annual Reviews, and data -driven adjustments were made to programs, workplans, and budgets . In general, however, the use of data for decision making was poor, especially at the zoba level, although this varied by disease program . Also, there is no mechanism for identifying and evaluating research priorities, so that evaluation research and its use for policy are weak . Exceptions to this observation were an evaluation (with an experimental research design ) carried out by the National Malaria Control Program, and an evaluation of “Best Practices� among the community-based subprojects. Some multisectoral partners conducted surveys, but their results were not systematically compiled and analyzed . The ICR (p. 15) reports that some multisectoral partners documented success stories, but these were not shared with the PMU or the Bank. a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards : Safeguard policies for Environmental Assessment (OP/BP/GP 4.01) and Pest Management (OP 4.09) were triggered. The project was classified as category “B� (partial assessment) at the time of appraisal because of the risks associated with the handling and disposal of medical wastes, but as a repeater project it was permitted to use the existing safeguards arrangements that had been developed and approved under HAMSET . These included the Health Care Waste Management Plan (HCWMP) of 2004 and the creation of an MOH task force to ensure dissemination and compliance . At the MTR, staffing for the Environmental Health Unit (EHU) was strengthened, contributing to improvements in the implementation of the HCWMP . The project entered potential problem project status in early 2009, however, when the failure to construct a safe storage facility for DDT and delays in preparing the HCWMP triggered the pest management and environmental safeguards . According to the ICR, "It is not surprising, then, that overall progress in HCWM was weak . By June 2009, only moderate progress had been made in following the recommendations set out in the 2005-2009 Health Care Waste Management Plan. Also, the development, distribution and implementation of HCWM policy, planned for 2007, was only developed and disseminated within the last six months of 2009, and was not yet fully implemented at project close " (p. 16). The project "emerged from potential problem status in January 2010, after significant progress on these safeguards had been made" (ICR, p. 12), with no operational implications. A safe storage facility for DDT is expected to be completed in 2011. According to the ICR, the team took many proactive steps to help in the resolution of health care waste management, including reallocation of funding across categories and providing expert consultant advice (ICR, page 31). The ICR does not indicate if this issue was resolved . The project team later explained that the issue was largely resolved, with health care waste management rated Satisfactory at closing . The necessary regulatory framework was developed and adopted, health care waste management committees at the national and zoba levels were established and staffed, and progress was made in the nation -wide distribution of segregation material for storing and disposing of health care waste . Key staff have been trained and are in place to oversee key aspects of health care waste management across the country . Fiduciary : All financial management reports prepared by the PMU were submitted on time and were judged to be of good quality. There were no unqualified audits . No accountability issues were identified, and at project closing, there were no outstanding financial management issues . The only difficulty was in the area of funds flow; on occasion, there were problems obtaining bank statements from remote zobas, resulting in delays in accounting for advances, the transfer of funds from the PMU to the zobas, and replenishment of the Designated Account . A 2009 Financial Management Assessment Report of the PMU, conducted by Bank staff, rated financial management as Satisfactory. For procurement, the project used the same institutional and control arrangements that were used for HAMSET, with overall responsibility assigned to the PMU . Delays were minor, and by closing, all procurements that has been envisaged under annual procurement plans had been executed . Unintended positive impacts : During implementation, considerable resources were expended on the development of a detailed plan for results-based financing (RBF) pilot activities under the reproductive health subcomponent . While the pilot was never implemented, groundwork was laid and interest sparked for future RBF initiatives, with potential long-term effects on health financing and access to reproductive health services in the country . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Significant Significant Outcome : Bank Performance : Satisfactory Moderately For most of the project period, the Satisfactory Implementation Status Reports (ISRs) did not include all of the key project indicators, meaning that the results framework did not effectively fulfill its function as a monitoring tool. This problem was not rectified until very late in the project period (late 2009). Also, although the coordinating unit for M&E had prepared an integrated work plan for the project, the Bank (and the PMU) often bypassed it, going directly to technical managers for data and therefore undermining the M&E Division’s core functions. Health care waste management was a persistent safeguards issue for the project . The team took many proactive steps to help in its resolution, including reallocation of funding across categories and providing expert consultant advice (ICR, page 31). The ICR does not indicate if this issue was resolved; the project team later added information on the satisfactory resolution of health care waste issues (see Section 11). Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: An implementor-based approach can realize synergies more effectively than an approach defined by individual diseases. Integration of three vertical disease programs, together with reproductive health and human resource strengthening, by implementors -- but only where sufficient capacity exists -- allows for multiple integrated interventions and facilitates the search for synergies and efficiencies . Continuity of staff matters. By the time HAMSET II closed, most staff in the PMU had worked together on HAMSET projects for ten years, facilitating collaboration, trust, and efficiency . If a project chooses to rely on external surveys to evaluate performance, it must exercise some control over the mechanisms and timing of those surveys . In this case, the project relied on DHS data for a large share of endline data, but the 2010 DHS data were not available at closing . As a result, some data presented as endine data were from two years prior to closing . 14. Assessment Recommended? Yes No Why? To verify the ratings and document lessons learned . 15. Comments on Quality of ICR: The ICR is clear, concise, and evidence -based. It harnesses a wide range of data sources to assess the project ’s achievements and outcomes, recognizing that the aim is not just to measure performance on specified indicators, but more broadly to assess achievement of development objectives . The section on project efficiency provides rich data and discussion. a.Quality of ICR Rating : Satisfactory