ICRR 13289 Report Number : ICRR13289 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 03/29/2010 PROJ ID : P073603 Appraisal Actual Project Name : Hiv/aids, Malaria And US$M ): Project Costs (US$M): 15.0 n/a Tuberculosis Control Project Country : Djibouti Loan/ US$M): Loan /Credit (US$M): 12.0 15.0 Sector Board : HE Cofinancing (US$M): US$M ): 2.3 n/a Sector (s): Other social services (60%) Health (30%) Central government administration (10%) Theme (s): Participation and civic engagement (23% - P) Malaria (22% - P) HIV/AIDS (22% - P) Gender (22% - P) Social analysis and monitoring (11% - S) L/C Number : CH042 Board Approval Date : 05/29/2003 Partners involved : UNDP, France Closing Date : 09/30/2008 09/30/2008 (Ministry of Foreign Affairs), UNICEF, UNFPA, WHO Evaluator : Panel Reviewer : Group Manager : Group : Judith Hahn Gaubatz Kris Hallberg IEGSE ICR Reviews IEGSE 2. Project Objectives and Components: a. Objectives: According to the Development Grant Agreement and Project Appraisal Document (PAD), the overall project objectives were to (i) contribute to the change in behavior of the Djiboutian population in order to contain or reduce the spread of the HIV/AIDS epidemic and to mitigate its impact on infected and affected persons; and (ii) contribute to the control of malaria and tuberculosis . For the purposes of this review, the project will be assessed against the following separate objectives : (i) to contribute to the change in behavior of the Djiboutian population in order to contain or reduce the spread of the HIV/AIDS epidemic; (ii) to mitigate the impact of the HIV/AIDS epidemic on infected and affected persons; (iii) to contribute to the control of malaria; and (iv) to contribute to the control of tuberculosis . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): (1) Capacity Building and Policy Development : (Appraisal: US$4.6 million; Actual:US$4.9 million ) Activities focused on strengthening the Government's capacity to combat HIV /AIDS, malaria and tuberculosis, specifically enhancing counseling, voluntary testing (VCT), and care and treatment of people living with HIV /AIDS (PLWHAs). AIDS /STIs, Malaria and Tuberculosis : (Appraisal: US$2.3 million, Actual: (2) Health Sector Responses to HIV /AIDS/ US$3.8 million ). Activities focused on increasing access to preventive measures for sexually transmitted infections (STIs), opportunistic diseases and malaria, as well as access to case management, support and treatment for PLWHAs. AIDS (Appraisal: US$2.6 million, Actual: US$3.7 million ). Activities focused on (3) Multisectoral Responses to HIV /AIDS: strengthening the management capacity of eleven Ministries (including Defense, Education, Youth and Sports, Justice, Interior, Employment, Communication, and Women ) to implement HIV/AIDS activities in their sectors, as well as supporting the Ministries in implementing activities aimed at their own personnel . (4) Support to Community Interventions (Appraisal: US$2.5 million, Actual: US$2.6 million ) Activities focused on the implementation of community subprojects prepared by community based associations (CBAs), which were to include condom distribution, information and education campaigns (IEC), gender issues, psycho -social support, VCT, prevention of mother-to-child transmission (PMTCT), prevention of malaria and support to tuberculosis patients . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Actual disbursements exceeded the planned disbursements, in terms of the US$ amount, due to exchange rate fluctuations. However, the XDR amount remained the same . According to the TTL, the final project cost is not available as numerous financiers contributed to the overall program and an actual total was not calculated . Annex 1 of the ICR shows that counterpart funding of S$ 0.72 million was planned but provides no actual contribution figure . 3. Relevance of Objectives & Design: The relevance of the project objectives is rated Substantial. Combating HIV/AIDS, tuberculosis and malaria is a high priority for the Bank and the Government . The Bank's Country Assessment Strategy for 2009-2012, as well as the Government's National Initiative for Social Development Plan launched in 2007, proposes to support access to basic social services and human development, including improving the quality of prevention and treatment of communicable diseases. The Government has also prepared national strategic plans for each of the three diseases . However, while the HIV/AIDS objective was well-articulated (namely, focusing on behavior change rather than on general HIV prevalence rates), the malaria and tuberculosis objectives were not as well -defined. The relevance of the project design is rated Substantial. The design supported responses in multiple sectors, with an explicit role for the health sector, which plays a crucial role in combating the three communicable diseases . The Ministry of Communication was notes as also having a critical role, particularly at the early stages of the project . (The PAD notes that support to the Ministry of Transport was not included as it was already receiving support from other donors). The design also emphasized capacity building interventions in order to address weaknesses in implementation capacity, both on managerial and technical issues . With regard to monitoring and evaluation (M&E), a seroprevalence survey was undertaken early on to provide a more accurate estimate of the HIV prevalence rate, which was determined to be 2.9 percent in the general population and more than 5 percent among persons aged 20-35. The survey also provided information on the most common forms of transmission, information which was then reflected in the project design as tailored interventions to the priority vulnerable groups (including HIV/AIDS education for students and youth, prevention activities in prisons and with police, and access to condoms for police force and military). 4. Achievement of Objectives (Efficacy): Objective 1: To contribute to the change in behavior in order to contain or reduce the spread of the HIV /AIDS epidemic Achievement of this objective is rated Substantial . The ICR presents evidence of an increase in condom use in some high risk groups, as well as a decrease in HIV prevalence among pregnant women aged 15-24 years, which can serve as a proxy for incidence among that age group . Outputs : The project supported the Ministry of National Education in introducing a mandatory training module on HIV /AIDS and the opening of school health clubs in numerous establishments . More than 4,000 school agendas for awareness and prevention messages on HIV /AIDS, malaria and TB were reviewed and disseminated on radio and television . The Ministry of Youth and Sports re -fitted its community development centers to disseminate information and offer basic products related to HIV /AIDS, malaria and tuberculosis. 750 peer educators were trained, including youth peers in schools and community development centers, peers within each of the 11 Ministries, and peers within 170 community-based associations (CBAs). The project supported two international conferences and various in -country awareness workshops, which contributed to the unanimous declaration by religious leaders that the use of condoms is a means of prevention . The project also contributed to the increased availability of condoms, as 2,385,979 condoms were distributed by 388 distribution sites. The project also supported a social marketing strategy which was completed in 2006 and included creation of a social marketing NGO. Outcomes : The ICR reports the following evidence of an increase in condom use : The percentage of youth that used a condom during the last casual sexual encounter increased from 5.0% in 2002 to 38.3% in 2007, compared to the target of 50% by the end of 2008. Condom use among the general population is reported as 27% in 2007 and among commercial sex workers as 94%, although there are no specific baseline figures reported to allow comparisons from before the project period and neither statistic is well -defined. (For example, it is not clear whether the 27% use by the general population is use at the last sexual act with any partner, including spouses, or only with non -marital or casual partners, or whether it represents ever use of a condom .). Condom use among uniformed workers remains at a low level, although it did increase from less than 1% to 13% by the project end. (The PAD reports that the rate of condom use during the last non -union sexual encounter among high -risk groups (uniformed persons, CSWs and dockers) is 20% and is targeted to increase to 50%, but does not provide a breakdown for each group specifically). With regard to the containment of the spread of HIV /AIDS, the ICR reports the following data as proxy indicators of HIV incidence: According to the sentinel surveillance data, HIV prevalence among pregnant women aged 15-24 years decreased from 2.7% in 2002 to 2.4% in 2005 and to 1.9% in 2008. HIV prevalence also decreased overall among sentinel surveillance groups from 2.5% in 2002 to 1.9% in 2008, according to additional information provided by the region. However, the PAD reports that prevalence is higher than 5 percent among persons aged 20-35, which may indicate that there is significant mortality among the 20-24 year old age group, thus the reported prevalence rate may reflect both new infections and deaths. HIV prevalence among STI patients decreased from 22.0% in 2002 to 9.7% in 2006. There was no target provided. However, trends in HIV prevalence among STI patients are difficult to interpret, given the fact that HIV and other STIs are spread in the same manner and increases in condom use would decrease the number of STI patients as well as HIV co-infected STI patients. The project design does include the tracking of STI prevalence rates among pregnant women, which might be a more accurate indicator of changes in risky behavior . However, the ICR reports that data were not collected on this indicator . Although the ICR presents the above evidence of an increase in condom use, there is little discussion of other types of behavior change (such as a decrease in the number of sexual partners ) which also affects HIV incidence rates . Objective 2: To mitigate the impact of the HIV /AIDS epidemic on infected and affected persons . Achievement of this objective is rated Substantial, with caveats . The ICR presents evidence of increased usage of services intended to provide support to PLWHAs, such as counseling, anti -retroviral (ARV) treatment, nutrition support, and legal protection . However, in particular there is a lack of evidence on any increase in proportion of persons covered. Outputs : The project supported the production and dissemination of normative documents for HIV /AIDS case management, training of medical staff on the use of treatment protocols, and the provision of ARV drugs, medical supplies, office and laboratory equipment. Voluntary counseling and testing protocols were completed in 2005, and initial training and refresher courses for VCT counselors and laboratory technicians were carried out . 100 psychosocial support and therapeutic extension workers were trained, as part of a key innovation of the project which introduced "psychosocial supporter" staff who were trained as part of the case management teams to provide psychological and social support to PLWHAs. The project also contributed to the scaling up of VCT services from one VCT center to 21 centers nationwide. Other outputs included: 2,550 persons benefited from dry rations distributed to home -based patients by CBAs (World Food Program provided the rations). 610 hospitalized HIV patients received three hot meals per day, though the time period for receiving the meals is not clear. 300 HIV-positive women received psychosocial support from CBAs . 600 orphans and other vulnerable persons received psychosocial support, nutrition, school supplies and school fees, and financing of income -generating activities for their guardians . Adoption of a law protecting the rights of PLWHAs and their families against discrimination and providing assistance in cases of litigation . In addition, the ICR reports that prior to the project period, only about a dozen NGOs and CBAs operated in the country. By the project's end, a total of 170 CBAs were supported by the project with institutional, managerial and capacity building activities. Outcomes : The ICR reports that more than 2,000 PLWHAs were registered for HIV/AIDS case management, of which 1,541 were receiving ARV treatment. The survival rate of those receiving treatment increased from 75% in 2005 to 88% in 2007, though there was no target provided and the length of survival measured is not clear . However, while the ICR reported the above evidence on the increased use of services, there is limited information provided on improved quality of services or of their coverage or effectiveness . Objective 3: To contribute to the control of malaria Achievement of this objective is rated Modest . There is limited evidence of containing the incidence of malaria, particularly as utilization rates of insecticide -impregnated bed nets (IIBNs) were low. Outputs : 42,000 IIBNs were purchased and supplied in high -prevalence areas. (An additional 118,000 were supplied by the Global Fund, UNICEF, and the African Development Bank over the same period, nationally .) 25 community subprojects were funded to conduct training on utilization, impregnation and re -impregnation of bed nets. Three vector-control spraying machines were purchased . More specific details on the usage of the machines is not reported in the ICR. Outcomes : The IIBN coverage survey of July 2007 showed that only 25% of sampled households had IIBNs and that only 18.9%-29.1% of vulnerable groups (pregnant women and children under the age of five ) had slept under IIBNs the night preceding the survey . The ICR suggests that the utilization rate might have been underestimated because the survey was conducted in a month when rainfall levels were 20-50% below normal. The PAD provides a target of 15,000 households using IIBNs; however, the ICR does not report what share of households this would represent . With regards to malaria treatment, the PAD includes hospital mortality rates due to malaria as an outcome indicator, but the ICR does not report any information on this outcome or its trend . Objective 4: To contribute to the control of tuberculosis Achievement of this objective is rated Modest . Although the percentage of lost cases among TB patients was reduced, there is limited evidence of containing the incidence of tuberculosis (which the PAD reports as the second highest in the world , at 588 cases per 100,000 inhabitants). Outputs : The main research center received equipment and drugs . 210 TB patients received nutritional support . 1,200 patients were treated for tuberculosis . Outcomes : The targeted reduction of "lost cases" (i.e. those initiating but not completing treatment ) from 24% to 15% was achieved. The 1,200 patients were treated with a success rate of 81%. There was no target provided . For all four objectives, it should be noted that while the ICR reports activities of other donors (i.e. the Global Fund began funding activities on HIV /AIDS, tuberculosis and malaria in 2004 and provided US$30.5 million during the period 2004-08), there is limited discussion on the issue of attribution and the specific contribution of the Bank to overall outcomes. 5. Efficiency (not applicable to DPLs): The ICR reports that important efficiencies were achieved in terms of coordinating activities with the Global Fund and the French Government through the Executive Secretariat, and adherence to common monitoring frameworks . The initial emphasis of the project on activities in 11 ministries in addition to the Ministry of Health – many of them with no particular comparative advantage in addressing the epidemic – was reduced to fewer higher priority ministries (Education and Youth and Sports ), improving the efficiency of the project in implementation relative to its design . However, while the project was designed to ensure coverage of the highest risk groups, the ICR provides only limited evidence that better access of these groups was achieved and their behavior changed . To the extent that the highest risk groups weren’t actually reached and affected, some of the efficiencies in the project design were not realized in its implementation. The ICR presents in a detailed annex the economic and financial analysis from the PAD, which estimated the present value of the projected accumulated costs of the epidemic (over the period of 2002-2028) as 13% of GDP of 2002. It updates these figures using information from implementation of project activities and re -estimates the present value to have been reduced to 3.8% of GDP of 2002. The analysis assumes that in the absence of the project HIV prevalence would have risen to nearly 17% by 2025, as opposed to less than 1% with the project in the same time frame, and that this impact is achieved by implementing awareness campaigns and better access to condoms . The analysis in the ICR finds that even though the benefits are calculated to be lower as of the end of the project ($52 million), they are four times the project costs ($12 million). The costs of ART for AIDS patients in the future are excluded, and the ICR notes that their inclusion would likely have raised the benefits of preventing future infections . This analysis, however, has a number of shortcomings : (a) The awareness intervention and better access to condoms are assumed to be effective in changing behavior and the benefits calculation assumes that the annual value of a human life saved is the equivalent of Djibouti ’s GDP/capita ($860). (b) The model dates from 1998 and its main outcome variable is HIV prevalence . ART was not widely used back then, but it is now . This affects the assumptions on the mortality rates but also the interpretation and usefulness of HIV prevalence as an outcome variable (since people under treatment remain HIV positive and live longer ). Incidence – the number of new cases— would be a more appropriate outcome variable for the model to track . (c) The costs include only the costs of the project, not the outlays of the government and other donors (the Global Fund financed $30.5 million over the same time frame, for example) or the continued funding of the programs into the future . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The overall outcome is rated Moderately Satisfactory . The project’s objectives and design were both substantially relevant, but there were moderate shortcomings in achievement of the malaria and TB objectives, and the ICR presented limited evidence of behavior change . According to the Harmonized Evaluation Criteria, the rating of moderately satisfactory applies when “there are moderate shortcomings in the operation ’s achievement of its objectives, in its efficiency, or in its relevance .� a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The risk to development outcome is rated Negligible to Low .. The government has sustained its high level of commitment, as reflected by a follow-up five-year strategic plan and funded program, and the securing of additional donor funding, (GF and IDA). However, sustainable funding for ARVs remains uncertain in the long term . a. Risk to Development Outcome Rating : Negligible to Low 8. Assessment of Bank Performance: at -entry is rated Satisfactory. As mentioned previously, several surveys were conducted during project Quality -at- preparation which established critical baseline information . Knowledge, attitude and practices surveys, as well as qualitative social assessments, were also conducted and informed the strategic directions of the project, although more specific M&E indicators related to behavior change (other than condom use) and targeted decreases in levels of TB and malaria were not included . Weak institutional capacity at the national and local level was recognized early on and capacity building activities were planned accordingly . Priority vulnerable groups were mapped. Strong commitment from political leadership was also ensured early in the project preparation process. A number of these issues were appropriately identified as risk factors and thus adequately addressed. Supervision is rated Satisfactory. Capacity building activities were effectively carried out to strengthen program management, including with regards to fiduciary functions . The ICR reports that extensive supervision missions were conducted early in the project period, including intensive review of manuals and guidelines to facilitate complex implementation. Implementation of activities directly related to achieving the project objectives were a clear focus. Joint supervision missions were conducted with GF, which was a key factor in successful collaboration for implementation effectiveness, and harmonization of M&E with other donors enabled a functional M&E system. However, the environmental and medical waste management plans were not implemented in a timely manner. at -Entry :Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Satisfactory 9. Assessment of Borrower Performance: Government performance is rated Satisfactory. The ICR reports that the Government had a high level of commitment to fight the three diseases . The Government issued several decrees critical to facilitating implementation, such as establishment of the Interministerial Committee and Secretariat, and establishment of the Medical College for HIV/AIDS. The permanent technical committee effectively carried out its technical oversight role, including reviewing action plans and annual budgets and ensuring consistency with the three national strategic plans. The Government mobilized additional donor funding and passed major legislation to protect the rights of PLWHAs. However, as mentioned previously, environmental management plans were not implemented in a timely manner. Implementing agency performance is rated Satisfactory. Line ministries carried out most activities as planned, although initially some Ministry focal points were not adequately committed and were frequently replaced, contributing to implementation delays The ICR reports that with the aid of supporting agencies, CBAs also implemented a number of project activities as planned . According to the region, a change was made to this component during the first year of implementation so that these supporting umbrella agencies would be contracted to supervise and provide capacity building support to the CBAs . Although this design change led to some implementation delays, it was a positive example of adaptability on the part of the implementing agency and did not significantly affect the delivery of outputs of this component . a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: M&E Design is rated Modest. The project objectives and a number of key indicators on HIV /AIDS were adequately defined and realistic given the time frame of the project . However, the design does not refer to other critical indicators of behavior change besides condom use, and also does not provide clear indicators to measure mitigation (i.e. quality of life for PLWHAs or increased levels of coverage ) or incidence of tuberculosis or malaria . M&E Implementation is rated Modest. The ICR reports that although the M&E system was mostly measuring process indicators at the beginning of the project, a concerted effort was made to coordinate reporting systems and produce more useful and relevant data . However, although the design included indicators for high risk groups other than youth, these are not reported on in the ICR . Also, a number of end-project surveys were not conducted, which, according to the ICR, was due to the fact that not enough time had elapsed since the Mid -Term Review surveys and that project funds were almost fully disbursed . M&E Utilization is rated Substantial. The ICR reports that the harmonized system has been used for managing activities. For example, timely monitoring of VCT services enabled partners to plan related services accordingly i .e. procurement of ARVs and provision of psychosocial services . a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): With regard to CBAs, the project supported significant expansion of the civil society sector . This included developing the capacity of large NGOs to become supporting agencies and thus helping to build capacities of CBAs. The environment safeguard rating was assessed as "B" due to medical waste management issues . According to the ICR, the environmental management plan was not produced until the last year of the project due to an early disagreement between the Bank and the Executive Secretariat on the implementation plan . Moreover, the implementation of the medical waste management plan was not initiated under the project, although it was agreed with the MOH to implement it under another Bank health project . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately The relevance of the objectives and Satisfactory design was substantial, but there were moderate shortcomings in achievement of the objectives, particularly for those on TB and malaria. According to the Harmonized Evaluation Criteria, this defines a moderately satisfactory outcome rating. Risk to Development Negligible to Low Negligible to Low Outcome : Bank Performance : Satisfactory Satisfactory Borrower Performance : Highly Satisfactory Satisfactory The project experienced some implementation delays, particularly on the part of line ministries. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Identifying critical baseline information can inform strategic directions of the project and lead to greater project success. Harmonization of relevant M&E indicators can greatly facilitate coordination of activities amongst partners, leading to greater achievement of overall outcomes . 14. Assessment Recommended? Yes No Why? Important lessons can be drawn out from this project, particularly in a region where the most effective approaches to the HIV/AIDS epidemic are yet to be fully understood . Also, the assessment could examine the synergies between the Bank and the Global Fund working in close partnership on this program . 15. Comments on Quality of ICR: The quality of the ICR is Satisfactory, with caveats . Although the ICR aims to present data related to the project's objectives and identifies factors that contributed to implementation effectiveness, a number of key indicators (those identified in the PAD as well as those mentioned above that should have been included ) were not reported and thus it could not fully substantiate claims of achieved goals . It would have been of great interest to learn more about the role played, if any, of the multisectoral approach, as well as the steps taken to build and sustain government commitment. To its credit, the ICR made an honest attempt to update the benefit -cost calculations of the PAD. However, the calculation was based on an outdated model of the epidemic, questionable assumptions on effectiveness and costs, and neglecting the costs of all donors . In the future, it might be more fruitful to approach the problem of efficiency with evidence of cost-effectiveness, which avoids some of these difficult assumptions . Given Djibouti’s small population (750,000) it would also be worth assessing whether the expenditure per capita implied by this project (about $4/person/year) in addition to even greater financing by the Global Fund and other donors simultaneously over this period was commensurate with the outcomes . a.Quality of ICR Rating : Satisfactory