ICRR 14269 Report Number : ICRR14269 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 12/09/2013 Country : Afghanistan Project ID : P101502 Appraisal Actual Project Name : Afghanistan Hiv/aids US$M ): Project Costs (US$M): 10.0 9.9 Prevention Project L/C Number : CH328 Loan/ US$M ): Loan /Credit (US$M): 10.0 9.9 Sector Board : Health, Nutrition and US$M): Cofinancing (US$M ): 0 0 Population Cofinanciers : Board Approval Date : 07/31/2007 Closing Date : 12/31/2010 11/30/2012 Sector (s): Health (70%); Central government administration (26%); Other social services (4%) Theme (s): HIV/AIDS (67% - P); Health system performance (33% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Judith Hahn Gaubatz Judyth L. Twigg Christopher D. Gerrard IEGPS2 2. Project Objectives and Components: a. Objectives: According to the Financing Agreement (page 4) and the Memorandum and Recommendation of the President (page 8), the objectives of the project were : To slow down the spread of HIV /AIDS; and To build up national capacity to respond to the epidemic . These objectives were to be achieved through (i) behavioral change in high risk groups; and (ii) improved knowledge of HIV/AIDS prevention mechanisms and reduced stigma related to HIV /AIDS among the general population. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: 1. Communications and Advocacy (Appraisal: US$1.1 million; Actual: US$1.1 million): This component aimed to increase knowledge of HIV prevention among opinion leaders and the general population, and to reduce stigma and discrimination against vulnerable groups at high risk of HIV and AIDS . Activities included: baseline and follow-up audience research; advocacy interventions among opinion leaders; and behavior change communication activities. 2. Strengthening HIV Surveillance (Appraisal: US$1.6 million; Actual: US$1.5 million): This component aimed to increase information about HIV/AIDS epidemiology in the country, particularly among high risk groups . Activities included: biological and behavioral surveys; knowledge, attitudes and practices surveys; high risk group mapping; and training on surveillance . 3. Targeted Interventions (Appraisal: US$4.1 million; Actual: US$5.7 million): This component aimed to implement prevention activities in four urban areas (Kabul, Mazar-i-Sharif, Hirat, and Jalalabad) among targeted high risk groups, among whom the HIV epidemic was concentrated . The high risk groups initially identified were injection drug users (IDUs), sex workers, truckers, and prisoners . Activities were to be implemented by NGOs and would also pilot innovative approaches such as drug substitution therapy . 4. Program Management (Appraisal: US$1.8 million; Actual: US$1.6 million): This component aimed to strengthen the capacity of the National AIDS Control Program (NACP) to carry out its core functions, including monitoring and evaluation. Activities included: training; technical assistance; multi -sectoral coordination efforts; and recruitment of national and international advisors . This component also included an Innovation Initiative, which aimed to mainstream multisectoral HIV responses by funding proposals for activities from line ministries other than the Ministry of Public Health . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project cost The actual project cost was US$ 9.92 million, out of the appraised US$10.0 million, due to exchange rate fluctuations. Financing The project was financed entirely by an IDA grant of US$ 10.0 million, of which US$9.92 million disbursed. Approximately US$128,000 was cancelled from the grant. Borrower contribution There was no planned borrower contribution . Dates June 2010: A project restructuring was undertaken to make the following changes to the project : (i) The Innovation Initiative activity in Component 4 was dropped due to a failed procurement and the low prospects for an acceptable bid. The funds were reallocated to targeted interventions and surveillance activities; (ii) The closing date was extended from December 2010 to June 2012, due to the delayed start-up and incomplete status of numerous project activities . June 2012: The project closing date was extended again from June 2012 to November 2012, to allow for completion of the second surveillance survey . 3. Relevance of Objectives & Design: a. Relevance of Objectives: Substantial . Although the HIV prevalence rate in the country remains below 0.5 percent in the general population and below 5 percent among high risk groups - most notably injection drug users (IDUs) - the increasing levels of injection drug use signal a high potential for rapid spread of HIV . Therefore, the government has sought to take concerted early action to increase awareness of HIV prevention methods and to improve national capacity to respond to a potential epidemic . The project objectives are consistent with the National Development Strategy and the National HIV /AIDS Strategy, both of which identify the maintenance of a low HIV prevalence rate as a key goal . The objectives are also relevant to the Bank's Interim Strategy Note for FY 12-14, which identifies improved service delivery as a key pillar, and the Bank's overall corporate priorities as reflected in the Millennium Development Goals. However, the measurability of the first objective, as it is articulated, was challenging given the nature of the HIV epidemic in the country . b. Relevance of Design: Substantial . The project design focused on both strengthening national capacity to monitor and manage the epidemic, particularly through surveillance, and implementing prevention activities among high risk groups, given the concentrated nature of the epidemic . The latter focus was critical to ensure that the main drivers of the epidemic - identified mainly as injection drug users (IDUs), but also potentially sex workers, prisoners, and truckers - were targeted for prevention efforts . Given the weak implementing capacity of the government, NGOs were to be contracted to deliver services to IDUs . However, efforts to allow the importation and use of the drug substitute for a critical pilot activity - opioid substitution therapy (OST) - were inadequate. As noted in the ICR (page 7), it was not clear from the project design "what process was followed to ascertain the critical point of how feasible it might be to bring about a more permissive policy environment in Afghanistan ." 4. Achievement of Objectives (Efficacy): To slow down the spread of HIV /AIDS Achievement of this objective is rated Modest , due to mixed achievement on indicators related to behavior change and knowledge. In addition, the level of coverage of injecting drug users (IDUs) in the project areas is unclear. Outputs Implementation of targeted interventions among high risk groups, including : Distribution of over 3.35 million safe injecting kits to IDUs; collection of over 2.13 million used needles and syringes. Conducting of 13,896 HIV tests through voluntary counseling and testing programs . Distribution of over one million condoms to IDUs . Implementation of an opioid substitution therapy (OST) program in one project site in Kabul . The program enrolled 71 drug users, out of a targeted 200. The ICR (page 33) reports that the following services were also provided, but quantity of services was not available: syndromic management of sexually -transmitted infections, overdose management, referral for tuberculosis services, referral for HIV treatment services, referral for drug detoxification, primary health checkups, and social services . The ICR (page 16) suggests that project coverage of urban IDUs was significant, as reflected by the indicator that 90.2% of IDUs had "used risk reduction services ." However, the ICR (page 17) reports that the project had reached 3,479 IDUs by project closing. With an estimated population of 12,541 IDUs in Kabul alone (ICR, page 18), the extent of project coverage of IDUs is unclear . Outcomes Among IDUs: The percentage of IDUs reporting use of sterile injection equipment at least once in the past three months was 92.9% in 2012. A comparison from 2009 could not be made due to the deemed inaccuracy of the 2009 data. According to the ICR, the percentage of IDUs reporting use of condom at last sexual encounter decreased from 27% in 2009 to 23.9% in 2012. However, the project team clarified that the final figure included an additional city. Final data for the original three cities showed that the percentage was maintained at 27% by 2012. The percentage of IDUs who correctly identify two ways of HIV prevention decreased from 71.4% in 2009 to 22.1% in 2012. Among other project-targeted groups (although these groups were later determined to be less critical given their zero HIV prevalence): According to the ICR, the percentage of sex workers reporting use of condom with their most recent client decreased from 58% in 2009 to 52.3% in 2012. However, the project team clarified that the baseline figure represented Kabul only, while the final figure represented Kabul, Mazar, and Herat . Final data from Kabul only showed an increase from 58% to 67.8%. According to the ICR, the percentage of sex workers who had used risk reduction services increased from 1.4% in 2009 to 8.3% in 2012, falling short of the target of 10%. However, as noted previously, the final figure represented three cities, rather than just Kabul . Final data provided by the project team showed an higher increase in Kabul only, from 1.4% to 11.7%. The percentage of sex workers who correctly identify two ways of HIV prevention increased from 12.8% in 2009 to 15.3% in 2012, falling short of the target of 20%. The percentage of prisoners who correctly identify two ways of HIV prevention decreased from 25.4% in 2009 to 6.6% in 2012. Although the ICR also reports estimated HIV prevalence rates among the targeted high risk groups, this indicator should be interpreted with caution given the lack of clarity in project coverage (and therefore attributable project impact): Among IDUs (in Kabul, Herat, Jalalabad, and Mazar -i-Sharif), the prevalence rate decreased from 7.1% in 2009 to 4.4% in 2012. Among sex workers (in Kabul and Herat), the prevalence rate was 0.3% in 2012; among men having sex with men: 0.4%; and among prisoners: 0.7%. To build up national capacity to respond to the epidemic This objective is rated Substantial due to effective implementation of numerous activities and evidence of expanded national capacity to monitor the epidemic . Outputs Conducting of target audience research and surveys . Development of advocacy strategy for opinion leaders . Development of core training and information packages for target audiences . Provision of communication training for NGOs, journalists, private practitioners, and others . Mapping of high risk groups in urban areas . Conducting of two rounds of Integrated Biological and Behavioral Surveillance surveys (IBBS) among IDUs, sex workers, truckers, and prisoners . Epidemiological modeling of HIV/AIDS impact. Development of new HIV testing and counseling guidelines . Assessment of opioid substitution therapy pilot . Outcomes The government issued a policy statement protecting the rights of people living with HIV /AIDS. 6 urban centers conducted risk mapping of high risk groups, exceeding the target of 4. All 7 sites for IDUs, and the 1 site for sex workers were meeting service quality standards . The IBBS was expanded from 4 to 6 sites (including southern and eastern parts of the country ) and now includes a newly identified high risk group of men having sex with men . Surveillance and monitoring data were incorporated in annual National AIDS Control Program (NACP) reports and action plans. Implementing entities reported uninterrupted funds flow and service delivery . 5. Efficiency: Efficiency is rated Substantial . The project design clearly targeted prevention activities among injecting drug users (IDUs), which evidence had established as the main drivers of the epidemic in the country . Although the costs these interventions may have been higher than interventions targeted to other high risk groups (i.e. female sex workers or truckers ) due to the high degree of marginalization of IDUs, the alternative approach of implementing activities across the general population would likely have been less efficient . The costs of the opioid substitution therapy activity in particular were high; however, the pilot nature of the activity ultimately contributed to cost efficiency as the decision was made to discontinue the program due to poor outcomes . There were significant delays in conducting the Integrated Biological and Behavioral Surveys, the results of which were expected to sharpen the project focus, but this did not occur until more than two years after project effectiveness. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Relevance of the project objectives and the project design are rated Substantial . Achievement of the objective to build national capacity to respond to the HIV /AIDS epidemic is rated Substantial due to evidence of improved capacity to monitor the epidemic. However, achievement of the objective to slow down the spread of HIV is rated Modest, as it is not clear whether the project was not able to reach a sufficient proportion of IDUs in the targeted urban areas to affect the spread of HIV . Efficiency is rated Substantial . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The targeted interventions to high risk groups are being continued in the follow up Bank project, as part of a broader health sector project (System Enhancement for Health Action in Transition, US$ 100.0 million, approved June 2013). The institutional arrangements - namely contracting out services to NGOs - are also being continued in the next project . However, it is unclear whether advocacy activities, which are critical to creating a supporting political and social environment for outreach to stigmatized populations, will be sustained . The continued fragile security environment will also pose significant risks to prevention outcomes . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: The project built upon the Bank's previous analytic work in social mapping of high risk groups, with the project design substantially informed by evidence on the dynamics of the epidemic in the country . Risks were clearly identified - including the low capacity of the government, the fragile security context, and the difficulties in implementing project activities among traditionally stigmatized populations . As mitigation measures, the project limited its geographic focus on four urban areas (Kabul, Jalalabad, Mazar-a-Sharif, and Herat) where the security situation was relatively more stable; however, the lack of security was cited (ICR, page 10) as a major factor in the difficulties in recruiting national and international consultants . Regarding the controversial nature of some of the proposed project interventions, mitigation measures were not effective in ensuring a more supportive political and social environment . The project design was also unclear in determining how sufficient coverage of the key high risk group - injecting drug users - would be achieved, given the limited number of NGOs . at -Entry Rating : Quality -at- Moderately Satisfactory b. Quality of supervision: The Bank team provided intensive supervision and used the opportunity provided by the Mid -Term Review in February 2010 to refine the project design and the results framework . Based on data gathered from the first surveillance survey, the focus on truckers was de -emphasized, while men having sex with men (MSM) were added as a key target group . The second round of the surveillance survey was postponed by one year in order to expand its geographic scope and include MSM . Some indicators provided little added value in understanding the specific dynamics in the country (i.e. knowledge in the general population or condom use among truckers), and therefore new indicators were added . Although some general policies were established in support of HIV/AIDS programming in the country, the Bank's policy dialogue was not able to bring about more critical changes in political attitudes towards high risk groups . There were no major fiduciary problems reported. Initial problems in safeguards compliance were addressed through increased supervision and training. Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: Despite the challenging socio -cultural environment in which the key target populations are highly marginalized, the government was able to provide a generally supportive policy environment for addressing HIV. As noted by the project team, the government, faced with the serious development challenge of a potential AIDS epidemic, made a "bold and progressive" decision to implement a program among sub-groups that are stigmatized and controversial . Institutional arrangements, namely contracting out to NGOs, were established, and a statement on confidentiality and voluntary testing was issued . However, obstacles to introducing drug substitution therapy in the country remained in place throughout the project period. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: The primary implementing agency was the National AIDS Control Program (NACP) within the Ministry of Public Health. There was weak capacity in the NACP, leading to problems such as high number of vacancies in key staff positions, procurement delays, and delayed disbursements . However, these issues were addressed throughout the project period, and there were no major fiduciary or safeguards problems reported overall. The ICR (page 27) also reports that there were conflicting expectations between the NACP and the contracted surveillance firm. The surveillance activities were "envisaged to be owned by and readily available to the NACP, routine and not subject to research review processes ." However, the contractor "clearly felt that their priority in the exercise was the highest possible quality epidemiological research, safeguarded by thorough independent review ." These conflicting positions led to significant delays in the surveillance activities. Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The M&E design placed strong emphasis on generating surveillance data, particularly for the targeted high risk groups. The project indicators appropriately focused on behavior change among the high risk groups, to measure the impact on HIV prevalence . The surveillance activities were contracted out to a high quality third party, given the low capacity of the government . b. M&E Implementation: Both rounds of the Integrated Biological and Behavioral Survey were conducted, with results from the first round leading to adjustments in the project design and indicators . The second round of the survey was postponed by one year to expand its geographic scope and to ensure inclusion of men having sex with men (MSM) in the survey questions. However, the elaboration of the survey by the external party took considerably longer than planned and experienced procedural delays, for both the first and second rounds . The new indicators were to be assessed through two rounds of health facility assessments; however, results from the second round in March 2013 were not available at the time of ICR preparation . c. M&E Utilization: As noted above, findings from the first round of the surveillance survey led to the de -emphasis on truckers and new inclusion of MSM as a target group, and to refinements to the results framework . M&E Quality Rating : Substantial 11. Other Issues a. Safeguards: The project was classified as a category "B" project due to medical waste issues, triggering the safeguard policy on Environmental Assessment (OP 4.01). A medical waste management plan was developed with the help of the Bank team. The ICR (page 12) reports that although there were initial issues such as the disposal of syringes in areas accessible by drug users and others, these were addressed by increased supervision and training of NGOs. Overall safeguard compliance was satisfactory . b. Fiduciary Compliance: Financial management: A number of financial management risks were identified during project appraisal, and therefore fiduciary performance was rigorously monitored . There were some minor issues such as delays in undertaking audits of NGOs and internal project activities, and some disbursement delays in payments to NGO contractors. Results of audit reports are not reported, although the ICR (page 11) reports that the above issues were resolved and no major financial management problems were reported . The project team subsequently reported that only one financial audit had a disclaimer, although the disclaimer related to changing estimates on account of the XDR exchange rate fluctuations, not project expenditures . Procurement: There were no major procurement problems reported, aside from initial delays in processing contracting firms and individual consultants due to low capacity of the government . c. Unintended Impacts (positive or negative): None reported. d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Significant Significant Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: Lessons drawn from the ICR (pages 27-28), adapted by IEG: Technically strong and evidence -based project interventions may have limited influence on program directions without a supportive political and social environment . In the case of this project, the controversial nature of the IDU interventions was not effectively mitigated by advocacy among opinion leaders or the general population . The importance of timely surveillance data needs to be balanced with ensuring quality and reliable results. In the case of this project, the conflicting expectations of timely data to inform project design and policy vs. high quality data to inform research led to implementation delays . These expectations should be clarified during project preparation . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The quality of the ICR is overall satisfactory, with the analysis focused on data and evidence . The ICR is candid in the discussion of project context and shortcomings, which was informative for drawing lessons for Bank projects in a similar setting. Regarding financial management, the ICR does not report whether project audits were timely and/or qualified, although the information was later provided by the project team . a.Quality of ICR Rating : Satisfactory