ICRR 13083 Report Number : ICRR13083 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 02/04/2011 PROJ ID : P074730 Appraisal Actual Project Name : National Hiv/aids US$M ): Project Costs (US$M): 20.87 13.00 Prevention Project Country : Sri Lanka Loan /Credit (US$M): Loan/ US$M ): 12.55 9.24 Sector Board : HE Cofinancing (US$M ): US$M): 6.36 1.9 Sector (s): Health (50%) Central government administration (25%) Other industry (15%) Media (10%) Theme (s): HIV/AIDS (23% - P) Other public sector governance (22% - P) Vulnerability assessment and monitoring (22% - P) Other communicable diseases (22% - P) Gender (11% - S) L/C Number : CH013 Board Approval Date : 05/30/2002 Partners involved : Global Fund to Fight Closing Date : 06/30/2008 06/30/2008 AIDS, Tuberculosis and Malaria Evaluator : Panel Reviewer : Group Manager : Group : Judyth L. Twigg George T. K. Pitman IEG ICR Review 1 IEGPS1 2. Project Objectives and Components: a. Objectives: The Development Grant Agreement (DGA) lists the project’s development objectives as (i) curbing the spread of HIV/AIDS infection and (b) reducing the risk of an emerging HIV -associated tuberculosis epidemic . The Project Appraisal Document (PAD) expand on the first objective, stating that the objective is to “limit the spread of HIV infection in Sri Lanka among its highly vulnerable subpopulations and the population at large, without stigmatizing those who are engaging in high risk behaviors .� This Review will use the project development objectives as stated in the DGA, as the expanded elements of the objectives as stated in the PAD are implied . A third objective was proposed, “to prolong the survival and improve the quality of life for people living with HIV /AIDS (PLWHA) through the establishment of a care and treatment program, � but according to the main text of the ICR, the project was never formally restructured and the objectives were never formally revised . The ICR gives two contradictory dates for the proposal of this third objective : May 2004 (p. 32), and April 2008 (p. 12), and the Summary of the Borrower’s ICR states simply that this third objective was added . Since this objective was never officially added to the project, it will not be assessed here . The ICR also cites the addition of a sub -component for a care and support program for PLWHA, with IDA informing the Ministry of Finance of the inclusion of this sub -component by letter dated September 14, 2004. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? No c. Components (or Key Conditions in the case of DPLs, as appropriate): 1. HIV Prevention Interventions (Estimated USD 6.64 million, or 31.8% of total project costs, Actual USD 7.10 million, or 54.6% of total project costs), which included two subcomponents : (a) interventions targeted to vulnerable populations, including male and female sex workers, workers in free trade zones, migrant workers, transport workers, drug users, men having sex with men, prisoners, and armed services personnel; and (b) broad-based programs for youth and general populations; 2. Tuberculosis Control (Estimated USD 8.60 million, or 41.2% of total project costs; Actual USD 1.60 million, or 12.3% of total project costs), which included two subcomponents : (a) enhancing leadership, organization, and functioning of the National TB Program; and (b) improving coverage and quality of directly -observed treatment, short-course (DOTS) and research for adapting and enhancing DOTS . 3. Institutional Strengthening (Estimated USD 5.63 million, or 27.0% of total project costs; Actual USD 4.3 million, or 33.1% of total project costs), which included three subcomponents : (a) strengthening the capacity of the agencies involved in the national response, including the Ministry of Health and Nutrition (MOHN), Director General of Health Services (DGHS), the National STD/AIDS Control Program (NSACP), provincial authorities, multisectoral agencies, and NGOs and other private/voluntary organizations; (b) improving the information base for policy decisions and program management; and (c) improving health care waste management . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Because of changes in the dollar /SDR exchange rate, the initial value of the Grant expressed in US dollars changed from USD 12.55 million at appraisal to USD 14.33 million at closing. The total amount disbursed, SDR 6.19 million (USD 9.24 million), therefore represents 73.6% of the original estimated Grant, and 64.5% of the final total Grant amount. Total cancellations amounted to SDR 3.41 million (USD $5.09 million according to the exchange rate at project closing). The Region added information that the cancellation was because the project could not complete all planned activities by the completion date, and an extension was not granted in light of implementation difficulties . The Borrower’s actual contribution was very close to that estimated at appraisal . 3. Relevance of Objectives & Design: Relevance of Objectives and Design is Substantial, based on substantial relevance of objectives and modest relevance of design. Relevance of Objectives : Substantial . At the time of appraisal, Sri Lanka was experiencing a low level HIV /AIDS epidemic (adult prevalence rate of 0.07%), but it was observed that behavioral factors that might facilitate the spread of infection were prevalent in the country . Neither the PAD nor the ICR offer statistics on HIV in high -risk groups at baseline, or behavioral data at baseline, that might give additional insight into these behavioral factors . Nevertheless, it was thought that acting early to prevent a larger -scale epidemic would cost a fraction of the resources that would otherwise be needed if HIV were to spread . Similarly, the incidence of TB was growing, albeit slowly, in the late 1990s, and the country’s vulnerability to an HIV epidemic further heightened concerns over a resurgence of TB, although there was no evidence at the time of appraisal that HIV -associated TB had become a problem. The rationale for the project’s development objectives was therefore a preemptive one . It is unclear whether the Bank or the Government have conducted analysis of the counterfactual, to determine whether the behavioral factors specific to Sri Lanka were likely to produce an HIV and /or TB epidemic similar to that experienced in other countries in the absence of other interventions . The Government of Sri Lanka is committed to Millennium Development Goals that include halting and reversing the spread of HIV . The Bank’s current Country Assistance Strategy (CAS) for Sri Lanka (2009-2012) cites as a priority keeping HIV prevalence low by increasing awareness . Relevance of Design : Modest . Given low prevalence in the country, the project made appropriate design choices to focus on high-risk groups, incorporate significant advocacy and awareness -building activities, shift from medical focus to a public health focus, engage non -health sector ministries (armed forces, labor, prisons), involve NGOs, and build on existing institutional arrangements . The project built effectively on existing strengths in blood safety and TB control in designing the HIV/TB co-infection strategy. However, baseline data for key objectives were vaguely specified or approximated. Sufficient political economy analysis was not conducted to recognize the lack of political will to harness/build NGO capacity and to intervene with high -risk groups. While a phased approach for NGO contracting and service delivery was mentioned in the PAD, effective plans for that phased approach were not put in place. Advanced procurement actions were not planned and completed in order to avoid what the ICR calls foreseeable later delays. There was no coherent communication strategy or detailed plan for a monitoring system . 4. Achievement of Objectives (Efficacy): Modest, based on modest achievement of curbing the spread of HIV /AIDS infection, and substantial achievement of reducing the risk of an HIV-related TB epidemic. The overall Modest efficacy rating is due to the significantly greater amount of project resources expended on HIV /AIDS, and substantial additional TB control resources contributed by the Global Fund to Fight AIDS, Tuberculosis, and Malaria during the project period, making it difficult to attribute achieved results on TB wholly to Bank -financed interventions. Achievement of curbing the spread of HIV /AIDS infection : Modest Outputs : Small NGO sub-projects were funded in two of three planned rounds : 47 sub-projects reached an unspecified number of persons in the first round (2004/2005), and 24 sub-projects reached 368,668 persons (5,000 of whom were members of risk groups) in the second round (2006). The Team Leader explains that some populations identified by the project as risk groups, including factory workers and drug users who were not injection drug users (IDUs), were not actually at high risk for contracting and spreading HIV, and two NGO -implemented subprojects targeting sex workers were the only ones targeting groups actually at risk . NGO-implemented subprojects targeting at-risk populations experienced low coverage and lack of significant scale -up, due to capacity constraints and procurement delays, and the NGO -based intervention design was not evidence -based. Fourteen of these sub-projects targeted students or out -of-school youth, reaching about 18,000 persons, but an External Review conducted by the NSACP in 2006 found that these sub-projects did not reach most-at-risk adolescents. The project also funded four large NGO contracts for interventions among risk groups; all four were signed in March 2007 and later, and therefore were implemented over too short a period to have an impact . The ICR cites the following as effectively implemented parts of the project : advocacy; youth-targeted awareness-building; non-health-sector interventions targeted at vulnerable populations such as the armed forces, prisoners, and migrant labor; STI management; and blood safety . Over 250 national political leaders, 850 regional political leaders, and 700 community-based organization leaders received awareness -raising training, including some study tours to Thailand . From 2005-2008, 465 prison welfare officers were trained, reaching 30,000 prisoners. From 2004-2008, 4,000 managers in the Ministry of Labor were trained, reaching 50,000 unspecified persons, and 135 training instructors in the Sri Lanka Foreign Employment Bureau (SLFEB) were trained, reaching 45,000 unspecified persons annually . Training also took place among an unspecified number of personnel in the armed forces and fisheries ministry . In 2005, Young Asia TV was contracted to implement a condom social marketing campaign, but they failed to do so . From 2002 through 2007, the NSACP procured over 5 million condoms with project support, for distribution through the armed forces (39%), STI clinics (37%), and NGOs (24%); annual condom distribution through the NSACP did not increase during the project period. In total, around 2.55 million individuals received messages about appropriate condom use . The Summary of the Borrower’s ICR asserts that because at -risk populations are part of the general population, it is plausible to assume that interventions aimed at the general population are reaching at -risk groups, questioning the need for targeted interventions . The project supported improvements in the physical infrastructure of blood banks, screening transfusions for HIV, Hepatitis C, and other STIs, and improving quality assurance. A massive blood safety and donor awareness campaign was implemented . Protocols were developed and training of an unspecified number of medical staff took place on preventing mother -to-child transmission (PMTCT). Outcomes : HIV prevalence was assessed at less than 1% at baseline, and measured at 0.03% at the end of 2007. HIV prevalence among sex workers was assessed at less than 1% at baseline, and measured at 0.4% at the end of 2007. Given the approximated baseline values and the large confidence intervals around these small numbers, it is unclear whether there has been an increase, decrease, or no change . The ICR indicates that there was low coverage of NGO -implemented interventions for sex workers, making attribution of any outcomes to the Bank -sponsored project difficult. The ICR reports that condom use increased from less than 10% (estimated) in 2002 to 62.9% in brothels, 70.4% in massage parlors, 81.9% among street-based sex workers, and 62.5% in karaoke parlors, in 2007; however, the 2002 data on condom use (not specific to any group) do not appear to be comparable to the 2007 data. The ICR also provides data on condom use among other important risk groups in 2007, and on awareness and attitudes among the general population, but it acknowledges that there are no available comparable baseline data . Voluntary blood donation increased from 86% in Colombo District and 27% island-wide in 2003, to 95% in Colombo District and 81% island-wide in 2008, with 100% of blood screened. Overall, there were implementation difficulties and lack of significant scale -up of targeted interventions, and the outcomes are unclear, given the questionable quality of much of the data presented . The ICR does not cite analysis about whether the observed results might have been achieved in the absence of any interventions at all, other than to speculate that the level of risky behaviors means that low HIV prevalence could not have been maintained purely due to extremely low injecting drug use, very limited sexual networking, reduction in tourism due to internal conflict, high literacy rates, relatively high status of women, and good access to health care . Achievement of reducing the risk of an HIV -related TB epidemic : Substantial . Outputs : Twenty chest clinics, nine TB wards, and the Central TB Laboratory were refurbished . Seventeen vehicles were provided for the use of chest clinics . HIV/TB co-infection treatment guidelines have been prepared but are not yet finalized . DOTS training materials were prepared, and an unspecified number of medical officers and primary health care personnel were trained. The Rapid Results Initiative (RRI) approach for TB was implemented in the two districts with the highest default rates in the country; it is unclear whether this intervention was sustained after the first round, and whether these interventions were implemented nation -wide. Substantially less was spent than planned for the TB control component (USD 1.60 million, of a planned USD 8.60 million). Outcomes : Incidence of TB declined from 45.6/100,000 in 2003 to 41.9 in 2007. From 2003 through 2007, detection rates of new sputum positive cases rose from 80% to 90%, treatment success of sputum positive cases rose from 80.4% to 88.0%, and default rates have declined from 10.6% to 6.7%. DOTS coverage has increased from 74% in 2003 to 97.5% in 2008. Multi-drug resistant cases of TB decreased from 17% of all cases in 2003 to 8% in 2007. In 2005, 0.07% of TB patients were HIV positive (no baseline figure is given). A 2007 UNAIDS/WHO External Review stated that “HIV/TB co-infection is currently not a significant health issue in Sri Lanka .� According to the project team leader, during the project period, the Bank was the main donor on TB, with the Global Fund contributing co-financing late in the project. However, during the lifetime of the project, the Global Fund also spent USD 5.00 million through a Round 1 grant (2003-2009) for TB control activities, compared with USD 1.60 million spent by the Bank. 5. Efficiency (not applicable to DPLs): Efficiency is Modest . A detailed economic analysis was undertaken during project preparation . The Net Present Value (NPV) of the program was estimated to be positive (USD 87 million) in the base case, using a 10% discount rate. The Net Fiscal Impact of the program as a purely preventive program was estimated to be positive in the base case (USD 5 million over the life of the program). The ICR suggests that Bank resources, including the team member based in-country, could have been used more effectively and inclusively . Interventions were not successfully directed toward reducing HIV transmission among high -risk groups and those with higher rates of partner exchange, which has been documented by UNAIDS as the most efficient means of reducing epidemics in their early stages. Also, while the PAD stressed the cost -effectiveness of emphasizing prevention over treatment in situations of low HIV prevalence, later in the project significant resources were added for treatment and prevention of mother-to-child transmission, even though prevalence remained quite low . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Based on Substantial relevance, Modest achievement of objectives (efficacy), and Modest efficiency, the project's outcome is rated Moderately Unsatisfactory . The project modestly achieved its main development objective, curbing the spread of HIV/AIDS, and the ICR offers very little baseline data to assess the specific contributions of the project . Interventions were not efficiently directed toward prevention among high -risk groups and those with high rates of partner exchange, and later in the project, there was a stress on treatment and prevention of mother -to-child transmission over prevention interventions proven to be more efficient in a low -prevalence environment. a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The Government has expressed commitment to continuing HIV control activities under the NSACP, with routine budget allocations scheduled to continue and a National Policy on HIV /AIDS approved. An application is also being th prepared for the GFATM 9 Round to further support prevention activities . There is no plan for a follow-on Bank project, although the Bank has agreed to support a social mapping exercise to build an evidence base for priority setting (to be financed from the Regional Funds ); an integrated bio-behavioral surveillance system (to be financed under the Health Sector Development Project ); further development of the national strategic action plan (to be financed by UNAIDS); and a global knowledge event on stigma reduction (from the projects supported by the Development Marketplace across the South Asia Region ). Prevention activities have been internalized in several sectors, including the policing and prison systems . The infrastructure and technical capacity of the national blood and TB control systems have been strengthened . However, low NGO capacity, Government-NGO relations, and scaling up interventions among risk groups are among the remaining challenges . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: at -Entry : Moderately Unsatisfactory . Project preparation included an important social Ensuring Quality -at- assessment to map vulnerable subpopulations and NGO activities, but there was no other analytic work on HIV and behaviors. A detailed supervision plan was put in place . From the beginning, it was clear that there were significant differences in the Bank ’s and the Government’s approach to prevention activities, with the Bank encouraging a greater focus on interventions among high -risk groups. Weak NGO capacity and the lack of effective systems to guide project implementation and quality assurance were identified during project preparation. However, the project’s design did not include appropriate measures to address these risks/problems. Quality of Supervision : Unsatisfactory . The ICR and the Summary of the Borrower ’s ICR note that there were frequent delays in getting “no objection� from the Bank for project activities, specific TORs, procurements, and personnel appointments, causing frequent delays . The Borrower also claims that there were contradictory messages sent by different Bank personnel throughout 2007-2008, again causing delays in the last year of the project. A new Project Implementation Plan (PIP), to be put in place after the MTR (March 2006), emerged in different versions without clarity on their approval status, causing confusion . A third development objective was introduced and acted upon, but the project objectives were never formally revised . Some aspects of the relationship between the Bank and the Government were clearly difficult throughout the project, and although the RRIs significantly enhanced the pace of implementation, it is not evident that complete and appropriate steps were taken to resolve differences . The project was in “problem� status for over two years, not satisfying the Bank ’ s norm for pro-activity within one year. at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Unsatisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: Government Performance : Moderately Unsatisfactory . The Government, at the highest levels, demonstrated sustained political will to implement advocacy and awareness -raising interventions aimed at the general public . Non-health-sector agencies were effectively mobilized . Several important policy initiatives, such as the National Policy on HIV/AIDS, were developed. However, the Government was also responsible for the lack of progress in resolving disagreements with the Bank, particularly with regard to the relative emphasis on prevention activities aimed at the general public as opposed to high -risk groups, and to use of NGO services to scale up prevention interventions for high-risk groups. Implementing Agency Performance : Unsatisfactory . In order to boost implementation progress, responsibility for implementing project activities had to be shifted from the office of the Director General for Health Services (DGHS) to a specially established Project Management Unit (PMU) in December of 2004. Problems with salary differentials, unclear lines of authority, staff shortages and turnover, frequent leadership changes, and poor financial and contract management remained, however . There were significant problems and delays in procurement and contracts, with deficiencies in documentation and record -keeping. Many project staff were not familiar with the PAD or the PIP. a. Government Performance :Moderately Unsatisfactory b. Implementing Agency Performance :Unsatisfactory c. Overall Borrower Performance :Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: M&E Design : Modest . The project design contains a set of clear indicators, and a logical results chain connecting inputs, outputs, and outcomes to these indicators . Baseline values for the project ’s key indicators, however, are expressed only in rough approximation . Several indicators that would have been useful for monitoring project outcomes were not included in the M&E framework at all, including HIV -related knowledge among risk groups, condom use in risk groups other than sex workers, and other knowledge - and behavior-related indicators. HIV prevalence is not a reliable indicator in a low -grade epidemic; other indicators are necessary to signal whether behavior has changed . The PAD contains provisions for the development of a Management Information System (MIS), a second-generation surveillance system, and operational research to fill knowledge gaps and guide interventions . Detailed plans for implementation of these provisions, however, were absent. M&E Implementation : Negligible . The deployment of the project’s M&E system was delayed due to difficulties in finding agreement on the monitoring framework and delays in procuring hardware . The system was put in place at the end of the project period . Sentinel sero-surveillance for HIV and syphilis has been conducted annually since 1993, covering important population groups (although not a representative sample of some important risk groups). The first round of a behavioral surveillance survey (BSS) was completed, after significant delays, in August 2007. The Summary of the Borrower’s ICR states that the NGO subprojects targeting students and out-of-school youth were evaluated, but the main ICR states that these were not evaluated and their impact was never measured. Overall, because the M&E system was not installed until the end of the project period, there was no mechanism to ensure development of effective prevention tools and strategies, and no way to make informed decisions about adjusting these tools and strategies during implementation . M&E Utilizatio n: Negligible . Use of data as a guide for strategic planning was poor . The first round of NGO interventions was not evaluated to learn lessons before launching the second round . Data from the BSS have not been used to inform interventions among risk groups . a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards : Considerable institutional strengthening took place for health care waste management, including the development of national policies and guidelines and training manuals . Fiduciary : Prompted by the failure to send updated /annual procurement plans to the Bank for clearance, a post-review was conducted in 2006. It found that the irregularities did not merit declaration of mis -procurement, but that there was a pattern of poor planning and management of procurement . Positive impact : The ICR states that the awareness -building activities of the project were very successful in reducing stigma among many key groups: government officials, the media, and the public, although it does not provide concrete evidence to support this claim . The project can claim some credit for creating commitment to addressing HIV/AIDS. A national PMTCT working group was formed . The project also provided CD4 (type of white blood cell) count monitoring equipment and reagents and drugs to treat opportunistic infections . The percentage of HIV patients receiving antiretroviral therapy (ART) (of those who need it) was 30%, with 150 patients on ART, in 2008, after a Rapid Results Initiative was begun in December 2004; the ICR states that the denominator of this target (the number of patients requiring ART) needs to be confirmed. 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately Based on substantial relevance, Unsatisfactory modest efficacy and modest efficiency . In failing to scale up HIV prevention interventions targeted at high -risk groups, the project did not use resources in a manner that was cost-effective in a low-prevalence environment. Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Unsatisfactory Unsatisfactory Borrower Performance : Moderately Moderately Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The capacity of NGOs to reach high -risk groups is consistently over -estimated . While the value of NGO participation is universally recognized, it is important to place NGO activities within the national context . Contracting of NGO services was a weak link in this project . If NGOs are to play an important role in implementation, there must be thorough initial assessment of NGO capacity , along with focused efforts at capacity-building if required. Governments are generally more likely to focus on prevention interventions in the general population rather than in high -risk and marginalized populations, and they are generally more likely to favor treatment than prevention interventions . Strong incentives and continuous Bank engagement are necessary to ensure appropriate design and implementation choices, particularly in a nascent epidemic like that in Sri Lanka . When disagreements between the Government and the Bank, or among Bank personnel, visibly impact project performance, Bank management must be attentive to the need to resolve differences and reach consensus. The failure to implement monitoring and evaluation severely limits the ability to establish plausible attribution of outcomes to government programs . There is no substitute for country -specific data. The lack of baseline data in this case led to possibly inappropriate forecasts about the future trajectory of the epidemic and overprogramming of resources . The perceived risks of a rapidly expanding epidemic must be very carefully assessed, rather than assumed . As the Summary of the Borrower’s ICR states, lessons learned from the USA, Western Europe, and Sub-Saharan Africa may not have been applicable to Sri Lanka . The lack of absorption of about 40% of allocated funds, combined with the maintenance of low prevalence in spite of poor project performance in many areas, raises questions about whether the project was overfunded . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR effectively and fairly highlights the project ’s achievements, attempts (on the basis of available evidence ) to determine the extent to which observed outcomes can be attributed to the project ’s interventions, and explains the difficulties encountered. The ICR gives two contradictory dates for the proposal of the third objective for care and treatment of PLWHA. It does not explain the reasons for the cancelled disbursements, nor does it explain why the Global Fund’s contributions were smaller than anticipated, nor why more was spent than anticipated on Components 1 and 3, and less than anticipated on Component 2. a.Quality of ICR Rating : Satisfactory