ICRR 12750 Report Number : ICRR12750 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 09/26/2007 PROJ ID : P060329 Appraisal Actual Project Name : HIV/AIDS Rapid US$M ): Project Costs (US$M): 16.2 18.55 Response Project Country : Gambia Loan/ US$M ): Loan /Credit (US$M): 15.0 16.69 Sector Board : HE US$M): Cofinancing (US$M ): Sector (s): Health (73%) Other social services (18%) Central government administration (7%) Sub-national government administration (2%) Theme (s): HIV/AIDS (50% - P) Decentralization (25% - S) Participation and civic engagement (25% - S) L/C Number : C3455 Board Approval Date : 01/16/2001 Partners involved : Closing Date : 12/31/2005 12/31/2006 Evaluator : Panel Reviewer : Group Manager : Group : Judith Hahn Gaubatz Kris Hallberg Alain A. Barbu IEGSG 2. Project Objectives and Components: a. Objectives: According to the Development Credit Agreement (DCA), the project objective was to assist the Borrower to organize a pre-emptive response to the HIV/AIDS epidemic through a multisectoral approach by : (i) maintaining the current low levels of the epidemic; (ii) reducing the spread and mitigating its effects; and (iii) increasing access to prevention services as well as care and support for those infected and affected by HIV /AIDS. The ICR notes that this original objective was problematic in that it emphasized a process objective rather than outcomes; thus after the Mid -Term Review, the project team modified the project objective as follows : to assist the Borrower to (i) reduce the spread of the epidemic; and (ii) to mitigate its effects. This revision was adopted by the project team but was not formally changed in the DCA. While only objectives articulated in the DCA are normally used for project evaluation, the revised objectives for this project are very similar in meaning and intent to the original and thus are used for this review. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): US$ 3.5 million; actual : US$5 (1) Capacity building and policy development (appraisal : US$3 US$ 5.55 million ), with support to (i) the national, divisional and community -level HIV/AIDS entities and (ii) the development of a national HIV /AIDS strategy and action plan; Multi -sectoral responses for HIV /AIDS prevention and care (appraisal : US$2 (2) Multi- US$ 2.65 million; actual US$ 0.57 million ), with (i) provision of services to line department staff and their dependents and (ii) capacity building for staff to provide services to their constituents; (3) Health sector responses to STI /HIV management (appraisal : US$1 US$ 1.85 million; actual US$ 2.3 million ), with (i) implementation of HIV/AIDS services and (ii) expansion of efforts on reproductive health and opportunistic infection control; and (4) Community and civil society initiatives (CCSI) ( appraisal : US$8 CCSI ) (appraisal 10 .13 million ) with grant US$ 8.2 million; actual US$ 10. resources provided to community and civil societies for HIV /AIDS activities. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: An Accelerated Results Initiative took place in early 2004 to intensify implementation efforts . A Quality Enhancement Review took place in December 2004 to address implementation issues and sharpen focus of project activities. The Borrower received a US$14.6 million grant from the Global Fund in late 2004 that was primarily used for treatment and care services – Phase 1 of the grant, amounting to US$6.2 million, was completed in September 2006. The DCA was amended in April 2005 to modify and add intermediate indicators; no changes were made to the key PDO indicators. The project closing date was extended for one year to allow for further implementation . Only a quarter of the appraised amount for Component 2 for multisectoral responses was disbursed as few line departments carried out activities as planned . Component 3 for health sector responses disbursed more than planned due to an expanded scope of activities per the amended DCA . The government provided 70% of the planned counterpart funding, a shortfall which the ICR states can be attributed to a very low exchange rate and an overestimated initial budget. 3. Relevance of Objectives & Design: Overall relevance of the project is rated substantial, with caveats. The relevance of the project objectives is rated substantial . HIV/AIDS is noted as a priority area in the Gambia ’s Poverty Reduction Strategy Paper (PRSP): two of the five PRSP pillars include goals to “stem the rapid growth of HIV� and to “mainstream HIV/AIDS awareness into all development programs � for the Gambian government. The Country Assistance Strategy (CAS) refers to these PRSP pillars as the strategic framework for the Bank ’s activities in the Gambia but has very limited discussion of the HIV /AIDS situation in the country. The relevance of the project design is rated substantial , but with caveats. The focus on building capacity, ensuring a multi-sectoral response and empowering community and civil -society initiatives was relevant to building a nation-wide response and consistent with the Bank's Multi -Country HIV/AIDS Program (MAP) design. In addition, designating a separate component for the health sector was an effective mechanism to ensure priority attention to health sector interventions . Also, given the concentrated state of the epidemic, the project design was relevant in highlighting the need to focus on high -risk groups such as commercial sex workers, truck drivers and uniformed personnel; however, more explicit implementation mechanisms could have been used to ensure that this focus was actually carried out. However, the original project objectives were not well articulated, emphasizing process outputs rather than outcomes. The project indicators were not appropriate for effectively assessing the project ’s impact (regarding the indicator on exposure to HIV/AIDS messages, the target was 90% exposure but the awareness level was already above 90% at the start of the project; regarding the indicator on HIV prevalence, this is not the appropriate measure,due to the influence of mortality rates, as the ICR notes ). While most risks identified in the PAD did not significantly materialize nor impact project implementation, two critical risks were not identified in the design : lack of commitment and lack of effectiveness of the national -level government and AIDS-coordinating bodies. 4. Achievement of Objectives (Efficacy): Overall achievement of objectives is rated negligible . This is based on negligible achievement of objective (i) and modest achievement of objective (ii), given the greater emphasis on prevention in the project design . 1. Reducing the spread of the HIV /AIDS epidemic, is rated negligible . One of the key performance indicators for this objective was “containment of HIV prevalence levels �. As noted in the ICR, the prevalence rate is not the appropriate measure. The ICR offers data on prevalence rates among pregnant women (15-19 years old) as a proxy: the rate was 1.1% in 2004 and decreased to 0.3% in 2005 – however, the time frame for these figures is too limited to establish the decrease as a long -term trend and the sample size for this group (15-19 year olds who visited ANCs at four sites) was likely small and therefore any changes in prevalence would not be statistically significant . Furthermore, HIV prevalence in this group is not a proxy for incidence in young men or adults in any other age group, condom use was constant and it appears that risk behavior increased . Sexually transmitted infections (STI) prevalence may have decreased . Outputs: The ICR reports that 61 facilities were providing STI treatment at the end of the project and that other project outputs included STI management manuals and training of health workers on STI management . The region has reported that the number of STI cases reported by DOSH increased from 4,592 in 2002 to 20,542 in 2006; however, there is no information provided on utilization or quality of the services, nor baseline figures to show whether the targeted availability of services was achieved . The Borrower’s ICR states that health sector services such as blood safety and prevention of nosocomial transmission were scaled up, but there are no specific data provided on these outputs . About 1,852 people from high-risk groups were beneficiaries of CCSI projects, though there is no information on the content or effectiveness of the projects . Outcomes: The ICR reports that the proportion of people (15-49 years old) who reported having had an STI during the past 12 months decreased from 8.1% in 2002 to 3.7% in 2005. However, reported STIs are often unreliable; there is no data on STI prevalence based on population -based surveys which would capture asymptomatic STI cases . The Borrower's ICR compares preliminary results of the 2006 Multiple Indicator Cluster Survey (MICS) with results from the 2000 MICS, in which the proportion of women (15-49 years) who correctly state three main ways of avoiding infection increased from 33.9% to 64.8%, who correctly identify mother-to-child as a means of transmission increased from 37.7% to 66.5%, and who have been tested for HIV increased from 7.2% to 13.5%. However, other key performance indicators related to HIV prevention reveal limited, if any, impact of project activities . According to behavioral surveys conducted in 2002 and 2005, knowledge about modes of HIV transmission and prevention did not improve (proportion who know at least two transmission modes and two prevention methods was 66.6% in 2002 and 65.2% in 2005). Condom use was unchanged among young people (proportion who used condom at last sexual contact was 72.5% in 2002 and 70.7% in 2005); risky sexual behavior increased (the proportion of young people who had one or more commercial and/or non-regular sexual partners in the last 12 months rose from 26.3% in 2002 and 42.9% in 2005). Exposure to HIV/AIDS messages was already at a high level in 2002 (92.2%) and remained high in 2005 (91.0%). Overall, there appear to be a limited number of outputs that would have affected risk behavior . Data on condom usage and participation in Voluntary counseling and testing (VCT) services among high risk groups was intended to be tracked but was not . 2. Mitigating the impact of the HIV /AIDS epidemic , is rated modest . There is limited evidence for outcomes associated with improving care for people living with HIV/AIDS (PLWHAs) and their families. The ICR provides some evidence of improved attitudes towards PLWHAs . Outputs: The ICR reports that 61 facilities were providing opportunistic infections (OI) treatment, including 11 facilities for tuberculosis (TB) treatment. There is no information provided on utilization or quality of the services, nor on baseline figures to show whether the targeted availability of services was achieved . About 605 PLWHAs were beneficiaries of CCSI projects; 3,881 CCSI sub-projects were approved overall . 202 “sensitization� sessions were conducted by line departments . No detailed information is provided on the content of the sessions or sub -projects. In addition, 418 out of 700 targeted PLWHAs were receiving anti-retrovial (ARV) treatment, seven PLWHA support groups were formed, 1274 orphans and 816 chronically ill adults received free community care; however, these interventions were done jointly with the Global Fund; therefore attribution to this project ’s inputs is difficult. Overall, there is very little evidence provided on the efficacy of treatment activities . Outcomes: Behavioral surveillance survey (BSS) data from 2002 and 2005 show that discriminatory attitudes towards PLWHAs decreased over the project period . 2002 2005 % of respondents willing to share meals with a HIV + 42.8 62.9 person % of respondents willing to care for a relative with HIV 76.8 87.3 % of respondents who would allow an HIV + student to 64.3 74.9 continue with school Source: 2002 and 2005 BSS, as reported in ICR. The ICR notes that US$15 million out of US$16.1 million of external funding for HIV programs was provided through the project prior to the Global Fund availability in late 2004, and that the Global Fund program was focused on treatment and care rather than prevention . Given this information and the apparent significant share of CCSI inputs on IEC (69% of sub-projects targeted the general public and 76% used lectures and workshops as the main method of delivery), it is plausible that the project contributed to improving attitudes towards PLWHAs . 5. Efficiency (not applicable to DPLs): The overall efficiency of the project is rated modest . As the ICR notes, only a very small proportion of project funding (namely CCSI funding) was used for projects specifically targeting high -risk groups. Given the low HIV prevalence rate in the country, more efficient results could have been achieved through better targeted use of funds . In addition, given the already high levels of knowledge, more efficient results could also have been achieved by focusing more on behavior change communications (BCC) rather than information and education campaigns (IEC) Data on cost-effectiveness of CCSI projects were not available . Overall weak implementation of M&E did not enable cost-effectiveness analysis, nor allow “learning-by-doing� to improve project implementation. Serious implementation delays also reduced efficiency . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Based on substantial relevance, negligible efficacy, and modest efficiency, the project outcome is rated Unsatisfactory . a. Outcome Rating : Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: While the project established important institutional structures and frameworks for a nationwide response to the HIV/AIDS epidemic, the capacity of the national -level coordinating body remains weak . The Government has demonstrated a low level of commitment to the project, in part reflected by the inability of the national body to secure legal status through an Act of Parliament, although a bill will be presented to Parliament in December 2007 to attempt to ensure legal status. No additional external funding has been made available as of yet, as the Bank has no follow-up project in the pipeline; however, the borrower is seeking further funding from the Global Fund . Lastly, while important surveillance information was produced by the project to inform future decision -making, little information was produced on the most effective interventions that should be implemented to stem the epidemic . The M&E plan is, however, being updated to ensure more systematic tracking . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: Ensuring Quality at Entry : Bank performance at quality-at-entry is rated moderately unsatisfactory . While the project design was consistent with a multisectoral approach with community -level interventions, there were significant shortcomings in preparation . Critical institutional capacity assessments were not carried out, leading to significant delays in implementation . The project design, while acknowledging the importance of prioritizing targeting of high risk groups and behavior change, did not specify measures to ensure that this focus was adequately carried out. The M&E design was not effective for learning by doing, nor for collecting data to inform decision-making or demonstrate results. Projects risks were not adequately assessed, including political constraints to ensuring sustained government commitment . Quality of Supervision : Bank performance at supervision is rated moderately satisfactory . The supervision team made a concerted effort to improve implementation through such measures as the Accelerated Results Initiative and a QER. These measures, along with the standard supervision missions, were intended to sharpen the project’s focus and improve the implementation agency ’s capabilities. at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: Government performance : Government performance is rated unsatisfactory . While the National AIDS Council (NAC) and National AIDS Secretariat (NAS) were made functional, the commitment level of the government waned considerably. The ICR notes that NAC did not convene regularly as planned and that the government reduced the salary scale of NAS staff to well below market level, thus contributing to difficulties in recruiting and maintaining qualified staff. Implementing Agency performance : Implementing agency performance is rated moderately unsatisfactory . According to the ICR, NAS made limited efforts to act on recommendations from the Bank team on improving management performance and project effectiveness . Project monitoring was not sufficiently carried out . The health sector department produced significant outputs, though these were in partnership with Global Fund activities and thus difficult to attribute solely to the Bank ’s project. There is limited information on the performance of other line departments . While disbursement levels for the CCSI component surpassed the target level, there is limited information on the effectiveness of activities of CCSI agencies . a. Government Performance :Unsatisfactory b. Implementing Agency Performance :Moderately Unsatisfactory c. Overall Borrower Performance :Unsatisfactory 10. M&E Design, Implementation, & Utilization: M&E design is rated modest . While the proposed indicators were in line with global practices, they were not appropriate for the nature of this specific project (“learning-by-doing�) nor for the state of the epidemic in the Gambia (concentrated). There were few indicators on institutional performance, other than the establishment of institutional structures and action plans . In addition to the indicators discussed previously, the only indicators for the CCSI component related to disbursement levels and number of participating entities; no other indicators related to effectiveness of community interventions were planned . Thus, even if the data were sufficiently collected, they would not have been of optimal usefulness to improve project effectiveness . Baseline and/or target figures for some of the intermediate indicators were not established, making it difficult to assess the project ’s achievement. Of particular note: the target for the PDO indicator on condom usage among young people was a 40% increase. However, no baseline figure was established at the beginning of the project; a survey conducted one year into project implementation subsequently showed that the condom usage rate was already at 72.5%. Similarly, the target for the indicator on exposure to HIV/AIDS messages was 50% of the population. However, again, no baseline figure was established and the same survey showed that this value was already at 96.2%. The project design did provide for important monitoring activities through the project ,such as establishing surveillance sites and supporting key operational research activities such as mapping of high risk groups, evaluation of specific types of interventions, and pilot testing of innovative approaches . M&E implementation is rated modest . A few critical M&E activities were carried out, namely annual sentinel surveillance studies (2001-2005) and two behavioral surveillance studies (2002 and 2005). However, most other M&E studies were not conducted . The key position of M&E specialist was not filled for a long time . Systematic monitoring of project activities, particularly the CCSI component, was not conducted . A CCSI assessment was conducted in 2006, but it took place too late to inform decision -making and focused primarily on administrative aspects rather than on results . M&E utilization is rated negligible . The project failed to provide an adequate data collection to assess implementation progress or project impact . Thus M&E data could not be used to guide and modify project activities or to attribute outcomes to the project . a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): An unintended positive impact was that the Borrower was able to leverage the project ’s activities, particularly the establishment of the institutional framework, to access Global Fund grants . As noted above, these grants were used in coordination with Bank financing to produce significant outputs in the health sector . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Unsatisfactory Based on substantial relevance, Unsatisfactory negligible efficacy and modest efficiency. Risk to Development Significant Significant Outcome : Bank Performance : Moderately Moderately There were significant shortcomings in Satisfactory Unsatisfactory the project preparation, including inappropriate indicators, insufficient focus of project activities given the state of the epidemic, and lack of proper institutional assessments . Borrower Performance : Moderately Unsatisfactory There was a drop in government Unsatisfactory commitment and a lack of action to address implementation problems. Quality of ICR : Exemplary NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Agreed-upon actions for improving implementation may not take place in the absence of an accountability mechanism. Bank staff made important recommendations to increase the effectiveness of activities, but these were not followed. For a project relying on a “learning-by-doing� approach, indicators related to effectiveness of activities must be monitored and must be discernible in a short time frame in order to be useful . The only indicator used for the CCSI component in this project was “disbursement level�. Sustained government commitment is necessary from project preparation through project closing to ensure a supportive environment. The lack of baseline data can result in setting inappropriate targets and emphasizing the wrong activities . Most project resources were spent on raising awareness, which was already high, rather than on changing behavior. Resources were also aimed at the general population rather than on prevention among high -risk groups whose behavior is the main driver in a concentrated epidemic . The Bank, the Borrower and civil society are more inclined to promote interventions for the general population rather than prevention activities for high -risk groups. This was a lost opportunity for the Bank to help the Borrower invest in more effective and cost -effective HIV/AIDS programs. The failure to conduct sound assessments of institutional needs, capacities and alternatives can result in institutions that are ineffective in achieving project objectives . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR presents a clear and candid assessment of what was and was not achieved by the project . It focuses the discussion of achievements on results – rather than on process – and presents an honest picture of the lack of strong data as evidence and the lack of adequate data to attribute overall results to the project . The ICR team attempted to analyze and draw results from raw data that was available, particularly on the CCSI sub -projects, as well as to validate the comparability of statistics presented as trends, particularly in comparing prompted and unprompted survey questions and presenting HIV prevalence trends by site (although sample size should be taken into account in the analysis as well) The ICR makes a good attempt to document the activities of other donors during the same period so as to inform the assessment of attribution . It makes important observations about the project indicators, including the difficulties of relying on HIV prevalence vs . incidence rates and the need to select indicators relevant to a concentrated vs. generalized epidemic. a.Quality of ICR Rating : Exemplary