ICRR 13245 Report Number : ICRR13245 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 10/20/2009 PROJ ID : P073442 Appraisal Actual Project Name : Hiv/aids Global US$M ): Project Costs (US$M): 7.00 6.77 Mitigation Support Project Country : Guinea-Bissau Loan/ US$M): Loan /Credit (US$M): 7.00 6.77 Sector Board : HE Cofinancing (US$M): US$M ): Sector (s): Other social services (40%) Sub-national government administration (40%) Health (20%) Theme (s): HIV/AIDS (33% - P) Population and reproductive health (17% - S) Personal and property rights (17% - S) Other communicable diseases (17% - S) Participation and civic engagement (16% - S) L/C Number : CH110 Board Approval Date : 06/02/2004 Partners involved : Closing Date : 12/31/2007 12/31/2008 Evaluator : Panel Reviewer : Group Manager : Group : Judy Twiggs Kris Hallberg Monika Huppi IEGSG 2. Project Objectives and Components: a. Objectives: Two sets of objectives applied over the following periods : 14 , 2005 to December 6, 2007 , covering 81. First period : January 14, 81 .4% of disbursements • Reduce the spread of HIV/AIDS infection in the Recipient’s population • Increase the access to STI /HIV/AIDS treatment • Mitigate the health and socio-economic impact of HIV/AIDS at the individual, household, and community level • Build strong and sustainable national capacity to respond to the HIV /AIDS epidemic Second period : December 7, 2007 to December 31, 18 .6% of disbursements 31 , 2008 , covering 18. • Increase access to HIV prevention, care, treatment, and mitigation services The ICR gives two reasons for the restructuring . First, the ICR says that the Mid Term Review (MTR) in October 2006 discovered that achievement of the project development objectives (PDO) could not be measured and therefore several key project indicators (KPIs) needed to be reformulated. Second, the ICR says that restructuring was intended to align the project with Multi -Country HIV/AIDS Program (MAP) implementation experience; in particular, the MAP experience had shown that it was unrealistic to expect, and thus target, a reduction in HIV prevalence over the span of a single project . The new objectives and associated new KPIs were intended to be more realistic and attainable. The restructuring package was part of an umbrella restructuring and amendment of the financial agreements for MAP projects. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 12/13/2007 c. Components (or Key Conditions in the case of DPLs, as appropriate): The components did not change when the project was restructured. A. Community and Civil Service Initiatives, and Capacity Building (original, US$ 2.50 million; actual, US$ 1.61 million ), focused on social mobilization and community -level responses to HIV/AIDS. Funds were to be used expand and strengthen HIV/AIDS activities of community-based organizations (CBOs) and civil society associations, or to contract NGOs or private sector entities . Funded groups were to be encouraged to focus, but not necessarily limit, their activities on the most vulnerable groups, including people living with HIV and AIDS (PLWHA), hospital workers, orphans, youth, commercial sex workers, and truck drivers . Among these, priority was given to orphans and PLWHA. B. Government Multi -Sector Response (original, US$ 2.20 million; actual, US$ 0.59 million ), to scale up and strengthen prevention, care, support, and mitigation of the social and economic impact of HIV /AIDS through programs and activities of the public sector . Included two subcomponents : 1. Support to the Ministry of Health by partially financing its work plan, including prevention activities, medical treatments, counseling, supply of drugs, and monitoring the progress of the epidemic . Funding sources from other partners were to complement project activities to entire national coverage . 2. Support to other Ministries and Government Agencies , focusing on mainstreaming HIV/AIDS awareness and prevention in the regular activities and workplans of key ministries (Education, Finance, Defense, Justice, Agriculture, and Transportation ). C. Institutional Development for Program Management (original, US$ 2.30 million; actual, US$ 4.57 million ), supporting the National AIDS Council (NAC), whose responsibilities focused mainly on advocacy and exerting leadership on the multi-sectoral response to HIV/AIDS, and the National AIDS Secretariat (NAS), including its regional components and focal points, to fulfill its national HIV /AIDS coordination mandate as well as technical support, financial management, and monitoring and evaluation . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Costs /Financing : The Development Grant Agreement (DGA) was amended for the first time in August 2006 to add a new disbursement category for support to sub -projects proposed and implemented by civil society organizations (CSOs), with a reallocation of SDR 700,000 into this new category. This category was to have been included in the original project design, but was omitted; the ICR does not provide an explanation for the initial omission . The project originally envisioned five operational surveillance centers, one in each of the five regions covered by the project. At the time of the MTR, it was determined that funding was insufficient to have the desired impact in all five regions, and there was significant overlap with the activities of the Global Fund in two of the regions . It was therefore agreed to focus Bank interventions on the other three regions . Dates : The closing date was extended by one year, from December 31, 2007 to December 31, 2008, as a part of the project’s restructuring in December 2007. 3. Relevance of Objectives & Design: 81 .4% of disbursements, 1/14/ First period (81. 12 /6/07) 14 /05 – 12/ 07 ) is rated Modest : Objectives – Substantial . At the time of project preparation, there were no comprehensive and reliable HIV statistics and only limited actions to prevent HIV /AIDS and mitigate its impact. It was projected that without intervention, 24 to 40 persons daily would be infected with HIV . The project supported the Government ’s medium- term strategy to expand and accelerate the national response to the epidemic . Addressing the HIV/AIDS epidemic continues to be a priority in the country, with estimated current prevalence at 3-7%. The danger of a rapidly spreading HIV /AIDS epidemic is featured prominently in the Bank ’s 2009 Interim Strategy Note for Guinea-Bissau. Design – Modest . According to the ICR, available data at the time of project preparation showed that HIV was spreading mostly along main transportation axes . Yet the Project Appraisal Document (PAD), although it contains a detailed and logical results chain and appropriately focuses interventions on risk groups (PLWHA, hospital workers, orphans, youth, commercial sex workers, and truck drivers ), does not focus community and civil society interventions specifically on prevention or on the risk groups most likely to spread HIV along transportation routes . It prioritizes instead orphans and PLWHA. The project appropriately focused on the five geographic regions with the highest HIV/AIDS prevalence rates. Outcome/impact indicators included HIV seroprevalence, which is a flawed indicator as it mixes incidence with mortality, but also included good proxies for HIV incidence (STI incidence and condom use ). Support to line ministries was not sufficiently prioritized among ministries . The PAD adequately anticipated key risks, including weak Government commitment, a weak health sector, inadequate NAS capacity, and insufficient data to measure project impact, but proposed mitigation measures were weak and /or not followed through. Overall, the project’s objectives were too ambitious given the fragility of the country in general, extremely weak implementation capacity, and lack of available information to measure the project ’s impact (this concern was raised prominently at the decision meeting, but there is no evidence that it was addressed in project design ). The project did not include an M&E system in its design, and therefore there was no M&E system in place at the start of the project . Second period (18. 12 /7/07 – 12/ 18 .6% of disbursements, 12/ 13 /08) 12 /13/ 08 ) is rated Modest : Objectives – Substantial . The restructuring of objectives was appropriately in line with the umbrella restructuring of the MAP projects. The fight against HIV/AIDS remained an important priority for the country . Design – Negligible . The revised design appropriately dropped HIV prevalence as an indicator, and added indicators for condom use among youth and sex workers, HIV testing, provision of treatment to reduce mother -to-child transmission (MTCT), and HIV knowledge among youth . However, the restructuring of the project was too modest given the difficulties that had been experienced; overall, the restructuring was a missed opportunity that should have gone beyond just changing the development objectives to include more substantial attention to M&E, general supervision, and impact assessments . 4. Achievement of Objectives (Efficacy): Following IEG/OPCS guidelines, this review will assess the achievement of each objective separately, noting the percentage of project disbursements and time period that apply to each objective . The ICR states that the ICR Mission was unable to obtain detailed data and information on project outcomes . 81.4% of disbursements, Reduce the Spread of HIV Infection in the Recipient ’s Population (objective applies to 81. 14 /2005 – 12/ 1/14/ 06 /2007): 12 /06/ 2007 ): Substantial Although the project did not meet the target for condom use among young people, it met or exceeded targets on other important prevention-related indicators, and it stressed distribution of condoms to key risk groups (truck drivers and migrant workers). Outcomes: HIV seroprevalence rates changed little : among antenatal care seekers, 4.6% in 2005 and 4.6% in 2007; among bloodbank attendees, 11.5% in 2005 and 8.8% in 2007; national estimate, > 4% in 2005 and between 2.1 and 6.0% in 2007 (median 3.8%). Prevalence is a flawed indicator of the spread of infection, however, as it mixes incidence with mortality. Meaningful proxies for incidence showed mixed results . The percentage of sex workers reporting using a condom with their most recent client increased from less than 20% in 2004 to 32% in 2008 (against a target of 25%). The number of persons aged 15 and older receiving counseling and testing for HIV and receiving their test results within the preceding 12 months increased from 993 in 2004 to 13,996 in 2008 (against a target of 5,000). However, the percentage of young women and men aged 15-24 reporting the use of a condom the last time they had sex with a non-marital, non-cohabiting sexual partner decreased from 35% in 2004 to 31% in 2006 (against a target of 65%). The number of pregnant women receiving antiretroviral medications to reduce the risk of mother-to-child transmission (MTCT) of HIV increased from 144 in 2004 to 258 in 2008 (against a target of 5,000). The percentage of youth aged 15-24 who can correctly identify at least two methods of HIV transmission increased from 5% in 2004 to 7% in 2006 (against a target of 65%). Outputs: Just over two million condoms were distributed in the first three quarters of 2008, against a target of one million, with the Transport Ministry arranging for the distribution of free condoms to truck drivers and migrant workers at gas stations, and the Sailor ’s Union distributing condoms for free . An unspecified number of educational billboards were installed, and 774 radio programs were transmitted in local languages . An increasing number of men and women were counseled and tested for HIV, outside MTCT (1,020 in 2005, 3,861 in 2006, 2,427 in the first quarter of 2007) (no target was defined). Six operational VCT centers were established by 2007, meeting the target. VCT services were concentrated along major transportation axes . 71 proposals from associations and NGOs were funded (up to USD 20,000 each) for unspecified subprojects; there is no information available to evaluate the effectiveness of impact of these projects . HIV /AIDS Treatment (objective applies to 100% Increase Access to STI /HIV/ 14 /2005 – 100 % of disbursements, 1/14/ 12 /31/ 12/ 2008 ): Modest 31 /2008): Outcomes: The number of PLWHA receiving antiretroviral therapy increased from zero in 2003 to 65 in 2005, 436 in 2006, and 496 in the first quarter of 2007. This indicator was originally constructed in terms of percentage of PLWHA receiving treatment (target of 20%), but the ICR does not provide percentage data . The number of PLWHA receiving medical treatment for opportunistic infections changed from 7,199 in 2003 to 9,712 in 2005, 6,957 in 2006, and 1,938 in the first quarter of 2007. Again, the indicator was originally constructed in terms of percentage of PLWHA receiving treatment (target of 20%), but the ICR does not provide percentage data . The ICR states that the supply of antiretroviral drugs did not materially increase as a result of the project, and that weak management of drugs and rupture of stocks of reagents and laboratory products remained a concern . The TTL states that, over the last 2-2.5 years, there have been sufficient antiretroviral medications available to treat all those in medical need of treatment . Outputs: The percentage of secondary and tertiary health facilities in pilot regions having the capacity to offer STI/HIV/AIDS care increased from zero in 2005 to 96.15% in 2007 (against a target of 100%). Mitigate the Health and Socio -Economic Impact of HIV /AIDS at the Individual, Household, and Community Level 81 .4% of disbursements, 1/14/ (objective applied to 81. 12 /06/ 14 /2005 – 12/ 2007 ): Modest 06 /2007): Outcomes: Three national associations of PLWHA were formed, with unspecified impact . The households of 4,544 orphans and vulnerable children (OVC) received free basic external support in caring for the child in 2008. The target for this indicator was set at 10%, but the ICR states that the percentage of OVC receiving care is unknown . The ICR states that there are no data to assess or measure impact on this objective . Anti-discrimination legislation was adopted. Outputs: Eight line ministries included HIV/AIDS mitigation in their national strategies and work plans (against a target of six). Partnerships with NGOs provided unspecified emotional, nutritional, and medical support to orphans, children that suffered sexual abuse, and children of HIV -positive mothers and PLWHA. PLWHA and sex workers received unspecified training enabling them to apply successfully for micro -credit and to develop skills for various income-generating activities. Build Strong and Sustainable National Capacity To Respond to the HIV /AIDS Epidemic (objective applies to 81 .4% of disbursements, 1/14/ 81. 12 /06/ 14 /2005 – 12/ 2007 ): Modest 06 /2007): The community response component built the capacity for advocacy and activism among PLWHA, expanding their national network and enabling them to reach out to larger numbers of people . However, the NAS is described by the ICR as still ineffective, with inadequate management and implementation capacity . It did not provide the expected leadership for the multi-sectoral response to HIV/AIDS, and overall coordination was weak . The lack of proactivity of NAS was a major factor in implementation delays . 18 .6% of disbursements, 12/ Increase Access to HIV Prevention Services (objective applies to 18. 07 /2007 – 12 /07/ 12 /31/ 12/ 2008 ): Substantial 31 /2008): The ICR does not provide sufficient data to assess this objective separately from the objective it replaced (“reduce the spread of HIV infection in the recipient ’s population�). There is no indication that the level of achievement of this new objective was significantly different than the level of achievement of the original related objective . 18 .6% of disbursements, 12/ Increase Access to Mitigation Services (objective applies to 18. 07 /2007 – 12/ 12 /07/ 31 /2008): 12 /31/ 2008 ): Modest The ICR does not provide sufficient data to assess this objective separately from the objective it replaced (“mitigate the health and socio-economic impact of HIV/AIDS at the individual, household, and community level �). There is no indication that the level of achievement of this new objective was significantly different than the level of achievement of the original related objective . 5. Efficiency (not applicable to DPLs): Overall efficiency is rated Modest. According to the ICR, the project ’s resources were not managed efficiently and in line with project objectives, and as a result the delivery of services was not efficient . Control over resources and budget follow-up on the part of project management were lax, and there was weak monitoring of activities . An illustration of these weaknesses is the fact that Components 1 and 2 were significantly underfunded (80.0% and 26.8%, respectively, of appraisal estimates ), while disbursements under Component 3, which covered institutional development for project management, were almost double the appraisal estimate . This allocation of resources did not appear to respond to strategic needs, but rather was due to weak management and oversight . ERR/FRR, Appraisal and ICR Estimate: N/A Appraisal: Net Present Value USD , IRR: N/A ICR estimate: Not performed due to insufficient data ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Period Objectives Relevance Efficacy Efficiency Outcome Weight Multiply (convert into (based on outcome number from 1 share of number times to 6) disbursem weight and ents for add each period) 1/14/2005 – Reduce Modest Substantial Modest Moderately .21 .63 12/06/2007 spread of Unsatisfactory HIV (3) Increase Modest Modest Modest Unsatisfactory .21 .42 access to (2) treatment Mitigate Modest Modest Modest Unsatisfactory .20 .40 impact (2) Build Modest Modest Modest Unsatisfactory .20 .40 national (2) capacity 12/07/2007 – Increase Modest Modest Modest Unsatisfactory .06 .12 12/31/2008 access to (2) treatment/ca re Increase Modest Substantial Modest Moderately .06 .18 access to Unsatisfactory prevention (3) services Increase Modest Modest Modest Unsatisfactory .06 .12 access to (2) mitigation services Overall outcome rating 2.27 Unsatisfactory a. Outcome Rating : Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Many of the project outcomes that would have been key to future sustainability of project efforts – drug supply, biosafety, and basic measurement of the epidemic – were not successfully achieved . The Global Fund approved a US $ 13.1 million grant in November 2008 that will support some activities initiated under the Bank project (such as the associations of PLWHA) but not others (Regional AIDS Centers). Funding, however, is no guarantee of sustainability, and the new Grant will have to contend with a still very weak health sector with weak capacity at the regional level. NAS is better equipped to coordinate a national program than it was just a few years ago, but those institutional gains will be lost if not continuously supported with external resources and technical capacity . There is little indication that the necessary efforts are planned to strengthen health policies, institutional capacity, and skills and motivation of health staff that are necessary to sustain movement forward . The Summary of the Borrower’s ICR stresses that, although a basic foundation was laid for effective management capacity related to project activities, there is no guarantee that there will be sustainability of the achieved results . a. Risk to Development Outcome Rating : High 8. Assessment of Bank Performance: at -Entry is rated Unsatisfactory . The project’s objectives were too ambitious given the weak Ensuring Quality -at- implementation capacity in the country and size of the grant . Appropriate buy-in was not obtained from an array of Government stakeholders during project design, with discussions limited to the Ministry of Health . Although key risks were identified, mitigation measures were weak or inadequate . No detailed or reliable cost estimates were prepared, and there was no evaluation of the costs of the proposed implementation structure relative to available funds and project objectives . There was no M&E system included in project design . Too many activities on which the success of the project depended were assumed to be carried out by other projects, particularly the overall development of health sector infrastructure and safeguard measures related to waste management and the construction of incinerators (National Health Development Project). Quality of Supervision is rated Unsatisfactory . The Bank team made commendable efforts to coordinate with external partners, especially UN agencies and the Global Fund, even though those partners were not always receptive or fully collaborative . However, the task team does not appear to have succeeded in building a solid partnership with the Borrower. The Bank did not take measures to ensure that there was sufficient experienced staffing of the NAS, resulting in numerous implementation delays . Project staff and the Bank team did not communicate well, at least in part because there were no Bank staff with appropriate language skills . Mapping the project to an environment unit (AFTEN) and then to the fragile states and social development unit (AFTCS) was not an effective way to implement a complex multisectoral operation in a fragile country with extremely weak capacity and a very weak health sector; having two unrelated units involved in supervision was not conducive to strong managerial and technical supervision oversight . There was insufficient support from the Bank in procurement supervision, and in particular for review of procurement plans and performance . The composition of the Bank’s teams during supervision missions was weak, with the Social Development Specialist participating in only one mission, the Public Health Specialist participating in only three missions, and no Safeguard Specialist participating as part of the team . The Borrower’s ICR refers to lack of follow-up on the part of the task team as one of the weakest point of project implementation . at -Entry :Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Unsatisfactory c. Overall Bank Performance :Unsatisfactory 9. Assessment of Borrower Performance: Government Performance is rated Moderately Unsatisfactory . Discussions of the Strategic National Plan (SNP) were limited to the Ministry of Health, excluding other Government stakeholders; as a result, Government ownership was weak initially, up to the prime ministerial level . This situation changed after the 2007 restructuring; as a result of Cabinet -level discussions, there was much greater involvement of the Prime Minister after November 2007 in HIV/AIDS awareness activities and better overall Government buy -in. The then Minister of Health was not cooperative in releasing staff to work at the NAS . Implementing Agency Performance is rated Unsatisfactory . Voluntary counseling and testing, as well as the community component, were spearheaded by highly motivated NAS staff . However, overall staffing of the NAS was inadequate throughout, lacking capacity to plan and implement the project . None of the staff had ever managed a program before. There was a lack of continuity of NAS staff between preparation and implementation, leading to misunderstandings and implementation delays . Lack of proactivity of NAS was a major factor in implementation delays. Procurement was the weakest link, primarily due to passive procurement execution and weak capacity to write terms of reference . It was agreed to restructure NAS and to bring in new staff for the final year of the project (following project restructuring); this did not happen, however, due to delays in recruitment of new staff. a. Government Performance :Moderately Unsatisfactory b. Implementing Agency Performance :Unsatisfactory c. Overall Borrower Performance :Unsatisfactory 10. M&E Design, Implementation, & Utilization: M&E Design : Negligible to Low . Despite a long list of indicators in the PAD, there was no M&E system included in project design, making it impossible to measure key performance indicators . Risk mitigation measures specified in the PAD proposed a data assessment as part of establishing an M&E system, but there is no evidence that a systematic data assessment was carried out, and the proposed mitigation measure did not adequately address the risk that data would be difficult to collect or not available in a timely manner . The lack of a baseline and an inadequate and delayed M&E system significantly impacted project performance . A consultant was hired to design an integrated M&E system that would apply to all HIV /AIDS activities in the country, but she could not complete the work due to health problems. M&E Implementation : Modest . The consultant who was hired to design an M&E system trained an M&E officer in data collection and reporting . However, implementation was difficult, and NAS was not able to produce reports effectively. As of early 2007, there were still delays in finalizing the M&E system, negatively impacting project performance. Overall M&E, especially data collection and reporting, remained inadequate and a major constraint throughout. M&E Utilization : Negligible to Low . Limited M&E capacity in NAS, the lack of a baseline for many indicators, and the lack of availability of a management information system for more than two years made it impossible to use M&E results to inform project activities and the formation of policy . a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards : The project was an environmental category B . The most important environment issue was waste disposal. An incinerator was supposed to be built in the National Hospital, but there were serious delays in implementing the waste management plan. By 2007, construction on the incinerator had not yet started, and so in the last year of the project, three mini -incinerators (using WHO-approved guidelines) were built instead at the National Laboratory and at two health centers . Medical waste disposal therefore did not meet safeguard standards during most of the life of the project . Safeguard measures were inadequately evaluated and depended on the multi-donor supported National Health Development Project (NHDP) for required investments in incinerators . Unintended Positive Impact : The project empowered local communities to respond to HIV /AIDS, and awareness activities apparently increased acceptance of PLWHA . This reduction of taboo and stigma may have contributed to social cohesion in the country . The ICR states that, although many key aspects of the project were not successful or cannot be evaluated due to insufficient data, the project has laid a foundation for HIV /AIDS work in the country which, if appropriately followed up, could make a difference in mitigating the impact of the epidemic . Unintended Negative Impact : There was inadequate collaboration between UN agencies (with the exception of WHO) and NAS, despite an agreement by the United Nations Development Program (UNDP) to provide technical support and build capacity in NAS, with significant negative impact on the project . UNDP, as implementing agency for a Global Fund grant, appeared to have limited interest in supporting capacity -building for NAS. 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Unsatisfactory Unsatisfactory Risk to Development High High Outcome : Bank Performance : Unsatisfactory Unsatisfactory Borrower Performance : Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The perceived need for rapid response to a serious epidemic emergency must be balanced by the kind of careful design that is necessary for project success . Projects that are designed to learn -by-doing must have sufficient data to gather emerging lessons and adjust as the project proceeds; if a good M&E system and baseline cannot be put in place early during project implementation, then a learning -by-doing project should not be approved. Recruiting staff with appropriate skills in a fragile state is likely to be a lengthy and difficult process, and this likely obstacle should be factored into project design . Steps should be taken, in such an environment, to front-load capacity building and technical support during a project ’s early phase, and project activities that require relatively less capacity should be deliberately sequenced earlier into the project . Creating a new institution is a complex and time -consuming task, particularly in a fragile state . In the case of NAC and NAS in Guinea-Bissau, the difficulties were compounded when the new institutions were required to work cross-sectorally, acquire buy-in from an array of Government structures, and include diverse representation from across society . The low capacity of NAS was a major factor affecting project implementation. In fragile and post -conflict countries, project objectives and design should be as straightforward and modest as possible . Project design must take full account of weak institutional capacity, the possibility of political instability and staff/ministerial turnover, the challenge of securing Government support and ownership, and the need for a strong supervision effort . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR presents an impressive amount of information and analysis, given the lack of data available regarding project outcomes. a.Quality of ICR Rating : Satisfactory