ICRR 12512 Report Number : ICRR12512 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 09/28/2006 PROJ ID :P071617 Appraisal Actual Project Name :AIDS Response Project Project Costs 27.8 26.5 (GARFUND) US$M ) (US$M) Country :Ghana Loan/ Loan US$M ) /Credit (US$M) 25.0 24.8 Sector (s):Board: ): HE - Health (85%), US$M ) Cofinancing (US$M) Central government administration (10%), Other social services (5%) L/C Number :C3458 FY ) Board Approval (FY) 01 Partners involved : Closing Date 06/30/2005 12/31/2005 Evaluator : Panel Reviewer : Division Manager : Division : Denise A. VaillancourtMartha Ainsworth Alain A. Barbu IEGSG 2. Project Objectives and Components a. Objectives The objective of the project is to intensify multisectoral activities designed to combat the spread of HIV /AIDS and reduce its impact on those already affected by HIV /AIDS. b. Components (or Key Conditions in the case of Adjustment Loans ): (Estimated costs of components presented below include contingencies .) 1. Prevention and Care Services (estimated cost: US$21.0 million, 75% of total; actual cost: US$21.6 million, 87% of total): Financing of non-health public agencies and civil society entities to provide a broad spectrum of preventive and care activities at national, regional and local levels . Seventy percent or US$15 million was to be earmarked for civil society entities. Prevention activities would be designed to reduce high risk behavior and exposure to risk, and reduce vulnerabilities by raising awareness and “de-stigmatizing� the disease with a view to improving access to prevention and care. Care activities would reduce the vulnerability of families affected by HIV /AIDS through the financing of services to protect their rights, income generation activities, care for orphans and home -based care for the ill. Financing would be provided through three windows : Window A for proposals from line ministries; Window B for proposals from civil society organizations, NGOs, associations of people living with HIV /AIDS (PLWHA), and districts; and Window C for seed money (not to exceed $2500) for very small CBOs and associations . 2. Strengthening Public /Private Institutions for HIV /AIDS Control and Care Giving (estimated cost: US$2.4, 9% of total; actual cost: US$1.4 million, 5.5% of total): Provision of training and technical assistance on technical and practical aspects of working with HIV /AIDS, and some training on community participation and project management . The training was to be subcontracted to NGOs and line ministries with requisite expertise . This component was intended to strengthen the impact of activities financed under Component 1. 3. Knowledge Management (estimated cost: US$1.4 million, 5% of total; actual cost: US$0.5 million, 2% of total): Support for the establishment of a mechanism /strategy to collect, organize and disseminate up -to-date information on AIDS prevention and care, best practices of existing projects, research results, and any reports that would be useful to implementing entities. This component also intended to bring together subproject representatives to share experiences with subprojects with similar orientations and to strengthen the impact of activities financed under Component 1. 4. Project Management, Monitoring and Evaluation (estimated cost: US$ 3.0 million, 11% of total; actual cost: US$1.4 million, 5.5% of total cost): Support of costs associated with project management and oversight . This component was designed to work through and support a new institution for National HIV /AIDS Program coordination. Under the office of the President, the Ghana AIDS Commission (GAC) was established, supported by a GAC Secretariat, which served as the PIU for the GARFUND . The institutional arrangement was meant to strengthen the GoG’s capacity to implement a balanced, diversified multi -sector response, engaging all relevant government sectors, NGOs and grassroots initiatives . Project financing of the GAC Secretariat covered the costs of doing business (project coordination, strategic management and oversight ) and salaries of all staff. A monitoring and evaluation subcomponent was included to assess the progress attained in the fight against AIDS nationwide on the basis of targets defined in the Strategic Framework and to monitor GARFUND implementation . c. Comments on Project Cost, Financing, Borrower Contribution, and Dates The ICR (p. 19) reports a total project cost of US$ 26.5 million or 95% of the original estimate of US$27.8 million. However, two annex tables show different estimates of total actual costs, at US$ 24.9 million and US$28.3 million, respectively. (Actual costs by component shown above add up to US$ 24.9, as this is the only table that itemizes costs by component.) World Bank loan data (SAP) show that of the original IDA credit of 19.6 million SDR, 18.08 million was disbursed, and 1.52 million was cancelled, contrary to the ICR statement that the credit was fully utilized . Actual government counterpart funding was US$ 1.7 million, or 94% of appraisal estimate. However, delays in counterpart financing and inadequate budgeting of contributions by the line ministries were both issues during implementation. The ICR does not report on total actual contributions of civil society to subprojects, estimated at US$1.0 million at the time of appraisal. Despite a long delay (about 16 months) in reaching effectiveness, largely attributable to elections and a change in Government, the project was executed over a period just over three and one half years, with one six-month extension to December 31, 2005. 3. Relevance of Objectives & Design : The overall relevance of the project is modest. The relevance of objectives is modest . Project objectives were responsive both to the Government ’s past (2001-2005) and current (2006-2010) HIV/AIDS Strategic Frameworks, and to the Bank ’s previous and current CAS, which emphasize poverty reduction and human development, with specific focus on the prevention and control of HIV/AIDS. However, the objectives lacked adequate specificity that would have accommodated the tracking and measurement of the behaviors, risks and coverage of the general population as well as of high -risk groups and high-risk geographic areas that (a) were reflected in national strategy; and (b) would have been more appropriate for framing and measuring the comparative advantages and successes of civil society organizations in serving the specific needs of both general and targeted populations . The relevance of the design is negligible . Project design was not sufficiently based on empirical evidence on the dynamics of the epidemic or on the awareness, attitudes, knowledge, risks, behaviors and vulnerabilities of various key groups that was available . Nor did it identify a sufficiently robust operations research program that would have provided enhanced knowledge of challenges on which basis the design and targeting of interventions could have been fine-tuned (stigma/discrimination, and gender dynamics, being two cases in point ). The project logframe was not systematic in developing a coherent results chain, linking inputs, outputs, outcomes and impact . The exclusion of the Ministry of Health (MoH) from project activities was not wise, given that : (a) SWAp monies could not be earmarked for HIV/AIDS; and (b) the exclusion of MoH caused tensions and lack of coordination between GAC and MoH which persisted throughout the life of the project . In addition, the design did not reflect adequate coordination/collaboration with development partners, leading to inadequate information on the total program costs, and on the technical and financial contributions of donors to program implementation . The design was not based on capacity assessments of various actors and was inadequate in providing quality control of activities (IEC and home-based care, for example), nor did it guide the prioritization of the public sector agencies and civil society actors in consideration of their respective experience and potential for impact . The institutional/organizational framework for program/project management and implementation did not build on existing capacity/frameworks, at central and decentralized levels . There was a failure by the Bank to assess adequately existing institutional arrangements for HIV /AIDS, that would have allowed a more systematic documentation of experience to date and that should have formed the basis for the institutional set -up established under the project. Design issues are due in part to the fact that this project was based on the emergency Africa Multi -Country AIDS Program (MAP), a horizontal APL approved by the Board in 2000, that emphasized the urgency of the Bank's response and the rapid provision of financial support to a multiplicity of actors and stakeholders, without the benefit of rigorous appraisal of the country -specific technical, economic, social and institutional elements of the program /project during preparation. 4. Achievement of Objectives (Efficacy) : Overall efficacy is rated as modest. 1. Reduce the spread of HIV infections – modest HIV prevalence among pregnant women 15 to 19 years, a proxy indicator for tracking incidence (rate of new infections) in that age group, decreased from 2.3% in 2002 to 0.8% in 2005. This decline may be partially attributable to changes in the composition of antenatal sentinel sites over time (e.g., addition of lower-prevalence rural sites). In any case, the ICR notes that the link between project interventions and a reduction in the spread of HIV infections could not be documented . Significant training was delivered by the project to a range of implementing agencies . However: (a) no measures of improved capacity are provided in the ICR; and (b) DFID capacity building activities targeted the same broad range of implementers and was significantly more important in financial terms than GARFUND support. The content and complementarity of GARFUND /DFID efforts are not discussed, making it difficult to establish attribution. During most of the life of the project there were tensions between the MoH and the GAC that undermined national HIV/AIDS efforts, given the unequivocal leading role of MoH on the technical aspects of the program . 2. Reduce the impact of AIDS on those infected and their families – modest The ICR (p. 7) reports that 30,000 PLWHA received home-based and community-based care services, with NGO support, in 50% of districts. It also reports that 482,526 people benefited from medical care and psychosocial support provided through the health system . While these services are not financed under the project, operations notes that project activities might have stimulated an increase in the use of these services . However, the breakdown of this data, by services provided, and by type of beneficiary (household members, OVCs, caretakers...) is not provided, nor is baseline data available to allow trend analysis of uptake in these services. It is also reported that 17,500 orphans have benefited from support ranging from payment of school fees, provision of nutritional supplements and medical assistance . No data on coverage of these groups is available. The quality. coverage, effectiveness and sustainability of services provided to PLWHA, orphans, and vulnerable children is not known and the impact of this support has not been measured . A referral system linking home-based and community-based care services to formal health services was not set up . This is likely due, at least in part, to the exclusion of support to, and weak coordination with, MoH . Income generation activities for PLWHA and their families were financed but neither the population coverage nor the type/amount of support is presented in the ICR . Further, the benefits of these activities in terms of mitigating the impoverishing effects of the disease on PLWHAs and their families have not been assessed . 5. Efficiency : Efficiency was modest overall . There was a poor fit of project objectives and design with the epidemiology of HIV/AIDS and the sociology and economics of behaviors particular to Ghana . As a consequence, there was insufficient targeting and prioritization of activities around : high risk groups, high-risk regions, high-risk behaviors, and places of high-risk activity or high vulnerability within a given geographic area (e.g., ports, borders, highway stops, towns with important migrant populations, young girls selling out of town ). The efficiency of activities was also low . For example, much effort and expenditure was devoted to information, education and communication (IEC) activities, although awareness was already very high . More attention to behavior change communication (BCC) would have enhanced the impact of these investments . The failure to build on existing efforts, institutions, and initiatives created inefficiencies in implementation, as : (a) a new institution and new staff took up their responsibilities; and (b) the MoH, which had lead and continues to play a critical role, was alienated. As a result, the project lost the technical support of the institution responsible for assuring technical quality of the national response . The push to expand rapidly the national response seems to have come at the expense of quality and impact of the 3000+ subprojects, some of which were carried out by inexperienced civil society and public sector organizations, and none of which were evaluated . 6. M&E Design, Implementation, & Utilization: Design . The results chain (inputs, outputs, outcomes, impacts ) was not clearly articulated or coherent . Some indicators were not appropriate measures of project success (e.g., trends in HIV prevalence as a measure of prevention success), while other critical indicators (that would have measured the program ’s intent to target certain groups, in line with comparative advantages of the various actors ) were missing. For example, the size of key target populations and the number of people reached within each group were not tracked, either for prevention or care . Even though nearly 90% of financing was to support subproject implementation, there was no provision for assessing the performance and outcomes of non -health public sector and civil society activities, even on a random basis . Roles and responsibilities for M&E were not clearly articulated and incentives for evidence -based (as opposed to disbursement-based) implementation were missing. Goals of "mainstreaming HIV/AIDS" in different sectors and "achieving a multisectoral response " were not sufficiently defined, nor were indicators chosen that would facilitate the tracking of progress in these areas . Implementation . The project logframe and indicators were not revised during implementation and a number were not adequately tracked. A National HIV/AIDS monitoring and evaluation plan was developed towards the end of the project. Nevertheless, GAC failed to systematically document relevant information and collate statistics to assess GARFUND’s achievements, provide feedback to those providing the data (regions, districts and civil society organizations), or take timely decisions to improve project performance . Persistence of parallel M&E reporting systems undermined efforts to improve knowledge about the response at various levels . The knowledge, behavior, and HIV status of high-risk groups was not adequately tracked . Planned behavioral surveys were not carried out . AIDS cases continue to be under -reported. Utilization . There is little evidence that available data were used for decision -making. This is due in large part to the absence of program data that would facilitate the tracking of program /project performance and impact (e.g., trends in behavior change in target groups ). The Region has highlighted to IEG a major effort, which is underway by the Region, to restructure project development objectives and key performance indicators of all MAPS in acknowledgement of the fact that HIV prevalence is not a good measure of program /project success. 7. Other (Safeguards, Fiduciary, Unintended Impacts--Positive & Negative): 8. Ratings : ICR ICR Review Reason for Disagreement /Comments Outcome : Satisfactory Unsatisfactory Relevance, efficacy and efficiency are all rated as modest . Insufficient data on program outcomes and project outputs make it difficult to assess project performance. Institutional Dev .: Substantial Modest The GAC Secretariat improved the capacity to engage large numbers of CSOs, but underestimated the task of ensuring efficient use of project funds, based on evidence, and guiding implementers to deliver on agreed targets . M&E was not adequately undertaken . Effective collaboration with MoH is still lacking at central and decentralized levels, although this is now modestly improving. Sustainability : Likely Unlikely While Ghana's HIV/AIDS program is accorded priority by government and civil society, the sustainability of key elements of the GARFUND approach is not evident, particularly with respect to: community- based care, in the absence of MoH involvement, referral, or support; continuation of over 3000 civil society activities, especially for care and mitigation; and the activities of non -health sector Ministries, many of which did not allocate budget for counterpart financing . Bank Performance : Satisfactory Unsatisfactory The project design was inappropriate to the context and there was insufficient focus on results (development objectives and M&E) throughout implementation. The MTR was a missed opportunity for correcting an overemphasis on process and disbursements in favor of a focus on results. Borrower Perf .: Satisfactory Satisfactory After initial delays, due in part to a change in government, the Borrower implemented the project in 3 1/2 years and succeeded in engaging a broad range of actors for a multisectoral response to HIV/AIDS. Shortcomings included: weaknesses in tracking the financial and technical performance of implementers; weak M&E, including failure to carry out behavior surveys; and low collaboration/coordination with MoH. Quality of ICR : Unsatisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness . 9. Lessons: The MAP "learning by doing strategy" cannot be realized in the absence of well -designed monitoring and evaluation activities and the incentives to implement them . There is an urgent need to strengthen the focus on strategic management of the epidemic and tailor the multisectoral approach to Ghana ’s concentrated epidemic. HIV prevalence, an indicator of the scope of the epidemic, should not be considered a measure of success of prevention efforts. Design of an HIV/AIDS project without MoH involvement and mandate can undermine both process and results . The financing of salaries through donor funds should be considered with caution where it concerns the financing of institutions that need continued and sustainable financing beyond the scope of projects . 10. Assessment Recommended? Yes No Why? A Project Performance Assessment Report (PPAR) is under preparation, which will incorporate findings from an IEG mission which visited Ghana in June 2006. 11. Comments on Quality of ICR: The ICR makes a strong effort to assess project performance, albeit with very limited data and deficient indicators . It documents convincingly that : project accomplishments are more process - than results-oriented; attribution is difficult to establish; and shortfalls in design and implementation are significant . The lessons learned provide a good distillation of experience, including a clear assessment of the inadequacies of HIV prevalence as a key performance indicator. However, the ICR does not adequately substantiate its generally favorable ratings of performance and outcome . For the most part it cites changes in the period 1998-2003, while the project became effective in mid -2002 and ended in 2005. Further, even after reception of revised statics prepared by the ICR team, IEG still notes inaccurate interpretation of DHS data. First, the ICR team documents that the percent of men in union using condoms at last intercourse rose from 15% to 39% (initially reported in the ICR as 45%) and for women from 6% to 15% (initially reported in the ICR as 35%) (DHS: 1998, 2003). Even with corrections, these data cannot be compared for the purposes of trend analysis, since the 1998 figures document general condom use and the 2003 figures document condom use at last high risk sex among those engaging in high risk sex . Second, the trends cited in knowledge of HIV/AIDS prevention are not comparable since 1998 data are derived from open-ended questions, while the 2003 data are derived from prompted questions . In addition to these issues, data on costs and financing are not internally consistent within the ICR (see Section 2.c).