ICRR 12437 Report Number : ICRR12437 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 08/18/2006 PROJ ID :P000825 Appraisal Actual Project Name :Part. Health/pop./nutrition. Project Costs 19.9 20.0 US$M ) (US$M) Country :Gambia Loan/ US$M ) Loan /Credit (US$M) 18.0 19.2 Sector (s):Board: ): HE - Health (99%), US$M ) Cofinancing (US$M) 0 0 Central government administration (1%) L/C Number :C3054 FY ) Board Approval (FY) 98 Partners involved : Closing Date 12/31/2003 06/30/2005 Evaluator : Panel Reviewer : Division Manager : Division : Denise A. VaillancourtKris Hallberg Alain A. Barbu IEGSG 2. Project Objectives and Components a. Objectives The overall project development objective (PDO) is to improve family health in The Gambia. This is defined in The Gambia as a combination of reproductive health, infant /child health, and good nutritional status . Specific project objectives are improved quality of : (a) reproductive health services; (b) infant and child health services; (c) nutrition services for women of reproductive age, infants and children; and (d) management and implementation of a family health program. Progress toward these objectives would be principally measured through indicators on access to, and quality of, family health services . While the PDO was not changed, the indicator on maternal malnutrition was dropped at the mid -term review, as it was considered to be too difficult to measure . b. Components (or Key Conditions in the case of Adjustment Loans ): A. Reproductive Health Services (estimated cost including contingencies : $4.7 million or 23% of total estimated cost; actual cost: $5.3 million or 26% of total cost). This component was designed to support : (i) improvements in maternal health services (coverage, quality, utilization, risk mitigation, and improved knowledge base about maternal mortality); (ii) the prevention of unwanted pregancies by improving supply and demand for family planning services through social marketing; and (iii) sexually-transmitted infection (STI)/Human Immunodeficiency Virus (HIV) prevention and control through extensive information, education, communication (IEC) to increase knowledge, change attitudes and improve preventive and care seeking behaviors . Project support includes financing of training, equipment, essential inputs and supplies . B. Integrated Management of Childhood Illnesses (estimated cost including contingencies : $0.9 million or 5% of total estimated cost; actual cost : $1.2 million or 6% of total cost). This component was designed to support the integration of vertical programs to combat childhood diseases (malaria, acute respiratory infections, diarrhea and malnutrition) through the introduction and implementation of the "Integrated Management of Childhood Illnesses " (IMCI) approach. Project support includes training development and delivery, IEC, outreach, operational research, supervision, monitoring and evaluation . C. Nutrition Policy and Services for Women, Infants and Children (estimated cost including contingencies : $2.3 million or 12% of total estimated cost; actual cost : $2.4 million or 12% of total cost). This component was designed to support: (i) nutrition policy formulation and institutional strengthening (support to policy process and training ); and (ii) community and micronutrient approaches to improve the nutritional status of women and young children (nutrition education, maternal food supplement programs, and micronutrients ). D. Management and Implementation of a Family Health Program (estimated cost including contingencies : $10.7 million or 54% of total estimated cost; actual cost : $10.5 million or 53% of total cost). This component was designed to support: (i) capacity building and policy development, including: cost recovery through the expansion of the Bamako Initiative; information education and communication (IEC); monitoring and evaluation through the establishment of financial and health information systems; health mapping; research application; staff development and retention; development of a financing policy; updating /implementation of national drug action program; development of a maintenance policy; (ii) upgrading and maintaining health infrastructure , including the rehabilitation of selected primary health care (PHC) facilities, the establishment of blood supply system, procurement of equipment and maintenance support; (iii) a Local Initiative Fund (provision of micro-grants on a pilot basis to enable communities to carry out activities in the areas of reproductive health, infant and child health, and nutrition to deal with barriers to health seeking behavior ); and (iv) project management (Project Implementation Unit [PIU] responsible for World Bank, ADB and IDB health projects ). PPF and Unallocated (estimated at $0.8 million and $0.5 million, respectively; actual PPF amount : $0.6 million). c. Comments on Project Cost, Financing, Borrower Contribution, and Dates The difference in planned vs . actual IDA financing, expressed in US$, is largely attributable to changes in the exchange rate. In actual fact, 99 % of the IDA allocation was disbursed (SDR 13.3 million disbursed vs. SDR 13.4 million approved). Government counterpart financing ($0.8 million) amounted to about half of its initial commitment of $1.9 million). Following reorganization of some components after the March 2002 mid-term review (MTR), implementation considerably improved and the project was extended by 18 months (from December 31, 2003 to June 30, 2005) with the aim of meeting the PDO. 3. Relevance of Objectives & Design : The relevance of the project's objective and design is found to be substantial overall. The objectives are relevant to health sector issues documented during the design stage, and commensurate with national sector policy objectives (to improve access and quality of basic health services ) and with CAS objectives (poverty alleviation and human development through health and education ). The design , however, was not fully relevant to health sector issues and challenges . It was complex relative to the limited capacity of the health sector, which was not assessed during preparation . The results chain was not entirely clear. Human resources quality, quantity and management were raised as risks, but not adequately addressed in project design. It did not seem to address Government's stated (at design) intention to decentralize health sector management. Overall quality at entry was assessed by a quality at entry review (QER) as unsatisfactory. This being said, two positive features of the design were : (i) the recognition that health is a product of other factors /sectors, but the limitation of project support to the health sector's contribution to family health; and (ii) the concerted effort to document and respond to beneficiary perspectives and to be participatory both in the design and implementation of project activities. 4. Achievement of Objectives (Efficacy) : The project lacks basic quantitative output and outcome data . Monitoring and evaluation (M&E) and studies to provide specific information on performance and outcome were not executed (see M&E section for details). Analysis of available data shows that overall project efficacy is found to be modest. Main achievements and shortfalls for each of the five project objectives are itemized and rated below . General objective : Achievement of improved family health was substantial overall . The Implementation Completion Report (ICR) documents a 20% reduction in the total fertility rate (TFR) from 6.4 to 5.1 (vs. a target of 10%). However, the 6.4 baseline, used in the ICR (source not cited) is higher than the PAD baseline of 6.0. Even if calculated on the basis of a 6.0 baseline fertility rate, the reduction of 15% still exceeds the 10% target. Actual TFR is drawn from 2003 Census data. This decline is likely attributable in part to higher female education and delays in marriage of younger cohorts of women, but also attributable in part to social marketing and availability of family planning services, supported under the project . The ICR documents a reduction in infant mortality from 92 to 75. The ICR baseline of 92 does not cite a source and is different from the PAD baseline of 80. Even with the PAD baseline, a trend of lowered infant mortality is still documented, although it is very modest, and falls short of the target of 65. Actual infant mortality rate is drawn from 2003 Census data. Data were unavailable to measure the target of a 50% reduction in STI prevalence among pregnant women . Child malnutrition was reduced by 28% (vs. target of 25%), reflecting a decrease from 26% of under fives being underweight to 20% (national nutrition survey 2005). Stunting decreased from 23% in 1996 to 18% in 2005 (national nutrition survey). Specific objectives : #1: Achievement of improved quality of reproductive health services was modest overall . A recent survey by the National Nutrition Agency shows that up to 57% of births were delivered in a health facility. Baseline data are unavailable. The project targeted 69% of deliveries to be carried out by skilled personnel. The 25% of home deliveries assisted by a traditional birth attendant (trained or untrained) are not included in these data. Modern contraceptive prevalence increased from 14% (1998) to 18% (WDR 2005), falling short of the project target of 22%. The positive trend is due in part to social marketing, distribution and supply of condoms and oral contraceptives, including an increase in the number of outlets (302 countrywide). In total 550,800 condoms and 73,260 cycle of pills were sold by the end of the project . The inadequate provision and management of STI drugs during the entire project undermined the syndromic management approach to STI treatment . STI prevention was likely positively affected as 65% of sexually active men and 40% of sexually active women reported using a condom in the last sexual contact with a non -regular partner. (No baseline available.) No data is available on key outputs, notably : training, utilization of prenatal services, referral facilities equipped and staffed to handle obstetric emergencies, knowledge /understanding of harmful effects of female genital mutilation (FGM). #2: Achievement of improved quality of infant and child health services was negligible overall . There is no reliable indicator for the availability of IMCI diagnostic tools and treatment protocols in basic facilities and the extent of the use of such tools by the village health workers . The project could not achieve its overall aim of curbing the behavior of the population with respect to delays in care-seeking due to traditional attitudes, beliefs about factors that cause diseases, referral mobility problems, limited involvement of fathers in facilitating child referrals . #3: Achievement of improved nutrition services for women of reproductive age, infants and children was substantial overall . Project support led to the development of a national nutritional agency, under the office of the vice -president, which has its own budget and recognition as a cross -cutting development issue. A national nutrition policy was formulated and adopted and a national nutrition council chaired by the vice-president was established. Percent of children under 4 months who are exclusively breastfed increased from 36% in 2000 (MICS 2000) to 49% in 2005 (national survey) exceeding the project target of 40%. No data is provided on whether /to what extent strategies on micronutrient deficiencies (to prevent and control iron deficiency, anemia, iodine deficiency disorder and Vitamin A deficiency ) were implemented, and with what outcome. #4: Achievement of improved management and implementation of a family health program was negligible overall . Health financing: national health accounts were not developed, work on a health financing policy never took off and the Bamako Initiative (cost recovery) ended at the pilot stage. None of the Knowledge, Attitude, Practice (KAP) studies were implemented. Health Management Information System (HMIS) infrastracture was put into place, but there is a big turnover of trained staff and the quality and timeliness of data and its use in decision -making are lacking. National drug policy was fully implemented as planned, but the issue of inadequate drug supplies persists . "Riders for Health" transport maintenance contract (agreed during MTR) was perceived as World Bank -driven, and a recent audit found that it was not performing to standard . The project supported the construction of four new (unplanned) health facilities and the rehabilitation of 13 others, but they were neither staffed nor equipped by the end of the project . Local Initiative Fund provided over 200 grants to communities, but only towards the end of the project; and it was never evaluated. 5. Efficiency : Project efficiency was modest overall . The interventions chosen may not have been the most cost -effective means of achieving project objectives . For example, malaria accounts for 60% of childhood deaths, and can be addressed by low-cost interventions such as the use of insecticide bednets, early detection and prompt treatment of malaria cases, and selective vector control . In addition, project implementation suffered from a number of inefficiencies, including: weak capacity of MoH vis-à-vis project complexity; absence of norms and standards for service delivery; an HMIS which is still not fully functional or sustainable; and infrastructure created or rehabilitated, but not equipped or staffed. 6. M&E Design, Implementation, & Utilization: Design . Project M&E was intended to use extensively the integrated computerized monitoring and information systems (for health, financial and maintenance data ) that was to be set up under the project . These systems were to be complemented by an ADB-financed human resources information system . Links among these various system components was expected to facilitate data exchange, both intra - and inter-sectoral. M&E was to be carried out by the line directorates in charge of implementing singular components, and coordination of M&E was given to the PIU . The project also envisaged the support of operations research and surveys and a new initiative to close the gap between research results and policy /program formulation. In short, the design focused too much on complex HMIS hardware and software and too little on quality and use of data by managers and decision -makers at the decentralized levels. Incentives for good M&E were also neglected . Implementation . Data collection, surveys and studies were not carried out as planned, making it very difficult to assess project performance in terms of outputs and outcomes . All qualitative surveys planned under the project, except a nutrition survey, were not done . The evaluation of Baby Friendly Communities and recording of birth weights were not carried out. Nor were the creation and maintenance of a databank for all Baby -Friendly Communities ever done. Use of Data for Decision -Making . Both the paucity and low quality of data made it difficult for data to inform decision-making. 7. Other (Safeguards, Fiduciary, Unintended Impacts--Positive & Negative): 8. Ratings : ICR ICR Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately The ICR rated project outcome as Unsatisfactory marginally satisfactory in its text, but was constrained by the four-point rating system (still in effect during ICR preparation) having to reflect the final outcome rating as either satisfactory or unsatisfactory. IEG's outcome rating is a function of three other ratings : substantial relevance (see Section 3); modest efficacy (see Section 4); and modest efficiency (see Section 5). There were major shortcomings in the achievement of several objectives, especially those related to service quality and outputs and to program management. Institutional Dev .: Modest Modest Sustainability : Unlikely Unlikely Bank Performance : Satisfactory Unsatisfactory Quality at entry was unsatisfactory and during the first 3 1/2 years of supervision no significant action was taken to improve unsatisfactory project implementation . Thanks to a very proactive team that assumed supervision responsibilities at the time of the MTR, there was a significant turnaround in project implementation during the remaining 18 months after the MTR. Disbursements picked-up significantly, project management was strengthened, and many planned activities were carried out, with an 18-month extension in the closing date. The Bank's team also encouraged the launch of a much needed Demographic and Health Survey that was initially resisted by Government. Despite this turnaround in implementation in a difficult country context, an important shortfall in the Bank's post-MTR performance was its lack of follow -up of key QER recommendations, including (among others) the establishment of a simplified M&E system to measure project performance and its attribution to outcomes, and the correction of project design flaws. Borrower Perf .: Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness . 9. Lessons: A project design that is not based on an institutional /organizational/capacity assessment is likely to exceed the capacity to implement it properly. Broad participation of stakeholders in a project's preparation and implementation risks eclipsing the need to focus on key strategic issues that will have the highest impact on health outcomes . Monitoring and evaluation of project /program performance is likely not to be implemented in the absence of : an operational M&E system, jointly agreed with the Borrower; measurable performance indicators; and proper incentives for the collection, analysis, and utilization of data . Nutrition interventions are likely to be successful when (among other things): services are combined with policy and institutional strengthening; nutrition authority is outside, but still linked to, the health sector; and design and implementation are overtly focused on the clients, at the community level . 10. Assessment Recommended? Yes No Why? To verify the ratings and to gain more insight on a number of issues and challenges emanating from this project experience, including : the design of a health operation in a difficult environment; effective monitoring and evaluation; and project impact and sustainability . 11. Comments on Quality of ICR: The ICR was of satisfactory quality overall, making good use of limited data, including the assessment of proxy indicators, where available. It was correct in pointing out (a) the lack of causal relationship between the improvement of some outcome indicators and project outputs; and (b) the lack of basic quantitative output and outcome data . A few minor shortcomings include the following : The ICR uses baseline data on key performance indicators (total fertility, infant mortality) that are different than baseline data used in the PAD, and neglects to clarify these discrepancies or to cite the source of the baseline used in the ICR. Micronutrient activities and performance indicators are not reported on at all, although they are important elements of the nutrition component . Actual costs by component presented in the text (pages 11-13) are different than those presented in Annex 2 on Project Costs and Financing (page 22).