ICRR 13536 Report Number : ICRR13536 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 03/16/2011 PROJ ID : P100935 Appraisal Actual Project Name : Avian Influenza US$M ): Project Costs (US$M): 13.00 3.70 Control And Human Pandemic Preparedness And Response Project Country : Afghanistan Loan /Credit (US$M): Loan/ US$M ): 8.00 3.20 Sector Board : ARD US$M): Cofinancing (US$M ): Sector (s): General public administration sector (68%) General agriculture fishing and forestry sector (16%) Health (12%) Solid waste management (3%) Other social services (1%) Theme (s): Natural disaster management (25% - P) Other communicable diseases (25% - P) Rural services and infrastructure (24% - P) Pollution management and environmental health (13% - S) Health system performance (13% - S) L/C Number : CH267 Board Approval Date : 01/11/2007 Partners involved : Closing Date : 03/31/2010 03/31/2010 Evaluator : Panel Reviewer : Group Manager : Group : Ridley Nelson Robert Mark Lacey IEG ICR Review 1 IEGPS1 2. Project Objectives and Components: a. Objectives: The objective of the project as stated in the PAD was to minimize the threat posed by HPAI * to the poultry sector and prepare for possible spread of avian influenza to humans, through necessary short and medium term measures to identify, control and mitigate the effects of the panzootic . The legal agreement has the same language . In November 2009, following an H1N1 outbreak, the PDO was "clarified" to allow support for H1N1 related activities. The language became, " ... to minimize the threat posed to humans by HPAI and other zoonoses and to prepare for, control, and respond to influenza pandemics and other infectious disease emergencies in humans ." Apart from adding the additional threat of "other zoonoses", the original objective words, "through ... short and medium term measures to identify, control and mitigate the effects of the panzootic " became, "to prepare for, control, and respond to influenza pandemics .." and it broadens the objective to "other infectious disease emergencies ." Thus, "identify" in the original becomes "prepare for", the word "control" remains the same in both and the word "mitigate" in the original becomes "respond to" in the later version. This ICR Review uses the latter version because, while the overall intent appears to be the same, the language "to prepare for, control and respond to " is more monitorable. * Note: HPAI = Highly Pathogenic Avian Influenza; H 5N1 is a sub-type of HPAI; H1N1 is another sub-type. New sub-types can spread quickly due to low population immunity . b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? No c. Components (or Key Conditions in the case of DPLs, as appropriate): There were the following four components . A. Animal Health . (Appraisal US$6.2 million, Actual US$1.1 million). This included strengthening HPAI control and outbreak containment capacity; human safety and assurance; strengthening HPAI clinical and laboratory diagnoses and surveillance for rapid and effective outbreak detection and containment capacity; improving the capacity of poultry stakeholders to adopt appropriate measures to counter HPAI presence; compensation facilities including procedures and manuals and the associated processes; technical assistance using international consultants; and waste management and sanitation including a national plan and standard guidelines and operational manual . B. Human Health . (Appraisal US$3.4 million, Actual US$1.0 million). This included enhancing public health program planning and coordination; strengthening the national public health surveillance system; and strengthening the health care system response capacity . C. Communication and Public Awareness . (Appraisal US$1.4 million, Actual US$0.1 million). This included extensive communication using media; and intensive communication using women staffed response teams including training at schools and other institutions . D. Project Management . (Appraisal US$1.4 million, Actual US$1.6 million). This included strengthening of the national policy and regulatory environment; and project coordination using the National Avian Influenza Committee (NAIC) and M&E. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Actual project costs were somewhat less than one third of the total costs planned at appraisal . In October 2007 there was a suspected outbreak of H 5NI in one district. The government requested a waiver of all the remaining conditions of disbursement, in particular the completion of key staffing . The Bank allowed disbursements to proceed on condition that staffing would be completed by the end of January 2008. In June 2009 the Financing Agreement was amended to clarify that taxes would be covered by the IDA grant and to create a disbursement category for financing taxes under the Grant . In November 2009, following an H1N1 outbreak, the PDO was clarified to allow support for H1N1 activities and in March 2010 an amount equivalent to US$4 million was canceled and made available for recommitment within the Afghanistan portfolio before June 30, 2011. Bank management did not approve an extension sought by the government so the project closed on time . 3. Relevance of Objectives & Design: Relevance of Objectives . Rated Substantial . Shortly prior to the project, WHO had warned of the risk of a pandemic in which millions of people could die . (Global influenza pandemics have occurred about an average of every 15 years.) The interventions proposed under the project were consistent with the FAO /World Organization of Animal Health/WHO Global Strategy for the Prevention and Control of HPAI . As part of the world-wide response, the Bank developed a Global Program for Avian Influenza Control and Human Pandemic Preparedness Response (GPAI). A multi-donor trust fund was established . At the time, Afghanistan's three main trading partners, China, Iran and Pakistan, all had confirmed cases of H 5N1. With substantial poultry imports into Afghanistan any outbreak in a neighboring country could easily spread, particularly given the porous nature of borders . The global public good aspect was important for the case of Afghanistan . Afghanistan lies on three major flyways for wild birds that can transmit disease, the Central Asian Flyway, the Black Sea /Mediterranean flyway, and the East Africa /West Asia flyway. Countries along these flyways had been infected with H 5N1. As of May 4, 2006, FAO had reported 26 confirmed cases of H5N1 in 4 provinces of Afghanistan. New provinces had been infected in quick succession from the first case in Nanghar. The ICR argues that the project objectives remained valid throughout project implementation. Given the unexpected and emergency nature of the challenge, it would be unreasonable to expect that the Bank CAS or country strategies would have included this as a priority . However, it was clearly deemed at the time a serious global and national threat . Relevance of Design . Rated Negligible . The ICR does not discuss the relevance of design in the assessment of outcomes, but it is the key issue with regard to this project . Discussion with Bank regional staff confirmed that appraisal used a template approach that did not take sufficient account of the situation in Afghanistan . The design was unrealistic for the country context given the capacity weaknesses, the opportunity cost of the resources, and the poor and uncertain security . In the event, it appears that the human and financial constraints did, indeed, impact seriously on implementation. 4. Achievement of Objectives (Efficacy): Achievement of the stated objective "to minimize the threat posed to humans by HPAI and other zoonoses and to prepare for, control, and respond to influenza pandemics and other infectious disease emergencies in humans ." was modest . On the positive side, there was some preparedness evident in : (i) revised veterinary and poultry sector legislation and regulations; (ii) manuals, contingency plans and containment procedures; (iii) training for staff in 8 zonal offices and training for 4,000 health workers; and (iv) establishment of an interministerial working group (which carried out outreach activities during the 2009 H1N1 outbreak, successfully according to the Region ). And according to the ICR, despite the absence of HPAI outbreaks since 2007, the project sustained awareness and adoption of recommended practices among the target population . But a beneficiary survey in July 2010 found the adoption of the five safe practices (keeping species separate; keeping sick birds separate; disinfecting regularly; quarantining new birds; reporting sick birds) was 70.3% compared to 73.1% at the baseline suggesting that awareness had only been sustained. Since this level of awareness shows virtually no change, it is difficult to attribute any improvement to the project unless it can be assumed that the "without project" counterfactual would have been a decline . On the negative side, this review finds more achievement in what might be termed the documentary aspects of preparedness e.g. manuals, plans, regulations, and procedures, than in the actual surveillance capability, both in the animal and human health areas. On the animal health component, less than 20 percent of allocated funds were disbursed. There were some modest achievements in the supply of kits and reagants and other inputs and in training capacity but "the planned surveillance, which was the main thrust of the component was not implemented and field simulation exercises could not be carried out before project closing " (ICR p. 9). Veterinary Field Units were not contracted to deliver training at the provincial level so field level skills remained limited . The ICR found that the opportunity was partly lost to establish more capacity through referral centers and to increase the surveillance build-up capacity. On human health , only 25 percent of allocated funds for the component were disbursed . WHO was never contracted due to a disagreement with the Bank's contractual clauses, in particular the Bank's right to audit . There were consequent weaknesses in the surveillance system, the diagnostic laboratory, rapid response teams, protective equipment and medication, and guidelines . Only one isolation facility was developed in Kabul but none in the four regional centers. The coordinator at the Ministry of Public Health was not a full -time avian flu coordinator and had other tasks. There was lack of familiarity with Bank procurement that resulted in delays . However, communication related to human health aspects was somewhat better . The ICR claims a high level of awareness in Kabul and Nangahar (at 94%). A national strategy was developed and communication manuals prepared . The plan was field tested during the H1N1 outbreak although this was a relatively modest event compared to a likely full Avian Flu threat including the bird component. But there was an intensive flu prevention campaign . Training was not possible in the highest risk border areas due to security problems . While, as noted above, an interministerial working group was formed, the planned National Avian Influenza Committee, that was supposed to facilitate coordination, never convened for the entire project (ICR p. 15). With regard to the control and mitigation of the panzootic, there was, in the end, no need for response or control since no pandemic occurred . However, this objective is interpreted for this project, a project that was designed for a moderate to low probability but a very high potential cost event, as asking the extent to which the project might be expected to respond to and control a pandemic should one occur . In rating this there is an important dimension of sustainability because the intent was to set up a system that would be ready for the eventuality not simply over the project period but into the future . The rating of this must partly reflect the modest assessment of the the preparedness element discussed above . But to this must be added the concern expressed in the ICR of declining ownership as the threat receded . While the capacity to respond in future does not appear to be negligible partly due to other on-going project support, it is clearly quite limited given the capacity, security problems and level of challenge. Overall, this review finds major weaknesses . 5. Efficiency (not applicable to DPLs): The fact that no pandemic actually occurred cannot be applied to the ex post analysis to assume zero benefits since the decision to proceed with the project was a calculated risk based on a moderate to low probability but high cost event. No ERR was calculated by the ICR since no pandemic occurred . However an expected ERR was estimated at appraisal ranging from 15% to 86% depending on assumptions with a base case of 37%. The analysis includes valuations with probabilities of different levels of protection . Human life is valued only in income terms and therefore is an under-estimate in social terms. As might be expected, by far the highest benefit is the reduction in lost lives. The next highest benefit value is the avoided costs of hospitalization . There is no inclusion of the benefits outside the national boundary through disease transmission to other countries or globally . In rating Efficiency the modest rating of Efficacy is taken into account since it implies only limited preparedness and hence a significant risk that the benefits assumed in the appraisal ERR base case would not have been realized should there have been a pandemic . All ERR scenarios analyzed include an assumption of a 50% level of protection and containment. Given the security situation now, the correlation between locations of disease threat and locations of security problems, the weak capacity, and questions about sustained commitment, it is difficult to see how such a high level containment as 50% would be achieved. On balance, therefore, Efficiency is rated Modest but there are considerable uncertainties. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 37% 100% ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: While the relevance of the project's disease control objectives remain substantial due to Afghanistan's location on major bird flyways, there were major weaknesses in design that paid little or no attention to the country's level of capacity or security situation (See Section 3). There was more achievement in the documentary aspects of preparedness (e.g. manuals, plans, regulations, and procedures ) than in the actual surveillance capacity, which, in the end, any pandemic response would have to depend on, both in the animal and human health areas . In these latter areas, there was limited progress and only about one quarter of the planned disbursements took place (See Section 4). Efficiency is rated modest (See Section 5). a. Outcome Rating : Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Continuing conflict leads to enormous uncertainties and implies a very high opportunity cost to skills and government management attention. The lack of realism in design and weak capacity is also conducive to high risk . a. Risk to Development Outcome Rating : High 8. Assessment of Bank Performance: Quality at Entry . As discussed under the Relevance of Design section and confirmed in discussion with Regional staff, there was clearly a substantial lack of realism about what could be achieved in this conflict ridden country. Design was far too reliant on a template applied elsewhere . There was no mention of the possible impact of the security situation in the Technical Annex (the appraisal document). In the appraisal discussions, the Bank team agreed to follow the standard approach of the Global Program for Avian Influenza despite the fact that Afghanistan has minimal capacity and major security issues for field work . Thus, preparation, instead of starting from a position that assessed what might be achieved given the wide array of constraints was based from the outset on an unrealistic model. In addition, contractual issues with other agencies were not resolved during preparation. Supervision . As is noted by the ICR (p. 23), "during supervision the Bank team was faced with the daunting task of assisting two weak agencies ... in implementing a complex project that was not ready for implementation ." They might have added "in a country facing major conflict ." The ICR argues that supervision made enormous efforts to get implementation moving. However, missions, particularly more recently, faced severe restrictions on field visits due to security problems . The ICR notes that the Bank team gave hands -on assistance in working out processes and procedures for transferring funds to the field . Overall, they were clearly faced with a very difficult task. at -Entry :Highly Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: Government Performance . The ICR notes that the Ministry of Finance, as the overall government supervising agency, maintained support throughout the life of the project . However, the coordination committee that reached beyond just the two main implementing agencies was also "government" and this committee never met. It must be acknowledged, however, that, for government, as for all other players, the challenges were enormous . Nevertheless, Government must take some responsibility for design weaknesses . Implementing Agency Performance . Overall project management was weak . The National Avian Influenza Committee that was to facilitate the overall coordination never met, partly due to lack of oversight and formulation of an agenda for consideration . With no outbreak, the level of ownership declined . There were two implementing agencies, the Ministry of Agriculture, Irrigation and Livestock (MAIL) and the Ministry of Public Health (MOPH). Neither performed well but the former was much weaker than the latter . In particular, the ICR notes that insufficient leadership was provided by the General Directorate of Animal Health and Production . There were substantial delays in implementation, procurement and staffing . Most issues were not addressed in a timely manner. The commitment of MAIL dwindled as the threat of a pandemic faded . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Moderately Unsatisfactory c. Overall Borrower Performance :Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: Design . The ICR reports that a results framework was designed with key project indicators developed . These are shown in the Technical Annex p . 35. While the results framework covered most of the main elements of the objectives it was weak on detail . Implementation . UNICEF, with own financing, conducted a baseline survey on knowledge, attitude and practice . For the human health component, M&E was carried out by Johns Hopkins University arranged through MOPH but there is no evidence in the ICR to assess the quality of this work . For animal health, M&E was weak mainly due to delays in staffing. According to the ICR, M&E improved towards the end . Outside the project, a third party evaluation of the primary and hospital sectors is currently being undertaken by Johns Hopkins, including a survey of health facilities . Utilization . Data collected were used for MAIL's quarterly progress report . Beyond that there is no evidence on how the data helped any project redirection . a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards . The project was classified as a Category B for the purposes of OP 4.01. An Environment Management plan was prepared, translated and disseminated in project areas . Waste management guidelines were prepared but were not put to the test since there was no outbreak . Fiduciary . Fiduciary safeguards faced the challenge of weak public administration . Accounting, statements, payments and auditing were centralized within MOF . The Public Financial Management measurement framework was assessed in 2007 and found to be better than average for low income countries . The ICR reports that supporting documents for all payments were maintained and that financial reporting improved over time . Audits were unqualified although two reports for IDA and the TF were submitted late in 2009. Furthermore, according to the ICR, there was some ineligible expenditure, totaling about US$ 125,000, but the largest part was simply a payment found to be from the wrong fund. 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Unsatisfactory There was more achievement in the Unsatisfactory documentary aspects of preparedness (e.g. manuals, plans, regulations, and procedures) than in the actual surveillance capacity, which, in the end, any pandemic response would have to depend on, both in the animal and human health areas. In these latter areas, there was limited progress and only about one quarter of the planned disbursements took place (See Section 4). Efficiency is rated modest (See Section 5), and design relevance negligible (See Section 3). Risk to Development High High Outcome : Bank Performance : Moderately Moderately Unsatisfactory Unsatisfactory Borrower Performance : Moderately Moderately Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The three main lessons are as follows : 1. In a country situation with major security and capacity issues, the starting point for project design should be the country constraints and an assessment of what is possible . An imposed supply-driven standard design package, while potentially offering some cost saving for a global initiative, is wholly unsuited to unusual or highly challenging circumstances. 2. Contractual issues with other international agencies need to be resolved during preparation . 3. In certain exceptional circumstances and conflict situations, where there is weak capacity, there may be advantages for speed of implementation to have donor financing channeled outside the government budget although funding through government can help strengthen systems for the longer term . Since the Bank cannot normally do this, key components on the critical path may be better handled by those agencies that can . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: On balance, the ICR is frank and informative about the issues and offers a lot of information about implementation . The main weakness is that the it is not critical enough of the weak project design, although it touches on this question in relation to Quality at Entry. The ICR is too focused on component achievements rather than the achievement of objectives although there were data limitations in measuring the key preparedness objective . There could have been more attention to sustainability and ownership and the likely impact of this on efficiency . a.Quality of ICR Rating : Satisfactory