ICRR 13893 Report Number : ICRR13893 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 12/19/2012 Country : Nepal Project ID : P100342 Appraisal Actual Project Name : Avian Influenza US$M ): Project Costs (US$M): 18.20 15.92 Control Project L/C Number : CH268 Loan /Credit (US$M): Loan/ US$M ): 18.20 15.92 Sector Board : Agriculture and Rural US$M): Cofinancing (US$M ): 0 0 Development Cofinanciers : Board Approval Date : 01/19/2007 Closing Date : 07/31/2011 07/31/2011 Sector (s): General public administration sector (60%); Health (21%); Agricultural extension and research (16%); Animal production (2%); Solid waste management (1%) Theme (s): Other communicable diseases (25% - P); Natural disaster management (25% - P); Rural services and infrastructure (24% - P); Pollution management and environmental health (13% - S); Health system performance (13% - S) Prepared by : Reviewed by : ICR Review Group : Coordinator : Stephen Hutton George T. K. Pitman Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: The project was part of the Global Program for Avian Influenza and Human Pandemic Preparedness and Response (GPAI) that was developed to expeditiously support country -specific preparedness using guidelines for accelerated project processing. Country qualifying criteria included : (i) a national strategic plan; (ii) national commitment; (iii) an implementation strategy through concerted action among ministries, donor community, and civil society; and (iv) a clearly defined monitoring and evaluation system . The objectives of the project as listed in the country -specific Technical Annex to the GPAI (page 3) were: "to minimize the threat in Nepal posed to humans by Highly Pathogenic Avian Influenza (HPAI) infection by controlling such infections among birds, especially domestic poultry, and to prepare for, control, and respond to possible human infections, especially an influenza epidemic and related emergencies ." The objectives in the Financial Agreement (page 4) were identical. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: a) Animal Health (appraisal estimate $6.12 million, actual $5.78 million). This was to support national prevention and control strategies for avian influenza through : (i) Strengthening surveillance of HPAI among animals by by training veterinary professionals, villagers and surveyors in disease surveillance, collecting samples from commercial and backyard poultry farms, and establishing a nationwide GIS-based surveillance system . (ii) Strengthening prevention and containment capacity by training veterinary professionals in prevention, control, and biosecurity, and strengthening quarantine services through vehicles and equipment . (iii) Increasing diagnostic capacity by upgrading veterinary laboratories (one to biosecurity level (BSL) 3 and seven to BSL 2), including equipment purchase and training . (iv) Improving field veterinary services, including setup of rapid response teams, provision of equipment, and creation of a vaccine bank . (v) Creating a compensation fund for birds culled during control operations . b) Human Health : (appraisal estimate $6.34 million estimated, actual $5.93 million). This was to support: (i) Improving surveillance of influenza and other influenza -like-illnesses by expanding the network of sentinel sites, improving data collection and handling, providing diagnostic supplies and training for rapid response teams, and supplying equipment, training and upgrades to existing human health laboratories and establishment of a BSL 3 laboratory. (ii) Improving influenza prevention and containment capacity by providing training, protective supplies and vaccines to high risk workers, by acquiring antivirus and vaccine for use in an outbreak, by improving capacity to quarantine patients, by reviewing and updating quarantine regulations, and by devising pandemic response plans. c) Public awareness and Information (appraisal estimate $1.65 million, actual $2.26 million). This was to support: i) improving public awareness and information on avian influenza issues, including raising attention among government, private sector and civil society ii) raising general public attention understanding of pandemic risks, and iii) improving poultry farmer ability to recognize influenza symptoms and to undertake safe disposal and protection. d) Implementation Support and Monitoring and Evaluation (appraisal estimate $1.72 million, actual $1.95 million). This was to support the strengthening of public agencies for the coordination and management of the Project . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project costs : Estimated at US$18.20 million at appraisal, project costs were revised down to US$ 16.84 million. This followed restructuring in March 2010, which canceled construction of biosecurity level 3 laboratories (instead the laboratories were upgraded only to level 2), and January 2011, which reallocated funds originally set aside for compensation to support field veterinary services and enhancement of laboratory capacity . Public awareness expenses were higher than planned because of additional awareness activities carried out during the 2009 influenza pandemic. Financing : The project was financed through an IDA Grant of Special Drawing Rights (SDR) 12.1 million, then equivalent to US$18.20 million. Actual disbursement was SDR 10.13 million, equivalent to US$15.64 million. The undisbursed amount of SDR 1.97 million was cancelled at closing. Borrower : No Borrower contribution was planned or received . Dates : The project closed, as scheduled, on July 31, 2011. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Substantial At appraisal, Nepal faced a serious risk from avian influenza . Outbreaks had been reported in wild birds in China and in poultry in India, and these outbreaks could have spread to Nepal through migration of wild birds or from importation of poultry across a porous border . The poultry sector is an important part of the economy, with an estimated 22 million poultry in the country of which 45% were kept in a few large commercial farms whilst 55% resided in backyard farms mostly in rural areas . Some biosecurity arrangements existed for commercial farms, but biosecurity was weak for backyard farms . Outbreaks among poultry would have significant economic costs, and could potentially infect humans . There was a global concern about the possibility of an influenza pandemic - while the probability of a pandemic was low, the costs could be severe . Prior to the project, preparedness levels were generally low . The department of livestock services had a veterinary service infrastructure, but one that was insufficiently prepared for avian influenza . Animal health monitoring and surveillance capacity was low - and nonexistent for backyard farms - and little capacity existed to respond to contain disease outbreaks among animals . No domestic diagnostic capacity existed . A functioning health sector existed (though constrained by quality of care and access ), but lacked sufficient equipment and training to respond to a major crisis. An inter-sector task force and a national avian influenza and influenza pandemic preparedness and response plan existed prior to the project . Project objectives were consistent with Bank practice of responding to emergencies, and were broadly relevant to the Nepal Interim Strategy Notes for FY 08-11 supporting "promoting access to better quality services " by improving public health service delivery . b. Relevance of Design: Substantial The project followed the broad design of the Global Program on Avian Influenza template, combining animal health, human health and awareness raising components . The design addressed several important aspects of reducing the risk of avian influenza, including surveillance, diagnosis, quarantine, outbreak control, compensation, and pandemic preparedness. The design of the animal health component was focused largely on identifying and containing outbreaks, rather than on improving biosecurity measures which might reduce the probability of an outbreak occurring. Filling this gap in the design would have been an important factor in achieving the objective to "minimize the threat posed to humans" by avian influenza. There was a disconnect between the emergency response nature of the project and the complicated procurement and civil works activities included in the project design . These complications led to implementation delays and difficulty in completing activities prior to project closure, which weakened the ability of the project to accomplish its objectives. The component to upgrade two laboratories to biosecurity level 3 was probably not needed, as level 2 laboratories would be sufficient for the initial diagnostic results needed to initiate stamping out operations . 4. Achievement of Objectives (Efficacy): The assessment of each of the two objectives : "to minimize the threat in Nepal posed to humans by Highly Pathogenic Avian Influenza (HPAI) infection by controlling such infections among birds, especially domestic poultry, and to prepare for, control, and respond to possible human infections, especially an influenza epidemic and related emergencies, " is presented below: Objective 1: Minimize threat to Nepal posed to humans by HPAI infection by controlling such infections among birds, especially domestic poultry : Substantial Outputs : Stamping-out operations were conducted, leading to culling of roughly 30,000 birds. Farmers were compensated for culled birds (with total payments of roughly $US 41,000), with most compensation payments occurring immediately . There was a delay in compensation for the first outbreak in Jhapa district due to disputes over compensation rates . Commercial farmers believed the compensation rate was too low and refused to allow their flocks to be culled until the rate was increased. Compensation rates were increased, but remained low for ducks, which may have inhibited reporting of outbreaks. The compensation system did not include transparency or grievance mechanisms . The originally planned quarantine border checkpost improvements were dropped because it was perceived that border stations would be ineffective due to the porous nature of the border . Instead the project supported strengthening of regional quarantine offices through purchases of vehicles and equipment. Surveillance improvements were carried out in 26 high-risk districts, exceeding the originally planned 12 districts. A total of 50,000 samples were collected, greatly exceeding the planned 28,800 samples. This surveillance program monitored 95% of commercial farmers and 25% of backyard farmers, though no details were available on the frequency of this surveillance . Vehicle purchases enhanced surveillance capacity . 86% of early warning system sites regularly submitted reports to the department of health services . Rapid response capacity was enhanced by an unquantified amount . Rapid response test results at the central veterinary laboratory were 90% consistent with World Organization for Animal Health tests, and quality control tests at the National Public Health Laboratory were 100% consistent. A communications program improved knowledge about animal handling, which could plausibly reduce the risk of transmission from birds to humans. 183,000 students and teachers received training, as did 64 civil society organizations, and 1,703 security personnel, traders, and farmers . Knowledge awareness and practice surveys recorded higher levels of awareness and safe behavior in areas the project targeted as compared to control areas not targeted by the project . General awareness of avian influenza increased from 61% to 82%, the proportion of children reporting that they washed their hands after touching birds increased from 55% to 94%, and the proportion of adults reporting knowledge of basic safety measures when handling sick birds increased by roughly 60 percentage points. Outcomes : Ten outbreaks of avian influenza among poultry occurred during the project and all were contained . There were no documented cases among humans . While attribution is difficult, the lack of capacity prior to the project makes it plausible that outbreaks would have been more severe without the project interventions . The project significantly improved the capacity to identify, respond to, and contain influenza outbreaks among poultry. But the project had only a modest impact on improving poultry biosecurity, and thus reducing the probability of an outbreak occurring. The department of livestock responded to, and successfully contained, an avian influenza outbreak in November 2011, after project closure. Objective 2: Prepare for, control and respond to possible human infections, especially an influenza epidemic and related emergencies : Substantial Outputs : Animal health standard operating procedures were developed for outbreak containment and compensation, surveillance, communications and others procedures, and these were refined based on observations from controlling outbreaks among birds. Legal frameworks for outbreaks control among animals and humans were prepared . Human health guidelines were prepared for surveillance, laboratory management, infection control, case management, and risk communication, and these were refined based on experience from the 2009-10 influenza pandemic. Human health influenza surveillance was expanded, with early warning systems established at seven additional sites, bringing the total up to 35 sites, of which 32 report regularly. The biosecurity level 3 laboratories were not achieved . By July 2009, the cost of upgrading to level 3 was identified as 84% higher than estimated at appraisal, it was recognized that maintenance and running costs would be $ 300,000 per year, and it became apparent that the laboratories could not be constructed by project closure, due to high global demand for scarce resources . The level 3 laboratory upgrade was canceled in a project restructuring in March 2010, and instead laboratories were designed to be upgraded to BSL 2. Nine laboratories were upgraded to BSL 2, but the BSL2 at the national public health laboratory was not fully operational by project closure due to lack of training and staffing . Although 5 intensive care units, 4 high- cost isolation wards and 9 least-cost isolation wards were established, these facilities were not fully operational at two sites because of inadequate staffing . With support from the WHO, the project provided protective equipment and antiviral treatments to animal and human health workers in 26 high-risk districts. During the 2009 H1N1 influenza pandemic, the department of health set up travel advisory desks that screened travelers and identified 39 suspected cases out of 337,563 travelers screened. Outcomes The outcome of these steps is largely unobservable, but it is plausible that surveillance, diagnostic capacity and isolation capacity have substantially improved influenza preparedness . The 2009 pandemic demonstrated the existence of capacity to follow operating procedures and to carry out the epidemic response plan. However, no evidence was available on the impact of these procedures or the travel screening on the number of people infected by the pandemic . 5. Efficiency: Modest No economic analysis was conducted at appraisal . The ICR includes an ex -post economic analysis that estimates the economic losses in the poultry industry if outbreaks had not been contained . The analysis notes that 0.3% of the poultry population was infected or culled during the project period, and assumes that 17% of the poultry population would have been lost had outbreaks not been contained (based on experience from Vietnam ), and assumes that outbreaks would not have been contained had the project not been implemented . This leads to an economic rate of return of 311% in a base scenario, but this is highly speculative . The analysis also notes unquantified benefits from prevented human cases of avian influenza . Project efficiency was improved by the decision to abandon the level 3 laboratory upgrades and focus instead on level 2 upgrades, as the benefits of the level 3 laboratories would not have justified the costs . Project implementation was initially slow, largely due to delayed procurement by UN partner agencies and by delays in staffing key positions in the health and livestock departments . UN agency involvement did not begin until roughly one year after project effectiveness, and few activities were implemented until mid 2008. High staff turnover slowed project implementation. Weaknesses in procurement and financial management led to further delays . The project closed on time, but delays meant that some activities had not been completed by project closure, and some facilities were not operational. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate Yes 311% 60% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Relevance of the objectives and design are rated substantial . The achievement of the objectives (efficacy) is rated substantial. Though there were some problems with project implementation, the project was successful in strengthening outbreak containment capacity to the point where all outbreaks were contained . Efficiency is rated modest due to weaknesses and delays in implementation . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR reports (page 9) that the government remains committed to sustaining key prevention and control activities, and allocated nearly $ 1 million in budget to the livestock and health departments for this purpose . The department of livestock responded to, and successfully contained, an avian influenza outbreak in November 2011, after project closure. Follow-up projects by the Bank (particularly the Zoonoses Control Project, approved in April 2012) may serve to further strengthen capacity to manage zoonoses . However, there is some risk that attention will decline if there are no further outbreaks . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The project was prepared fairly rapidly (moving from concept to appraisal in 8 months), following the blueprints of the global program on avian influenza design . Rapid preparation and a limited budget for preparation may have contributed to weaknesses in the M&E system, which focused on achievement of outputs rather than on intermediate or final outcomes and provided few opportunities to improve project implementation (see section 10). The decision to include laboratory upgrades to create two level 3 laboratories was arguably unwise . Basic avian influenza diagnosis would have been possible within a level 2 laboratory. While a level 3 laboratory would have offered additional benefits by offering the ability to confirm diagnoses within the country (which would be valuable during a pandemic), there was no clear case for separate laboratories for human and animal health sectors . There was some indication that the level 3 upgrades might be infeasible and uneconomic even prior to appraisal, and this turned out to be the case . The Bank team raised concerns during preparation about the feasibility, costs and staffing constraints for establishing biosecurity level 3 laboratories, but acquiesced to stakeholders who argued that the laboratories were justified to strengthen capacity beyond avian influenza and to provide in-country diagnostic confirmation capacity . The range of investments identified at appraisal reduced the threat from avian influenza by improving the ability to identify, contain and respond to outbreaks, but did not include measures to reduce the probability of outbreaks by improving biosecurity. Procurement arrangements were cumbersome and ill -suited to an emergency project . The project may have been more effective in supporting wider health sector goals if the project had been designed to also support disease control beyond avian influenza. at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: The ICR reports (page 15) that the Nepal-based Bank team provided substantial support on financial management, procurement and safeguards . When it became obvious that the BSL 3 laboratories could not be justified, the Bank successfully restructured the project to proceed with level 2 laboratories instead. But negotiating this cancellation took a great deal of Bank time, and this may have led to neglect of other responsibilities, such as supporting adequate policy dialogue in development of long-term disease prevention and control mechanisms . In June 2010 the government requested an additional US$ 1.6 million from the Avian and Human Influenza Facility trust fund to enhance veterinary response capacity; the Bank supported this provided that project implementation and disbursement were accelerated, but this acceleration was not achieved . The ICR reports (page 16) that the Bank team worked effectively with other partners during implementation, and that a joint approach to supervision facilitated project restructuring . Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: Because of the multiple avian influenza outbreaks in Nepal, the government displayed ongoing ownership and commitment to project objectives, as demonstrated by specific budget allocations for avian influenza prevention and control. The government insisted on including biosecurity level 3 laboratories in the design, and initially resisted efforts to cancel these upgrades, but later concurred that the upgrades would be financially infeasible. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance: The project was jointly implemented by the Department of Livestock Services and the Department of Health Services at the national level, and by district counterparts of these departments . The Department of Livestock Services displayed a high level of ownership in the project and offered consistent leadership, while the Department of Health services had frequent turnover in the project coordinator position . The implementing agencies demonstrated a significant degree of coordination at the district level in responding to avian influenza outbreaks and in working with partner agencies . But implementing agency performance suffered due to gaps in staffing and low initial capacity in project management, financial management, and procurement . M&E performance was weak and fiduciary problems affected disbursements (see sections 10 and 11). Additional financing was not feasible because of an inability to accelerate project implementation . Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The M&E system was based heavily on the Global Program on Avian Influenza blueprint . The framework in the original design suffered from a number of weaknesses, some of which would have been difficult to avoid given the inherent challenges in monitoring a project where the desired outcome is an absence of harmful events occurring . The framework focused primarily on achievement of outputs, rather than on intermediate or final outcomes . The only outcome indicators were the absence of HPAI in poultry and containment of outbreaks, and an indicator on behavioral change which was not specified clearly . Output Indicators recorded completion of activities, but did little to capture the impact of those activities . The framework assigned duties for collection to the Health and Livestock Departments. While the project tracked the percent of surveillance sites that submitted reports to the Epidemiology and Disease Control Division, the Epidemiology and Disease Control Division of the Department of Health Services was not otherwise involved in the M&E system . b. M&E Implementation: The implementing agencies faced significant capacity constraints in implementing M&E, particularly on the animal health side, where the agencies were unfamiliar with World Bank requirements or implementing M&E systems . Data collection and reporting was weak - neither the animal health nor human health agencies reported on outcome or intermediate output indicators until after the mid -term review. At restructuring, 15 if the 18 intermediate output indictors were revised, clarified or dropped . After this, reporting improved. Communications activities were monitored through knowledge, attitude and practice surveys of district managers, farmers and children conducted by UNICEF . But the ICR notes (page 8) that the project did not obtain awareness information from poultry farmers and health workers, making the data difficult to interpret . c. M&E Utilization: While M&E data were collected, this was done purely for project reporting purposes and data were not used to improve project management. M&E Quality Rating : Negligible 11. Other Issues a. Safeguards: The project was categorized ‘B’ under OP4.01 Environmental Assessment. No other safeguards were triggered . The livestock department prepared an environmental management plan, including waste management, safe handling of chemicals. The health services department used a waste management plan developed for a separate Bank health sector project. During the outbreak in Pokhara, the district avian influenza technical committee promised a community near the disposal pit that their main access road would be rehabilitated, but this had not been completed by project closure . A groundwater quality test was not undertaken prior to project closure . The ICR did not report on whether safeguard compliance was otherwise satisfactory . b. Fiduciary Compliance: The ICR (page 9) noted that financial management and procurement were moderately unsatisfactory, due to limited staff capacity and high staff turnover . A mismatch developed between expenditure and disbursements because of delays in reconciliation at the central level . Most financial management and audit reports were delayed due to weak capacity and slow reconciliation - only 40% of annual and trimester reports were prepared and submitted on time . In 2009, disbursement had to be temporarily suspended due to delayed submission of audited accounts . The emergency response nature of the project design meant that the procurement plan was front -loaded, which exacerbated weak procurement capacities . Complaints from bidders on laboratory equipment contracts led to investigations by the Bank’s Department of Institutional Integrity as well as Nepal's Commission for the Investigation of Abuse of Authority. The case was cleared, but a new tender was issued with less detailed specifications, and this led to delays in commissioning veterinary laboratories . c. Unintended Impacts (positive or negative): d. Other: The project was the first interaction between the Bank and the livestock agencies in Nepal, and may have opened the way to further and more significant cooperation in future . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR identified a number of lessons (page 17), including: Investment in biosecurity level 3 laboratories should be carefully considered at appraisal . Level 3 laboratories are expensive and have significant maintenance costs and capacity requirements, and may not be justified for small or low income countries . Collaboration with UN and donor agencies can strengthen technical institutional capacity . Other agencies can provide expertise in areas where the Bank lacks capacity . Engagement of UN agencies in global programs should be conducted at a headquarters level, not at the individual project level . It can take up to a year to contract UN agency partners, so project implementation will be delayed if issues are not negotiated until after project approval . 14. Assessment Recommended? Yes No Why? A more detailed assessment could collect additional evidence on the contribution of the project to outbreak containment, outbreak response speed, poultry surveillance quality and other areas . 15. Comments on Quality of ICR: The ICR is thorough and clearly written . The discussion of outcomes is generally evidence -based. The ICR was helpful in explaining the reasons for various project delays, and in outlining weaknesses in M&E implementation . It noted and explained weaknesses in safeguards, financial management and procurement, but it could have been more effective in linking project outputs into outcomes; little evidence was available on project impacts, particularly on how project outputs contributed to the response to the 2009 pandemic. More evidence for some lessons could have been provided within the report . While the report commented on delays in audit reports, it did not describe the content of these reports. The statement of project objectives in the ICR includes 'other zoonotic diseases'; this was not in the appraisal document or financing agreement, and the objectives were not revised . a.Quality of ICR Rating : Satisfactory