ICRR 13622 Report Number : ICRR13622 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 08/10/2011 PROJ ID : P099832 Appraisal Actual Project Name : Avian Influenza US$M ): Project Costs (US$M): 9.5 10.5 Preparedness Project Country : Armenia Loan/ US$M ): Loan /Credit (US$M): 6.4 5.4 Sector Board : US$M): Cofinancing (US$M ): Sector (s): General public administration sector (42%) Health (27%) Agricultural extension and research (18%) Animal production (11%) Other social services (2%) Theme (s): Rural services and infrastructure (33% - P) Other communicable diseases (33% - P) Pollution management and environmental health (17% - S) Health system performance (17% - S) L/C Number : C4178 Board Approval Date : 06/02/2006 Partners involved : Closing Date : 07/31/2009 07/31/2010 Evaluator : Panel Reviewer : Group Manager : Group : John R. Heath Ridley Nelson IEG ICR Review 1 IEGPS1 2. Project Objectives and Components: a. Objectives: The Project Appraisal Document (PAD) and the Financing Agreement (FA) have identical wording for the project development objective (PDO), which may be broken down as follows : Objective 1: Minimize the threat posed to humans by the Highly Pathogenic Avian Influenza (HPAI) infection and other zoonoses in domestic poultry; and Objective 2: Prepare for the control and response to an influenza pandemic and other infectious disease emergencies in humans. The ICR (p. 2) says that project development objective was amended in 2008 to read: "The objective of the Project is to minimize the threat in Armenia posed by the HPAI infection and other livestock diseases , and to prepare for the control and response to an influenza pandemic and other zoonoses or infectious disease emergencies in humans " (Italics added to show difference from original PDO : the change expanded the range of animals covered, in response to the outbreak of African Swine Fever in 2007.) b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 10/27/2008 c. Components (or Key Conditions in the case of DPLs, as appropriate): The description and cost of components is taken from the PAD (Annex 4a). I: Animal Health (Estimated cost at appraisal, US$5.6 million; Actual cost by closing, US$ 5.4 million), comprising: Development of a national policy framework and strategic plan; Strengthening of disease surveillance and diagnostic capacity; and Preparation of an outbreak containment plan . II: II : Human Health (Estimated cost at appraisal, US$2.8 million; Actual cost by closing, US$ 4.1 million), comprising: Enhancement of public health program planning and coordination; Strengthening of national public health surveillance systems; and Strengthening of health system response capacity . III: III : Public Awareness and Implementation Support (Estimated cost at appraisal, US$1.5 million; Actual cost by closing, US$ 1.0 million), comprising: Information and communications services; National coordination; Project implementation; and Monitoring and evaluation. IV: IV : Support for Critical Imports (Estimated cost at appraisal, US$0.2 million; Actual cost by closing, US$ 0.0 million) d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project funding expanded with the addition of US$ 2 million from the Animal and Human Influenza Facility, which became effective on June 8, 2007. The Borrower was expected to finance US$ 2.2 million of the overall project cost; the actual contribution was US$ 2.4 million. The project was formally restructured on October 27, 2008, by which time US$3.3 million (61 percent of IDA funds) had been disbursed. As noted in Section 2a above, restructuring involved a change in the wording of the project development objective to accommodate a broadening of the animal disease threat. Component I was adjusted, with laboratory staff training expanded to include African Swine Fever (ASF) diagnosis and surveillance . Project financing for the culling compensation fund was increased by US$ 300,000 to US$1,050,000, with US$750,000 earmarked for ASF compensation. Coverage of Component III was broadened to include ASF. Also, funds for matching grants to backyard poultry producers were reallocated to the development of a diagnostic manual that would address the most prevalent animal diseases . The closing date was extended from July 31, 2009 to July 31, 2010 to allow rehabilitation and modernization of the Central Veterinary Laboratory to be completed. 3. Relevance of Objectives & Design: Relevance of Original Objectives (Rating : Substantial ) Outbreaks of HPAI begun in late 2003 in Southeast Asia and spread to other regions, damaging prospects for the poultry industry and potentially threatening human health around the world . In the case of Armenia, the FY2009-12 Country Assistance Strategy, which was current when this operation closed, identifies the high prevalence of livestock diseases among the issues to address, specifying as a target the implementation of a national comprehensive control program for brucellosis and other zoonotic diseases (Annex 2). Events in Armenia after project approval bore out the relevance of the operation's objectives : there were outbreaks of African Swine Fever in 2007 and 2008 and the AH1N1 swine flu virus was detected in 2009. Relevance of Revised Objectives (Rating Substantial ) The only difference between the original and revised objectives was a broadening of diseases that could be included . There was an expansion to "other zoonoses" beyond just poultry (a zoonosis is a disease of animals that could be transferred to humans) and the addition of "other livestock diseases ", this presumably referring to diseases beyond just those that can be transferred to humans . Such expansion was relevant to the changing situation allowing project resources to be applied more widely and for skills to be sustained through responses to similar diseases . Relevance of Original Design (Rating : Substantial ) Project design was consistent with the guidelines for the Global Program for Avian Influenza and with Bank guidelines for emergency policy interventions (OP/BP 8.50). To ensure that there would be no conflicts with, or duplication of, other donor activities, the design of the project took into account findings from a HPAI risk assessment carried out in January-February 2006, as well as the recommendations in WHO, FAO and USAID reports on Armenia's HPAI preparedness. In 2007, an assessment by the Quality Assurance Group rated project components and processes as moderately unsatisfactory . However, IEG finds that the mix of components and, in particular, the adoption of a combined approach to tackling animal and human disease, was appropriate : the high prevalence of brucellosis among humans shows the need to address animal disease control and public health threats with the same package of interventions . Given that there was a formally approved restructuring the before and after restructuring ratings are combined . The restructuring expansion to other disease prevention measures was sound with potentially gains in efficiency and perhaps sustainability, although possibly with some risks in the sustainability of preparedness due to dilution of focus . Relevance of Revised Design (Rating : Substantial ) The design was only changed to the extent of the broadening outlined above . This did not call for a significant redesign, but allowed for the broadening of project interventions to other disease threats . 4. Achievement of Objectives (Efficacy): Note: Objectives 1 and 2 above apply to both pre- and post-restructuring periods. Objective 1: Minimize the threat posed to humans by the Highly Pathogenic Avian Influenza HPAI ) infection and (HPAI) other zoonoses in domestic poultry (Rating : Modest ). There was no HPAI outbreak or resulting human pandemic but it is impossible to assess how much this nonevent was the result of project activities . One of the indicators proposed at appraisal was a poll of experts asked to assess readiness to cope with an outbreak; for reasons not explained in the ICR, the poll was never conducted . Another indicator was the monitoring of poultry breeding stock farms : the target was for all farms to be monitored but only 40 percent were covered. A further indicator was progress toward the target of providing demonstrations on 2000 farms of techniques for raising backyard poultry; no demonstrations were carried out . Also, the proposed fund for compensating poultry owners forced to cull their animals was never set up . On the other hand, a national avian influenza plan was developed and adopted by government, and a veterinary services self -assessment was carried out. Based on what it saw during the mission the ICR team concludes that Armenia is now at an "advanced state" of HPAI readiness. Objective 1 was expanded in 2008 to include other livestock diseases . The ICR (pp. 10-11) says that the threat posed by the full gamut of livestock diseases has been contained, owing to project -sponsored improvements in surveillance, diagnostic, control and response capacity in both animal and human health services . No specific indicators were developed to measure performance of activities specific to animal diseases other than HPAI . A mixed picture emerges. On the one hand, "the rapid and effective response to the outbreak of African Swine Fever , which was contained in Tavush and Lori marzes, in 2007 and 2008" (ICR, p. 11) may suggest that disease threat had been minimized. Also, "efficient and safe culling...was successfully tested and demonstrated during a field -level simulation" (ICR, p. 21). On the other hand, with respect to African Swine Fever compensation, the ICR (p. 21) reports, "Independent monitors employed by the project found much to be concerned about in the actual administration of this program, because adequate operating and verification principles and procedures had not been developed and applied". Furthermore, the ICR notes two gaps in the country's capacity to minimize the threat of zoonoses and other livestock diseases : "there is still no acceptable, officially monitored and enforced national system for disposing safely of highly infectious animal carcasses other than through burial, and there is still no institutionalized arrangement to compensate livestock owners when their animals are killed by an epizootic disease or culled on government orders " (ICR, p. 11). The absence of a compensation fund may reduce the incentive for farmers to report disease outbreaks . Weaknesses in containing a common disease with a narrower threat than avian flu bodes ill for attempts to tackle an avian flu outbreak . Objective 2: Prepare for the control and response to an influenza pandemic and other infectious disease emergencies in humans (Rating : Modest ). On a positive note, national strategic plans for improving public health surveillance and disease control were prepared in line with WHO recommendations and are funded and ready for implementation . The expected number of labs (2) for routine influenza diagnosis were rehabilitated and equipped and around 2,300 people were trained in influenza virus surveillance and control . Less positively, some of the proposed activities to raise public awareness were not carried out including the research -based risk communication strategies and the Knowledge, Attitudes and Practices survey. While the project team has advised that some communications activities were carried by other donors, due to the absence of outbreaks and the decline in the perceived threat of disease emergencies, the government was reluctant to commit to the communications strategy that was part of project design . However, government did set up a secretariat to coordinate communications . As noted, Government has not yet set up an animal disease compensation fund or budgeted for essential epidemiological surveillance and control measures . The ICR (p. 10) says that the government lacks the budget to address these matters and key legislation has yet to be passed. During the project period, no new human influenza vaccine was available . Revised Objective and Post -restructuring Split Rating . As noted above, the only difference between the original and revised objectives was a broadening of diseases that could be included . There was an expansion to "other zoonoses" beyond just poultry and the addition of "other livestock diseases ". The original objectives as formulated above remained. A split rating is called for due to the formal restructuring . As noted under Relevance of Design, the restructuring adjustment of objectives was probably positive overall, albeit with some risk of loss of Avian Flu focus, however, we do not find enough gains in achievement in the post -restructuring period to raise the above ratings . The aggregate rating therefore remains the same . 5. Efficiency (not applicable to DPLs): The ICR (p. 32) concedes that the economic analysis is "less convincing than desired " because there is no reliable information on the proportion of the human population that would develop disease or the proportion that would die as the result of an outbreak. Given that the scale of the potential losses is unknown it is difficult to estimate a benefit stream based on losses that were avoided because people did not become sick or livestock was not destroyed . However, the ICR shows that for 30 different scenarios the economic rate of return exceeds the discount rate . In its comments, the Region has pointed out that some measure of the potential benefit of the response can be deduced from other countries where Avian Influenza outbreaks have occurred . While there remain uncertainties, on balance, Efficiency is rated substantial . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 70% 100% ICR estimate Yes 53% 100% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Given the outbreak of HPAI in many countries after 2003, the relevance of the project objective was substantial for both the pre- and post-restructuring period. The project design adhered to the relevant guidelines and, despite weaknesses in the results framework, the relevance of project design was substantial, again for both periods . The project achieved its two objectives to only a modest extent, owing to a tapering off of government commitment leading to budget shortfalls and the absence of a Compensation Fund that may make it less likely that farmers will report disease outbreaks . Efficiency is rated Substantial . Together these findings point to an outcome rating of moderately satisfactory . For the reasons indicated under the Relevance and Efficacy sections, this rating remains the same for the post-restructuring period (the pre-restructuring representing about 60% of disbursements) so no weighting of the pre- and post-restructuring periods by disbursement is necessary in aggregating the ratings . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The ICR (p. 13) refers to the "uncertain prospects of adequate future budgetary support for the operation, including staff salaries, and maintenance of the upgraded laboratory and other service facilities ." Government commitment to project objectives seemed to weaken once there was less talk of a disease threat . It failed to back a communications strategy and some elements of the planned communication interventions, although some was covered by other donors. Government has not yet set up an animal disease compensation fund or budgeted for essential epidemiological surveillance and control measures . The ICR (p. 10) says that the government lacks the budget to address these matters and key legislation has yet to be passed . One positive outcome seems likely to be sustained : high-quality training materials were developed under the project and these are being used by veterinarians and university students (ICR, p. 16). a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: Quality at Entry In common with other avian flu operations prepared at this time, there was pressure on the project team to prepare and appraise the project as swiftly as possible . The rapid response of the Bank is a positive aspect . But swiftness carried a price: in some respects, the project was not ready to implement when the loan was approved . "Several subcomponents envisaged in the PAD could not be implemented at all because they were inadequately prepared and appraised and proved to be unrealistic (poultry sector restructuring ) or in violation of existing laws and regulations (Bekkari pit construction, incinerators )" (ICR, p. 13). The Results Framework and the provision for monitoring and evaluation were inadequate at appraisal and this deficiency was not rectified during implementation. Some priorities, such as the need to rehabilitate the central veterinary lab, were overlooked at appraisal and project design had to be adjusted later to accommodate them . Preparation could have been more thorough: for example, environmental policy restrictions were not sufficiently well understood during preparation, making it necessary to drop the procurement of mobile incinerators and use of existing burial pits . The project was rated moderately unsatisfactory in a 2007 Quality at Entry assessment. Quality of Supervision Bank supervision was satisfactory on financial management, procurement and technical issues but weak in other respects. The Bank did not improve the Results Framework, or monitoring and evaluation, and failed to ensure that regular M&E reports were issued . A key aspect of the Environmental Management Plan was not complied with (see Section 11 below). Project restructuring was begun in February 2008 but was not completed until late that year. at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Unsatisfactory c. Overall Bank Performance :Moderately Unsatisfactory 9. Assessment of Borrower Performance: Government Performance The ICR says (p. 15) that the government remained committed to project objectives throughout but the evidence in the last paragraph of Section 4 above suggests problems with commitment . The ICR (p. 15) notes that the government failed to take the steps needed to establish the Compensation Fund that was mooted when the project was prepared. This was a critical omission since the absence of such a fund may have reduced the incentive for farmers to report disease outbreaks . In its comments, the Region pointed out that “While the fund was not formally or legally established, the project supported the Government to develop all of the legal framework and operational manuals to implement it . The Government, in fact, used the project recommendations and approach to provide compensation to farmers during the African Swine Fever outbreak .� However, there are still questions about when the fund will be formally or legally established and to what extent the current system has established sufficiently secure expectations for farmers of adequate and timely compensation . Implementing Agency Performance The project was not well-monitored or evaluated (Section 10) -- design flaws in the system of monitoring were not addressed during project implementation and there was a lack of detailed reporting . The ICR notes (p. 15) that the agency's failure to employ an M&E specialist, and its failure to produce semi -annual progress reports were significant weaknesses. The agency also shares responsibility for the delay in launching the staff training program (which did not take place until mid-2008). a. Government Performance :Moderately Unsatisfactory b. Implementing Agency Performance :Moderately Unsatisfactory c. Overall Borrower Performance :Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: Design Although there is ample detail on the Results Framework and the monitoring arrangements in the PAD, there was a lack of realism in the specification of indicators and targets . The Quality at Entry review in 2007 recommended a simpler framework with more emphasis on outcomes and impact (ICR, p. 9). There were too many vague, hard-to-measure, indicators. Implementation Flaws in the system of monitoring were not addressed during project implementation and there was a lack of detailed reporting. Plans to recruit an M&E specialist were never acted on . Utilization There is no indication that M&E findings influenced project management or that M&E capacity built by the project would be likely to make a positive impact over the longer term . a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards An Environmental Management Plan was approved in 2006 but in one important respect was not complied with : an unsatisfactory attempt was made to develop and apply safe animal carcass disposal procedures . Contrary to the terms of the Plan, indicators for environmental monitoring were not developed . Fiduciary Comment in the ICR is limited to the observation that fiduciary policies were complied with and all audits were satisfactory. Unintended Impacts . No unintended impacts are reported . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Significant Budget shortfalls, the failure to pass Outcome : key legislation, the lack of a compensation fund, and signs of declining government commitment increase the risk to development outcome. See Section 7. Bank Performance : Moderately Moderately Unsatisfactory Unsatisfactory Borrower Performance : Moderately Moderately The Government failed to take the Satisfactory Unsatisfactory steps needed to establish the planned Compensation Fund, flaws in the system of monitoring were not addressed during project implementation and there was a lack of detailed reporting. In its comments, the Region pointed out that “While the fund was not formally or legally established, the project supported the Government to develop all of the legal framework and operational manuals to implement it. The Government, in fact, used the project recommendations and approach to provide compensation to farmers during the African Swine Fever outbreak.� However, there are still questions about when the fund will be formally or legally established and to what extent the current system has established sufficiently secure expectations for farmers of adequate and timely compensation. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The ICR proposes the following lessons : Some aspects of emergency response (e.g., procurement of protective gear ) can be carried out quickly and do not require detailed preparation; this is not the case for institutional strengthening . If donor coordination is to work, the implementation schedules of the various agencies need to be flexible and the costs entailed by coordination need to be acknowledged up front . If M&E is to be an effective management tool, indicators and targets need to be realistic and progress needs to be reported regularly so that problems can be identified early on and prompt corrective action taken . Project-specific coordinating bodies located outside the civil service create parallel lines of communication and weaken the engagement of the line agencies that should be responsible for the sector . 14. Assessment Recommended? Yes No Why? To verify the ratings and document lessons learned in an area with limited evaluation evidence so far . 15. Comments on Quality of ICR: This was a difficult ICR to write given the inadequacy of the monitoring evidence . The evidence that is available is clearly and concisely presented . A valiant attempt is made to demonstrate that resources were used efficiently but, as the ICR itself concedes (p. 32) the result is not very convincing . The lessons drawn could have been derived from a wide range of operations and do not provide pointers to how challenges specific to disease response may be better handled in the future; but, in fairness, because, in this case, the disease threat failed to materialize, the scope for learning was somewhat limited. More detail on fiduciary compliance should have been provided . a.Quality of ICR Rating : Satisfactory