ICRR 13796 Report Number : ICRR13796 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 08/13/2012 Country : Indonesia Project ID : P103654 Appraisal Actual Project Name : Indonesia-avian And US$M ): Project Costs (US$M): US$15.00 US$3.75 Human Influenza Control And Preparedness L/C Number : Loan/ US$M ): Loan /Credit (US$M): US$15.00 US$3.75 Sector Board : Agriculture and Rural US$M): Cofinancing (US$M ): Development Cofinanciers : Board Approval Date : 12/15/2006 Closing Date : 07/31/2009 12/31/2009 Sector (s): Animal production (80%); Health (20%) Theme (s): Natural disaster management (34% - P); Rural services and infrastructure (33% - P); Other communicable diseases (33% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : Ridley Nelson George T. K. Pitman Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: The objective of the project, as stated in the Legal Agreements was to : assist the recipient in controlling the Highly Pathogenic Avian Influenza (HPAI) infection at the source in its domestic poultry, reducing the amount of HPAI virus circulating in the environment, and thereby reducing the risks of human infection as well as possible mutation of the HPAI virus to a form which is more easily transmitted from human to human . There were no other versions of the objective . There was no Project Appraisal Document (PAD) or Technical Annex. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: Component A. Participatory Animal Disease Surveillance and Response . (Appraisal US$3.01 million; Actual US$1.03 million). This was to reinforce the continuity and sustainability of participatory disease surveillance and response through local disease control centers in about 12 districts through the provision of consulting services, goods and operating support . Component B . Community -Based Preventive Vaccination . (Appraisal US$5.73 million; Actual US$1.29 million). This was to improve vaccination effectiveness by introducing a community -based approach to preventive vaccination through provision of poultry vaccines, related equipment and operational support . It also included training for staff and community vaccinators. Component C . Culling Compensation System . (Appraisal US$2.95 million; Actual US$0.00 million). This was to undertake a study for a culling compensation mechanism to provide compensation to affected poultry owners . It included the payment of compensation and an evaluation of the system after testing . Component D . Project Impact Monitoring and Evaluation . (Appraisal US$0.83 million; Actual US$0.79 million). This was to monitor and evaluate the epidemiological impact of the control interventions taken under the project and to conduct studies to address key issues through consultants' services, goods and operational support . Component E . Poultry Sector Restructuring Options Study . (US$0.2 4 million; Actual US$0.00 million). This was to use consultants to conduct a comprehensive study of options for the restructuring of the poultry sector . The study was also expected to recommend a strategy and plan to reduce transmission risks and to regulate poultry movement . Component F . Coordination of HPAI Control, Pandemic Influenza Preparedness, and Community -Based Public Information Campaign . (Appraisal US$2.25 million; Actual US$0.64 million). To support the National Committee for Avian Influenza Control and Pandemic Influenza Preparedness in its overall coordination of control, preparedness, and a community-based public information campaign through the provision of consultant services and operational support. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Costs . Of the total project cost of US$15.00 million only US$3.75 million (25%) was spent. This was the result of loss of government commitment, two missed budget deadlines, and the lack of progress on a compensation system . The only component that disbursed close to 100% of the planned expenditures was the Project Impact Monitoring and Evaluation Component. There was no expenditure under the culling and the poultry sector restructuring components. Financing . Funding was through two World Bank administered grant trust funds provided by the EU and Japan, one for US$10.00 million, the other for US$5.00 million. Financial disbursement for the project as a whole was 25% of planned and the undisbursed balance of US$ 11.35 million was cancelled at project completion . Borrower . There was no direct borrower contribution either planned or actual . However, the Task Team stated that some government funding had reached national avian influenza interventions implemented in parallel to the project . Dates . The project was extended six months to December 31, 2009 from its original closing date. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Modest . HPAI was first suspected in August 2003 in a commercial flock (ICR p. 9). By December 2004, poultry deaths due to HPAI were estimated at more than 8 million in over 100 districts. By June 2009, 31 of the country's 33 provinces had been affected . The disease is considered endemic in Java, Sumatra, Sulawesi, and probably Bali . Relevance is rated in terms of relevance at project closing to the Borrower's and the Bank's strategy . At the time of appraisal HPAI was not reflected in the Bank's Country Assistance Strategy, nor on any government sector strategy, but, given the emergency situation, this would not be expected . Relevance of objectives against the government's strategy was substantial early in the project but declined over the project period down to modest and to negligible as government lost interest. While at the time of preparation and appraisal, a national strategy was developed (2005) to respond to the emergence of HPAI, it mainly focused on vaccination and an information and communication campaign. The disease threat itself, both nationally and globally, at the time of preparation and appraisal was high based on World Health Organization and Food and Agriculture Organization assessments . It probably still remains so in Indonesia since HPAI is endemic but a high level of threat is not now the perception of government . Other diseases, with wider human or animal impact, are seen as having at least as much, if not higher, priority . The project at the time of appraisal was relevant to the Bank's global program to address this threat . Although the number of human deaths globally has declined since the peak in 2006, the disease remains a potential threat globally and in Indonesia. b. Relevance of Design: Substantial . The design was limited in substance in the absence of a PAD or even a Technical Annex . However, it had a number of the elements of the emerging standard model for avian flu developed under the Global Program for Avian Influenza (GPAI). The results chain (ICR p. 30) focussed on effective disease surveillance for early detection in poultry; enhanced biosecurity in poultry farms to reduce outbreaks; control of bird movement to reduce spread; changes in industry practices in production and marketing to reduce risk; culling of infected poultry to reduce spread; biosecure carcass disposal to prevent spread; and proper vaccination to reduce infection . It did not include a human health element, giving priority to halting HPAI spread from poultry . This potentially offered more focus on prevention of an outbreak in poultry and thus indirectly to humans, but at some risk should a human pandemic occur . The selection of 11 districts to be supported by the project was based on a decision -making process that shared out different districts among different donors, the Bank being a relative latecomer . 4. Achievement of Objectives (Efficacy): The project was rated as a problem project for the Bank throughout implementation . Effectively it was only implemented for little more than the final year of the originally planned project period and for the added 6 month extension. This was due mainly to the government's failure to ensure that the project was incorporated into the national budget approval process for two years running . This meant that implementation eventually began at a time when the HPAI threat was perceived to have declined . HPAI ) infection at the source in domestic poultry by reducing Controlling the Highly Pathogenic Avian Influenza (HPAI) the amount of HPAI virus circulating in the environment . Negligible . Outputs . The participatory animal disease surveillance and response (PDS/R) activities were originally intended to support 12 high HPAI risk districts in Central Java . Instead, mainly due to the delayed start and other donor programs, it supported 70 districts where PDS/R had already been initiated. These 70 districts were mainly lower risk districts (ICR p.12). However, USAID support was directed to the original 12 higher risk districts. A total of 140 PDS/R staff supporting this program were trained . The project supported two poultry vaccination activities (ICR p. 26): the operational research activity and the intensive vaccination activity . The latter was over a four month period at the end of 2009. However, the vaccine itself was funded by USAID. The project trained 40 Community Vaccination Coordinators and 680 Community Vaccinators. It carried out a pilot program in small farm poultry flocks . The impact of this training was not assessed . The proposed culling compensation system was not implemented partly due to the complexity of overlaying this on existing systems but also government concerns about setting payment precedents . The ICR reports (p. 27) that early on in the HPAI outbreak some compensation was paid in some areas but it became too difficult to manage and was abandoned. Under impact monitoring and evaluation , the studies were implemented using USAID funding, however project funding also contributed to this since it was used initially to advance funds for the studies to the International Livestock Research Institute (ILRI) and FAO. The proposed poultry sector restructuring study was never implemented. This was partly because a law on the raising and movement of poultry had already been passed, although this appears to have been only for Jakarta . On the proposed coordination of preparedness and the associated information campaign , there was some modest achievement, although the evidence is limited . It is reported (ICR p.13) that coordination between District Surveillance Officers and Participatory Disease Surveillance and Response activities, between the Ministries of Health and Agriculture, and between the Ministry of People's Welfare and the National Committee, and across all these participants, improved. However, there were still a number of coordination weaknesses, most notably the failure of the national committee to process the budget approval for two years running, a failure that by closing had cut the project to one quarter of its planned funding . Some 3,000 people were trained in the information and education campaign . Outcomes . There is limited evidence to link these modest and short -lived output achievements to outcomes in terms of controlling infection at source and "reducing the amount of HPAI circulating in the environment ". Given the fluctuations of the disease both globally and locally, overall preparedness calls for sustainability of preparedness and response capacity. This is difficult to achieve with such a truncated intervention . According to the ICR (p. 26), the participatory animal disease surveillance and response activities were one of the highlights of the program and project but there is no survey evidence on the impact . The ICR mission found (ICR p. 25) that many community members were still unaware of the surveillance needs and the risk . Some mentioned that they still do not report dead chickens . One positive outcome of the vaccination work reported by the ICR (p. 26) was the improved understanding of how to implement the cold storage chain needed to deliver vaccine to the districts . The lack of compensation was a significant weakness because the incentives to report and act on emergent disease may be significantly reduced, either raising the risk of disease spread in poultry or raising the costs of government surveillance to substitute for reduced farmer reporting . (Some observers apparently argue (ICR p. 27) that, in Indonesia, the lack of compensation may not have had much impact since farmers are concerned for the well -being of their community and family. This does not appear to be consistent with most global findings, although IEG is aware of some cases in other countries where backyard owners have been ready to slaughter purely on the incentive of family health considerations.) The ICR does not indicate the impact of the two day training for 3,000 staff handling the information campaign except to note that, based on interviews, there were still many poultry owners who were unaware of the risks of their poultry management practices. Whether the poultry restructuring proposal, if it had been implemented, would have actually led to an outcome of industry restructuring may be questionable if it had ended up proposing significant regulation of backyard poultry . Such a transformation is likely to throw up many social issues . However, the project was only supporting the study, not the implementation of the findings . Reducing the risks of human infection as well as possible mutation of the HPAI virus to a form which is more easily transmitted from human to human . Negligible Outputs . Unlike many other Bank-supported Avian Influenza projects, this project did not have a human disease response component. The focus was on reducing the disease in poultry which would lower the risk of transmission to humans . It is difficult to separate this element of the objectives from the poultry -focussed objective, at least in the absence of the feared genetic change to enable human to human transmission, since the human threat comes from poultry . Therefore all of the outputs and outcomes presented above are relevant also to this element of the objectives . Outcomes . The ICR does not present any quantified outcome level evidence on this element of the objectives . It might be argued that reduction in poultry at risk through the vaccination program and the supporting organizational and coordination enhancements would have reduced the risk of a human to human transmission mutation but the probabilities cannot be quantified . In any case, as noted above, in many components, including the program of preventive vaccination of poultry, there is limited attribution to the project because the project was implemented for such a short time and provided only about 25 percent of the resources planned and that within a multi -donor intervention. 5. Efficiency: Rated Modest . Efficiency is difficult to rate with little evidence on cost effectiveness . The project was designed to prevent the escalation of a disease that did not, in the end, threaten as strongly as anticipated although HPAI remains endemic in Indonesia. There were a few achievements outlined above but even most of these have questionable attribution to the project . Clearly, the notional benefit stream in terms of HPAI response preparedness that was attributable to the project was far below that anticipated at appraisal . However, the cost stream was also only 25% of that planned. The original design exhibited important potential complementarities, for example, the connection between government organized surveillance and a compensation policy mechanism that ensures an incentive for farmers to report diseased animals . However, given the weak performance of the project, such potential complementarities were largely lost . It can be assumed that, although unquantified, there are substantial inefficiencies of management in a project that was awaiting a budget that never materialized for a full two years given the delays that that entailed . No economic rate of return was estimated at the time of appraisal and none was estimated in the ICR . However, the ICR (p. 15) presents some values of birds lost to HPAI in Indonesia during the 2003-2004 crisis at between US$16 and 32 million. When losses to the broiler and layer breeders and producers were added, as well as indirect losses, the total was estimated at US$387 million. As these figures suggest, the losses to a major pandemic are likely to be high. However, the probabilities of such a high cost event are unknown and are uncertain given the limited achievement of the project over such a short period . It is not possible to attribute significant benefits in terms of HPAI preparedness to the project itself . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Relevance of the objectives is rated modest due to the declining priority given by government to avian influenza over the project period; relevance of the design to the objectives is rated substantial . Project implementation was very short due to the failure to have it covered under the national budget for two years running . Efficacy of both objectives is rated negligible. As noted by the ICR, the project had limited influence on the overall program and, even in the few positive areas, there are attribution questions . In the absence of any rate of return analysis or evidence of cost-effectiveness, and given the long delay in implementation, efficiency is rated modest . a. Outcome Rating : Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: Given the limited development outcomes, there is little that is at risk . The ICR assessment of risk (p.18) assesses the overall national program, reaching well beyond the project itself although it offers some indications of the overall avian influenza program risks. There are risks in the smallholder poultry sector of lack of containment of disease due to inadequate housing partly due to the shift of government support to focus on the caged poultry sector (ICR p. 18). There are risks in marketing (ICR p. 19) due to the thousands of traditional, largely uncontrolled, markets that are selling freshly killed birds. However, the most substantial risk is the fact that, as noted earlier, the perception of government is that HPAI is no longer a priority relative to the many other infectious diseases of humans and poultry . Thus, there will remain a risk of HPAI spreading too quickly to be manageable . The project, in its very truncated form, was able to do little to reduce that risk . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: With no PAD or Technical Annex there was limited substance in the design (ICR p. 19) and little indication of the results chain. While design was built partly on the emerging standard Global Program for Avian Influenza model but without the human health elements, it failed to provide sufficient detail or results -focussed outcome indicators. The design failed to adequately address the capacity limitations of the national committee which had staff with very limited experience in planning and budgeting . Given the weakness of government commitment, a design that channeled all funding through government was, in retrospect, a weakness, although this may have been difficult to see at the time. The Task Team pointed out that some other donor avian influenza interventions that passed through nongovernment entities performed better . The long delay in project preparation of one and a half years meant that by the time the project was effective the threat of HPAI was perceived to have declined . In retrospect, given the length of time taken to start implementation, adequate preparation and a full appraisal document would have been possible . The Bank team should have paid much more attention to working with the Ministry of Finance in anticipating budget processing problems which were not new in working with Indonesia . at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: Following the weak preparation and appraisal, supervision did not recover the lost ground . The Bank Task Team regularly expressed concern over the implementation problems, proposing actions that could have been taken. There was something of a vicious circle in that, because there was no initial project activity to supervise, no formal supervision missions were undertaken . The ICR notes that, notwithstanding many interactions with the borrower, there were no formal aides-memoire or back to office reports from such interactions . The first Implementation Status Report came 18 months after project approval. The one extension given was an exception to normal Bank practice given that the project was in unsatisfactory status at the time . Drawing from discussion with the Task Team, the lack of earlier cancellation after the second year of failure to get the budget approved, appears to have been partly a result of pressure from the Avian and Human Influenza Facility (based at the Bank Headquarters and established with support from donors to address HPAI ) to avoid cancellation of one of the largest grants to a country posing one of the greatest HPAI threats . This would have been seen as a significant chink in the global HPAI armor . Quality of Supervision Rating : Unsatisfactory Overall Bank Performance Rating : Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The main weakness in government performance was lack of commitment . As noted by the ICR, as the number of fatal human cases remained low compared to other infectious diseases, the government lowered the priority of HPAI to regular disease status i .e. no longer emergency disease status . Critical for implementation was the fact that the government did not take effective measures to ensure that the grant funds were put on a budget fast-track. Government Performance Rating Unsatisfactory b. Implementing Agency Performance: There were two implementing agencies responsible for the project, the National Committee for Avian Influenza and Pandemic Influenza Preparedness (Komnas FBPI) and the Directorate General of Livestock Services (DGLS). Komnas FBPI provided insufficient support to project implementation . Since Livestock Services (DGLS) activities depended largely on progress with the Komnas responsibilities, the weakness in the latter also held up progress by the former. Once the budgetary logjam was finally broken, DGLS appears to have done a commendable job during the last year catch -up phase. However, by this time it was far too late to improve poor outcomes brought about by earlier inaction . Implementing Agency Performance Rating : Unsatisfactory Overall Borrower Performance Rating : Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: Komnas FBPI was responsible for overall project monitoring but there was no clear definition of staffing or M&E systems. Expenditures were to be monitored by each implementer as part of the project data management system but this did not constitute an M&E system, rather it seems to have been, at least potentially, an MIS aimed more at monitoring expenditure and outputs . As noted, there was weakness in outcome indicators . b. M&E Implementation: There were no staff specifically responsible for M&E . Lack of the budgetary approval meant that there was no implementation to monitor until the final year . Towards the end there was an "acceleration team" under the Coordinating Ministry for Peoples Welfare that tracked effectiveness with a questionnaire . c. M&E Utilization: There was little to utilize given the design, budget problem and implementation weaknesses . There is now an HPAI cases database that is updated monthly that was developed towards the end and after the project . This may have benefitted from the project but attribution is unclear . M&E Quality Rating : Negligible 11. Other Issues a. Safeguards: No safeguard policies were triggered . The project was classified as a Category C under OP 4.01 Environmental Assessment and no Environmental Assessment was required (ICR p. 9). This appears to be inconsistent with other avian influenza projects reviewed by IEG with similar components which were classified as Category B . For example, in other countries the safe disposal of infected birds has been one issue addressed in the EAs . b. Fiduciary Compliance: There were two financial audits during the project . The first, in 2009, found no financial issues but noted the implementation issues. The ICR notes (p. 9) that "with the submission of the June 2010 audit, the Bank found that fiduciary compliance was met during project implementation ." Procurement was satisfactory . It is worth noting that, according to the ICR, the sole -source contracts given to the Food and Agriculture Organization and the International Livestock Research Institute were a first for the government . c. Unintended Impacts (positive or negative): The ICR notes (p. 18) that the scope of the animal health surveillance supported by the project could be enlarged to other diseases and that this has already started . d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Unsatisfactory Unsatisfactory There was modest relevance of objectives due to loss of priority by closing and during much of implementation and substantial relevance of design. There was negligible achievement of both objectives in terms of preparedness mainly due to the long delay due to the failure to get the project into the budget. Efficiency was modest partly due to lack of cost effectiveness evidence and the slow start -up. Overall, the shortcomings were major. Risk to Development Significant Significant (The ICR rating is "Substantial.") Outcome : Bank Performance : Unsatisfactory Unsatisfactory Borrower Performance : Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR offers eight lessons of which the first three below are particularly important . They are given here with some change in formulation. The fourth lesson is an IEG formulated lesson drawn from this review . 1. Choice of instrument, or package of instruments, is particularly important in a communicable disease emergency situation that calls for the sustained establishment of animal and human disease response preparedness . The choice also needs to weigh single disease preparedness versus longer -term, broader, disease preparedness . The ICR (p. 23) asks the right question: "Does the Bank need to be present in an emergency situation when the issues require a specific technical framework first? Or should the Bank wait to see how such a framework is being established and can be scaled up, while preparing a project? " 2. During project preparation on a project that will need rapid budget approval, close coordination is needed with the Ministry of Finance, particularly in a country where there have been previous problems for projects in getting budget approval. 3. A multiplicity of implementing agencies is detrimental to speedy implementation, particularly in an emergency project. If unavoidable, it needs careful institutional design . 4. Formal supervision missions are needed even when little implementation is being achieved, first, because management needs to stay informed on, and responsive to, an emerging problem, and, second, because slow implementation may signal a relevance or design flaw that needs to be addressed promptly and at a high level . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is generally informative and open about the problems in this project, although somewhat generous on some ratings. The quality of evidence is limited but stemming more from the lack of implementation . Given the lack of a PAD or Technical Annex, it could have been more informative about what the project design actually was . The analysis is limited but, again, for the same reason, the project did not achieve much of what it set out to do . The lessons are based on the limited evidence and are thoughtful and generally well formulated . To the extent allowed by the limited achievement, the ICR was results oriented and the ICR mission appeared to have attempted to fill outcome gaps in the field, for example in the area of poultry farmers awareness . The report showed no inconsistencies and followed the guidelines . It was generally concise. a.Quality of ICR Rating : Satisfactory