ICRR 12916 Report Number : ICRR12916 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 07/23/2008 PROJ ID : P058969 Appraisal Actual Project Name : Cultural Heritage US$M ): Project Costs (US$M): 8.9 9.05 Preservation Project Country : Azerbaijan Loan/ US$M ): Loan /Credit (US$M): 7.5 8.08 Sector Board : SDV US$M): Cofinancing (US$M ): Sector (s): Other social services (83%) Central government administration (17%) Theme (s): Conflict prevention and post-conflict reconstruction (67% - P) Participation and civic engagement (33% - S) L/C Number : C3207 Board Approval Date : 05/13/1999 Partners involved : Closing Date : 12/31/2004 06/30/2007 Evaluator : Panel Reviewer : Group Manager : Group : Anna Amato Kris Hallberg Monika Huppi IEGSG 2. Project Objectives and Components: a. Objectives: The Project Appraisal Document (PAD) and the Implementation Completion Report (ICR) state that the project development objective was to enhance the awareness of, and support for, Azerbaijan ’s culture and through that, help strengthen national identity and social cohe sion at a difficult time of national transition . This objective was planned to be achieved through: (i) the conservation and partial reconstruction of four historical monuments of global significance: the Shirvanshah Palace in Baku, the Palace of Khans in Sheki and the Karabaglar and Momine Khatum Mausoleums in Nakhchivan which were in imminent danger ; and (ii) the strengthening of Azerbaijan ’s capacity to conserve and restore historical monuments . The Development Credit Agreement only includes the sub -objectives (i) and (ii) and does not include the overall objective preceding these . This evaluation uses the objectives as expressed in the PAD and the ICR, since these are the ones that were cited and used in the implementation of the project and because the broader development outcome is targeted as the final objective of the project . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): 1. Shirvanshah Palace (Appraisal Cost US$3.09 million; After Mid-term Review (MTR) US$2.58 million; Actual Cost US$2.48 million). Included (i) concept paper for ensemble detailed plan for the palace residence (2nd floor); (ii) reconstruction of the palace residence; (iii) conservation of Divanhane; and (iv) purchase of a low pressure cleansing device. The palace is in the old city of Icheri Scher . Shortly after project approval, UNESCO declared Baku ’s Icheri Seher (IS) as a World Heritage Site and soon added it to its endangered site list . 2. Sheki Palace of Khans (Appraisal Cost US$3.11 million; After MTR US$2.62 million; Actual Cost US$2.11 million) included (i) concept paper for restoration; (ii) static, restoration and climate control; (iii) restoration of woodwork and glass; (iv) woodworking tools; and (v) restoration of wall paintings. 3. Karabaglar (Appraisal Cost US$0.86 million; After MTR US$0.72 million; Actual Cost US$0.82 million) included (i) study of twin minarets' stability and preparation of project concept; (ii) bracing and rain proofing minarets; and (iii) conserve/restore mausoleum and site. 4. Momine -Khatum Mausoleum (Appraisal Cost US$0.28 million; After MTR US$0.24 million; Actual Cost US$0.37 million) included (i) study of movement of mausoleum; and (ii) conservation work (cleaning, restoration of cupola, waterproofing) 5. Institution Building (Appraisal Cost US$1.56 million; After MTR US$1.33 million; Actual Cost US$1.33 million) included a study tour to Europe, training of young restoration specialists, equipment for the Azerberpa archives, inventory and computerization of archives, equipment for Institute, auditor expenses, a local procurement consultant, technical consultancy support and operating costs for the Project Implementing Unit (PIU). 6. Additional Mid -Term Review Component : (Actual Cost US$1.94) The following activities were added following the MTR to promote economic and social development : (i) development of Bed and Breakfast (B&B), and handicraft activities in Sheki; (ii) training for tour operators in Nakhchivan; (iii) organization of a Festival in Baku ’s historical Icheri Seher (IS); (iv) conservation of the Mosque and the Ovdan of the Shirvanshah complex in IS; (v) landscape improvements in Icheri Seher, Nakhchivan and Sheki; (vi) the refurbishment of an Information Centre in IS (later dropped); and (vii) the rehabilitation of Caserne (Sheki) for the creation of a business incubator and showroom for the above mentioned handicraft producers . Towards the end of the project the following were added : (viii) development of a management plan for Icheri Seher; and (ix) preparation of a study/analysis of the current institutional setup of the Cultural Heritage sector in Azerbaijan, including the development of proposals aimed at increasing the financial sustainability of the sector and related institutions . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The project had two extensions . After the Mid-Term Review, at the government's request, the Development Credit Agreement was amended to reallocate available loan proceeds and extend the project for two years to December 31, 2006, in order to allow added activities to be completed . Due to the reported good management of project resources, careful supervision, and the revaluation of the SDR , the project generated savings estimated at US$ 1.5 million by the time of the Mid-Term Review in February 2003. The Bank therefore agreed with the Government on a list of new complementary activities that would promote community development, direct involvement of local population in the project, and a further increase in employment opportunities .The second extension of six months was to allow a proper finalization of the Management Plan for Baku ’s old city of Icheri Seher, which was delayed due to lack of collaboration between the Ministry of Culture and Tourism (MCT), and the City of Baku administration . 3. Relevance of Objectives & Design: The relevance of the Objectives as written at appraisal are rated as Modest. for a number of reasons: (1) The objectives were too broad and difficult to measure . The Bank agreed to fund the project upon request of the government, to recognize the strategic importance of the cultural sector as a means to build identity and social cohesion in the relatively young nation of Azerbaijan (which gained its independence in 1991). But this is too abstract an objective for a project to focus on . (2) The objectives were not in line with the then -current CAS (although they are now, with the current CAS focussing on supporting sustainable and balanced growth of non -oil economic sectors, such as tourism ). (3) The objectives should have directly focussed on national objectives of economic diversification, growth and poverty alleviation rather than conservation /restoration in isolation. This could have been done by linking growth of tourism to cultural heritage site restoration . The relevance of Project Design is also rated as Modest: The project (i) initially failed to link conservation /restoration with local economic development and community participation; (ii) it wasn't consistent with Bank guidelines adopted by the Board, which specifically direct that cultural heritage activities should be linked to poverty reduction in projects; (iii) it didn't allocate resources for important complementary activities; and (iv) the design of the M&E system was poor (indicators were not defined to measure the achievement of all of the objectives; the logframe was unclear; no targets were established for the indicators; there was no plan to collect baseline data; and no methodology to establish attribution of outcomes to the project ). 4. Achievement of Objectives (Efficacy): Enhancing the awareness of, and support for, Azerbaijan ’s culture and through that, helping strengthen national identity and social cohesion : Modestly achieved . The indicators that were measured, as well as the increased budgeting for cultural heritage activities might indicate enhanced awareness and support for culture . For instance, the number of visitors to cultural sites increased by 24 percent. However, it is difficult to make the leap, and there is no evidence to show, that it has helped to strengthen national identity and social cohesion . Conservation and partial reconstruction of four historical monuments of global significance : Substantially achieved . The physical restoration through Components 1 through 4 associated with this sub-objective as appraised were completed satisfactorily and in fact more sites were restored than planned . All of the studies and plans were completed. In addition, the activities which were added at the MTR and which aimed at promoting economic benefits, tourism and community participation related to the restored sites, were also mostly completed satisfactorily . All component targets were met or exceeded, and all four key Project Indicators showed positive trends, including : increase in number of NGOs involved in conservation and restoration which went from 0 to 32, and the number of new site restorations which increased from 25 to 48. Strengthening Azerbaijan ’s capacity to conserve and restore historical monuments : Modestly achieved It is hard to find evidence of this as the indicators are incomplete and there were not baselines but there is anecdotal evidence at both the site level and the ministerial level of increased institutional capacity . However, the ICR also provides stories that say institutional development has been problematic . Most of the activities in the capacity building component (Component 5) and the added institution-building activities of Component 6 were completed satisfactorily. The PAD did not define any indicators to measure institutional capacity . The closest is one PDO indicator which showed that funding of public and private sources for conservation and restoration of historical and cultural sites has increased from an annual budget of 142,000 Azeri Manats to 1,450,000 Azeri Manats. 5. Efficiency (not applicable to DPLs): Efficiency is rated non-evaluable due to lack of information in the ICR . No economic or financial analysis of the project was done at appraisal . The reason given was that the project's benefits were mostly public goods and appropriate methodologies for estimating them were not developed during preparation . Economic analysis could and should have been done at appraisal and again at completion . Even though it is hard to value benefits such as "national identity", there could have been an analysis of the economic benefit of increased tourism . The project did work efficiently and generated savings (part of which were due to exchange rate ) in the beginning of project implementation of US$1.5 million which were then funneled into additional activities at MTR . Both the PAD and the ICR could have done more to analyze cost effectiveness . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The relevance of the project was mixed . The project and its objectives were not relevant to the CAS at the time of approval, but could be seen as relevant to the current CAS . The design originally did not include provisions for promoting economic development and tourism, although activities were added after the MTR to overcome this shortcoming which would have related to larger country objectives . One of the objectives was substantially achieved, but the other two objectives were only modestly achieved . Efficiency was not evaluable . Due to the substantial achievement of one objective, and the change midstream to make the project more relevant, the outcome of the project was satisfactory, although only moderately so . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: There is much evidence that the focus on these sites will be sustained : MCT authorities, including the Minister are convinced of the value of the new approaches to conservation of cultural heritage introduced through this project . . The adopted management plan for the IS old city by the relevant authorities and the on -going institutional study, together with the current preparation of a follow -up project by the MCT will probably consolidate the achievements and advance in further actions towards the development of the sector . Funding for cultural activities has substantially increased. Some moderate risks exist regarding the maintenance of the Karabaglar site since insufficient attention is being provided for the maintenance and supervision of the site . Also the monuments in Nakhchivan are located in a well-known earthquake zone. Although restoration technologies were used to minimize this risk by making structures more resilient to earthquakes, the threat will always exist . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Quality at Entry was moderately unsatisfactory . The Bank responded to the GOA request for assistance to recover endangered monuments, and through supporting the institutions and symbols of nationality, to help Azerbaijan to develop positive economic and social development. The project was approved by IDA's Board of Directors despite considerable criticism that it did not adhere to the principles for cultural heritage projects adopted by the Board itself, and further that it did little or nothing toward IDA ’s overarching objective of poverty reduction. The PAD was weak, particularly because it lacked a reasonable logical framework and the PDO indicators were too loosely linked to the project activities and because no baseline was presented or required . At the same time the PAD did provide a reasonable framework for proper implementation of restoration works and the team did respond quickly to adjust project design following the Board ’s recommendations. Quality of supervision was satisfactory . During and after the second year, following up on the IDA Board ’s criticism of project flaws, the project team initiated an intense dialogue with the client to improve the PIU team by strengthening its capacity, improving its integration, accelerating procurement and complementing the original project activities with activities to promote economic and social development of the project areas, using the Community Driven Development approach . To this end, strong assistance was provided for the dissemination of procurement guidelines and the International Competitive Bidding pre -qualifications documents allowing for adequate time for the preparation of cost -effective proposals. Strong TA support was mobilized by the supervision team to help the relative initial weakness of the PIU . The Bank staff based in the country office helped to provide strategic support to the PIU throughout implementation . at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Government project ownership has been very strong from the very beginning . The government was flexible in changing its views such as the expansion of the scope of the project beyond “bricks and mortar� to a development approach engaging local communities in conservation of sites . It must be noted that the government appreciated the convincing arguments brought forward by the Bank team based on best practices from similar initiatives, as noted in the Borrower ’s own Report. The Minister showed a particularly favorable attitude towards CDD activities, in fact he promoted participation of both the Sheki Bed and Breakfast and Handicraft associations to the country’s Tourism Fair in April 2007. The PIU was initially quite weak, which, compounded with the poor attention provided by the Bank supervision team at those initial stages, resulted in losing about a year . After some major PIU reshuffling, in the second and third years of project implementation the PIU underwent a steep learning curve and became increasingly competent. Financial management, project accounting and reporting, internal control procedures, planning and budgeting; counterpart funding; and external audits; were done since then in a correct professional manner . Some weaknesses can be tracked back to the areas of M&E and the capacity of the project director to maintain a high quality PIU staff at all times a. Government Performance :Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Satisfactory 10. M&E Design, Implementation, & Utilization: Design: The M&E suffered from the lack of a complete set of indicators, and the indicators that were prepared were too broad . No baseline data were given during preparation . The international expert hired to develop an Monitoring Information System failed to propose an adequate plan as well as to focus on systematic outreach activities because of a lack of acceptable data sources . A methodology which would have allowed for comparisons with a counterfactual should have been considered . Implementation and Utilization: As the project changed its approach, the PIU still failed to develop a system, which would have allowed better reporting and closer follow -up on the challenges faced by the sector . For example, the lack of information on the actual users of the archives and the difficulty in acquiring reliable data on number of visitors and revenues from project sites, are clear indicators of capacity and planning constraints faced by the government agencies supported by the project . It also shows their weakness in putting in place a transparent and effective revenue mechanism. Also, a Project Oversight Commission was set up which did not meet very frequently due to an apparent lack of interest by its members . a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): No safeguard or fiduciary issues were found . Unintended positive impacts included the leveraging of additional resources for cultural heritage restoration and promotion partially due to the project . The Ministry of Culture and Tourism increased its budget allocations from 142,000 Azeri manats in 2001 to 1.45 million Azeri manats in 2006 -- a ten-fold increase. The World Heritage-designated old city of Icheri Seher was removed from the UNESCO's list of Endangered Sites. 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately There were moderate shortcomings in Satisfactory the operation’s achievement of its objectives, and in the relevance of the objectives. Efficiency was not evaluable. Risk to Development Moderate Moderate Outcome : Bank Performance : Moderately Moderately Satisfactory Satisfactory Borrower Performance : Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The restoration, conservation and touristic promotion of cultural heritage sites can be a valuable asset to economic growth and development . This project was originally not focused on a return on the restoration investments, or on generating a stream of income to cover the costs for maintenance of the sites or as a contribution to local and regional economic growth . The project did manage to add this type of intervention going beyond civil works during implementation . However, adding such activities at the beginning of the project would have led to broader and more sustainable results . Engaging communities in economic and social activities around cultural sites is key to a successful cultural heritage site project . No other activities have been as successful in building widespread understanding, stewardship and real commitment to conservation . The importance and reliability of data and its collection should be agreed upon in detail and up -front with implementing agencies . The domino effect of this initial lack reaches far into the future .For instance, in this project, poor or no control over ticket sales at sites was a major obstacle to analyze the financial sustainability of the sites. Very stringent procurement rules can prevent corruption, add to project savings, and show the importance of adapting procurement rules to country circumstances . It also strengthens the Bank ’s fiduciary role in monitoring of contracts in the context of poor governance . Institutional strategies and longer -term Bank engagement are needed to build comprehensive and effective capacity in this sector . 14. Assessment Recommended? Yes No Why? Examples of using Cultural Heritage and site restorations as an engine for local economic development are valuable lessons for other situations . Also, it would be interesting to see if the results are being maintained and if tourism near project sites has increased . 15. Comments on Quality of ICR: The ICR is generally satisfactory, but with some shortcomings . There is limited evidence on the achievement of outcomes, mainly due to the project's failure to set up and fully implement the planned M&E system . Also, in Annex 1, Project Costs and Financing, the revised cost estimates from the MTR are listed as the appraisal estimates . a.Quality of ICR Rating : Satisfactory