ICRR 13185 Report Number : ICRR13185 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 03/04/2010 PROJ ID : P072394 Appraisal Actual Project Name : Energy Transit US$M ): Project Costs (US$M): 12.3 13.1 Institution Building Project Country : Georgia Loan/ US$M): Loan /Credit (US$M): 9.6 11.4 Sector Board : EMT Cofinancing (US$M): US$M ): Sector (s): Central government administration (62%) Oil and gas (29%) Law and justice (9%) Theme (s): Pollution management and environmental health (29% - P) Environmental policies and institutions (29% - P) Other trade and integration (28% - P) Other rule of law (14% - S) L/C Number : C3473 Board Approval Date : 03/13/2001 Partners involved : Closing Date : 07/31/2005 08/31/2008 Evaluator : Panel Reviewer : Group Manager : Group : Robert Mark Lacey Fernando Manibog IEGSE ICR Reviews IEGSE 2. Project Objectives and Components: a. Objectives: The project development objective, according to the Development Credit Agreement (DCA), was to enhance Georgia's capacity to negotiate and implement oil and gas agreements in a manner that maximizes economic benefits, and minimizes social and environmental costs . The wording in the PAD is identical. The agreements would concern principally hydrocarbon transit by pipeline, benefiting from Georgia's geographic situation . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): The project had five components : (a) Environmental and Technical Advice for Pipeline Implementation (US$8.4 million at appraisal, US$10.2 million at closure) to finance consultancy services for reviewing environmental and social impacts, permits, compliance monitoring and public outreach, as well as capacity building and the provision of advisers for negotiations . (b) Legal Advice for Pipeline Implementation (US$1.0 million at appraisal, US$0.2 million at closure) to finance specialized legal services to advise on issues arising during pipeline construction and operation, including the revision, where appropriate, of Georgian legislation . (c) Annual Company Audits at international standards for the Georgian Gas International Corporation (GIC) and Georgian International Oil Corporation (GIOC) (US$0.2 million at appraisal, US$0.1 million at closure); the credit would finance part of the cost of the audits in the first year of the project only (in March 2006, GIC and GIOC, together with the oil exploration company Saqnavtobi, were merged into the Georgia Oil and Gas Corporation (GOGC) which became their legal successor ). (d) Non-pipeline Oil Spill Prevention and Mitigation (US$1.1 million at appraisal, US$1.8 million at closure) to finance specialist equipment, consultancy services and training to facilitate spill response, monitoring and enforcement by the Ministry of the Environment. (e) Project implementation (US$0.5 million at appraisal, US$0.7 million at closure) to fund operations and capacity building in the already existing Project Implementation Unit (PIU), which was also responsible for managing the implementation of other ongoing Bank -supported projects (Structural reform and Judicial Reform). The main beneficiaries of the project are the public entities involved in gas and oil transit and in related environmental protection. They include notably the Ministry of the Environment, GIOC, and GIC (later merged into GOGC). d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The depreciation of the US dollar against the SDR enabled savings of about US$ 4 million (SDR 2.5 million). These were used to finance an additional environmental adviser and to procure gas flow monitoring equipment . The closing date was extended three times : first, to allow use of undisbursed funds and continuation of environmental advisory services; second, to complete the mapping of the gas pipeline and procure spill response equipment; and third, to procure gas flow monitoring equipment . The expenditure of only 20% of the appraisal estimate for the legal advice component, often a key element of the Bank's value added, was caused by the cancelation of one of the two legal expert's input since the Bank and the Borrower could not agree on the scope and duration of the work, and a reduction in the other's contribution due to suspension of negotiations between the two interested parties. 3. Relevance of Objectives & Design: The project's development objectives are highly relevant. They respond directly to the Government's policy priority to maximize the benefits accruing to Georgia from oil and gas transit . The income generated from this transit, and from related activities, is the main source of government revenue . The objectives are also relevant to the Bank's strategy for Georgia as expressed in the 1998, 2001 and current CASs with their respective emphases on strengthening public finance, protection of natural resources and the environment, building capacity to manage transit as one the country's major assets, and generation of growth and employment . Project design drew on the lessons from earlier operations in Georgia, including in the oil and gas sector . Through extensive consultations with potential beneficiary organizations, the design team attempted to ensure that the essential preconditions were present to enable Georgia to benefit from the technical assistance on offer through the project. Major institutional risks that could have undermined project benefits were, thanks to previous experience in the country, correctly identified and mitigated . These included in particular the danger of disputes and lack of adequate coordination among the considerable number of entities involved in oil and gas transit in Georgia, some of which were to share the same consultancy services . Rather than relying on the PIU to engage with these highly political problems, the Bank proposed, at the design stage, the establishment of a high -level Steering Committee (SC) headed by the Minister of Finance, to facilitate coordination . This was appropriate and accepted by the Authorities. However, issues arose during implementation regarding the respective roles of the PIU and SC, and from the high-level composition of the latter (for example, scheduling meetings proved difficult since all participants had to be personally present; also, each time a new Minister was appointed, her /his membership of the SC had to be approved by Presidential decree ). Some of these problems could have been foreseen at the design stage . The relevance of project design is rated as substantial , and the relevance rating for objectives and design taken together is substantial . 4. Achievement of Objectives (Efficacy): The project's development objective -- enhancement of Georgia's capacity to negotiate and implement oil and gas agreements in a manner that maximizes economic benefits, and minimizes social and environmental costs -- has been modestly achieved. On the positive side, important progress has been made in strengthening the Government ’ s capacity to monitor pipeline construction and post -construction operations, and to implement Host Government Agreements (HGAs) that were already in existence for oil, and under negotiation for gas, before the project began . The project also helped to ensure appropriate construction standards for pipelines and related facilities and a redefinition of Georgia's role, both as a transit country and as a buyer of transited gas at favorable prices . Project pipeline construction is governed by the Environmental and Social Impact Assessments financed by the project . There were, nonetheless, two important areas where project achievement by closure was less than had been anticipated. First, GOGC has still not been successfully audited to international standards . An audit was carried out of GIC and GIOC in 2006, prior to the merger of the two companies into GOGC . GIC received an auditor's disclaimer since its assets were not properly inventoried or valued according to international standards . The company made some progress in addressing this issue and in moving towards international accounting standards . This was, however, insufficient, and, following the merger, GOGC contracted international consultants to inventory and value assets according to international standards . The completion of this task, expected in 2009, has been delayed, apparently because of complexities arising from the merger . The failure, to date, to achieve international auditing standards could well militate against Georgia ’s ability to negotiate international transit agreements that optimize the benefits to its economy. Second, although a national marine and land -based Oil Spill Response Plan (OSRP) has been finalized at the technical level with the help of project -financed advisors, and adequately equipped organizations established for spill response management, the Plan has still not been approved or formally adopted by the Government. This was explained by constant changes in government structure and in the agencies responsible for oil spill, which required frequent amendments to the OSRP . Although the Bank continued financing oil spill equipment in order to ensure at least a basic level of readiness to deal with a spill should it occur, the absence of a formally approved plan carries risks and could undermine the achievement of the objective of minimizing environmental costs. 5. Efficiency (not applicable to DPLs): As this was a technical assistance operation, there was no rate of return calculation, and insufficient information was provided in the ICR to enable an assessment of efficiency according to other criteria . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: With relevance substantial and efficacy modest, the IEG rating for project outcome is moderately satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: There are three significant risks to development outcome . The first and main one stems from regional instability . The armed conflict of August 2008 caused no direct damage to pipeline or other transit infrastructure, but operations by both pipeline and rail had to be suspended . The second -- that of inability to respond adequately to oil spills -- remains active so long as the Government fails to adopt formally and regularly update an OSRP . The third -- doubts arising from the lack of an unqualified audit of GOGC -- may undermine Georgia’s negotiating capacity. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: Design was technically sound, the level and extent of consultation appropriate and the lessons from previous institution-building operations in Georgia were learnt and applied . A number of risks were identified and mitigated; the three that materialized -- armed conflict, inability to achieve international accounting standards, and the failure to adopt formally an OSR -- were not foreseen, but they would have been very difficult to predict at the time of preparation. Although the idea of the SC as a coordinating body was a good one, more careful preparation could have alleviated some of the difficulties that arose from the SC ’s composition and its working relationship with the PIU. The project came too late to influence some significant decisions -- in particular the routing of the main transit pipeline had already been determined by the Georgian Government and Parliament early in 1999. M&E design was modest (see Section 10 below). Quality at entry was moderately satisfactory . This was a difficult project to supervise . Considerable uncertainty always accompanies oil and gas pipeline projects, and in this case they were compounded by the large number of affected institutions, political turbulence both nationally and regionally, and frequent absence of consent and even trust among the beneficiaries . Despite these challenges, the Bank team maintained cordial and productive relations with the beneficiaries and with the numerous other government institutions involved, and sometimes acted as an "honest broker." The team was pro-active in resolving difficult issues in a timely manner . Resources for supervision were adequate -- 15 missions took place over the seven year implementation period, and efficiency was enhanced by combining some of them with preparation of the Methane Leak Reduction project . In addition, there were numerous informal contacts which resolved pending issues . Task management was eventually delegated to the country office which enhanced interaction. Although there were four task team leaders, continuity was assured by sector experts and the involvement of the country office . Supervision was satisfactory . at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: The intensity of government interest in the project varied both in accordance with the level of the officials concerned, and over time. During preparation, the Government was enthusiastic, pro -active and keen to receive technical assistance to bolster its negotiating capacity . The first two years of implementation were the last two years before the "Rose Revolution" of 2003. During this period, the Government largely maintained its commitment at a senior level, but progress was impeded by lack of ownership, experience and capacity among the lower ranks and by rivalry and weak coordination among the beneficiary institutions (see Section 3 above). Following the "Rose Revolution," the interest of senior officials waned as the need for technical assistance declined, but at the level of the directly benefiting institutions -- notably the Environment Ministry and the oil and gas companies -- enthusiasm and engagement grew along with their capacity . The political changes associated with the "Rose Revolution" nevertheless caused serious delays during the project's later years . Government performance is rated as moderately satisfactory . Implementation was adversely affected in the initial stages by the refusal of the PIU to become pro -actively engaged with the project in view of the rivalries and poor coordination among the different entities involved . Moreover, although the PIU had experience of previous Bank -financed projects, it did not possess all the expertise necessary for the oil and gas sectors The SC therefore became involved in operational decisions for which it was totally unsuited. As implementation progressed, and the PIU's capacity increased, so did its confidence and active commitment . Although the unit suffered from several changes of management, it succeeded in retaining its enhanced technical and administrative capacity, and in the later stages was performing satisfactorily. Implementing agency performance is rated as moderately satisfactory , and IEG's rating for borrower performance as a whole is moderately satisfactory . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: Annex 1 of the PAD contains a long list of performance indicators to monitor progress in project implementation . However, these were not satisfactorily developed into a systematic set of measurable or quantifiable goals . Output indicators, for instance, include "fulfillment of legal obligations" and "benefits realized and legislative amendments adopted." The final column of the Annex table, initially called "critical assumptions" and then "from components to outputs," has indicators such as "good coordination between agencies " and "consultants are well managed." The text of the PAD is better -- there it is stated that the project's success will be assessed with regard to five key performance indicators (expected benefits from transit agreements actually realized; compliance with legal agreements and Georgian legislation; audit reports published; actual examples of enforcement of liability legislation; and respect of inter-agency agreements), though even here it is unclear how all of these would be measured . No specific institutional arrangements were made for M&E in the PAD, and no agency was given responsibility for it . M&E design is rated as modest . M&E implementation took place during supervision missions, through sector policy dialogue, and during supervision/preparation of other sector operations . Key performance indicators were refined during implementation, and progress in achieving them was adequately monitored in mission aide -memoires and implementation status reports. The situation at closure was also clearly set out in the ICR . M&E implementation is rated as substantial . While measuring progress in achieving project targets was largely the work of Bank staff, the capacity of Georgian institutions to monitor the construction and operation of oil and gas transit facilities, including their environmental and social impact, has been considerably enhanced as a result of the project . M&E utilization is rated as substantial , and IEG's rating for overall M&E quality is also substantial . a. M&E Quality Rating : Substantial 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): The project was classified as category C for the purposes of OP 4.01; as it was a technical assistance operation funding principally advisory services, no environmental or social safeguards were triggered . Financial monitoring was conducted in a complete and timely manner . Annual project audits were issued mostly on time and were unqualified except for a relatively minor tax issue in 2006 which was addressed. Procurement compliance was satisfactory . 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately There are two unresolved issues – the Satisfactory GOGC has still not been successfully audited to international standards and the Oil Spill Response Plan has not been formally approved (please see Section 4). Risk to Development Moderate Significant Risks from armed conflict in the region, Outcome : as well as from the unresolved issues mentioned above, are significant . Bank Performance : Satisfactory Moderately There were moderate design Satisfactory shortcomings (please see Sections 3 and 8 above). Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: While it is appropriate to establish a high level SC to facilitate coordination, especially where several institutions, among which there is a history of conflict, are involved, careful attention should be paid to the SC's internal working arrangements and its relationship to other implementing agencies . Operational decisions should not be the job of the SC. It is moreover, important to have a system of delegation in place should Ministerial level participants be unable always to attend, without thereby undermining the SC's authority . Project preparation and appraisal should be timed in accordance with the client's requirements . While the project was prepared with alacrity, the process was begun too late to advise on the critical issue of transit route selection. Financing advisory services for negotiation of contracts in sectors such as oil and gas is delicate, since it is likely that experts will be well acquainted with each other and with members of the negotiating party on the industry side. It is essential to build up a level of trust on the part of the Borrower that such acquaintanceships do not necessarily involve corruption nor undermine the objectivity of the advice given . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR contains the elements necessary to evaluate the project . There are, nonetheless, four observations . First, the relationship between this project and its predecessors (notably the Oil Institution Building Project ) is not adequately explained -- in particular, it should have been clearly pointed out that HGAs were either already in place or close to completion, and that the role of this project was to assist with their implementation . In this sense, the statement in the Data Sheet that "no legal agreements dealing with transit issues existed [before the project]" is especially misleading. Second, the discussion of relevance covers only objectives and not design and implementation. Third, the reader is not provided with any information on the important issue of the project's timing with regard to route selection until the section on lessons learned . Fourth, the Bank task team members for project preparation and appraisal are not listed in Annex 4. a.Quality of ICR Rating : Satisfactory