ICRR 13362 Report Number : ICRR13362 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 09/12/2012 PROJ ID : P055068 Appraisal Actual Project Name : Irrigation & Drainage Project Costs (US$M): US$M ): 32.80 51.84 Community Development Project Country : Georgia Loan/ US$M): Loan /Credit (US$M): 27.00 41.38 Sector Board : ARD US$M): Cofinancing (US$M ): Sector (s): Irrigation and drainage (55%) Central government administration (45%) Theme (s): Rural services and infrastructure (29% - P) Water resource management (29% - P) Participation and civic engagement (28% - P) Law reform (14% - S) L/C Number : C3542; CH176 Board Approval Date : 06/28/2001 Partners involved : Closing Date : 04/30/2007 04/30/2009 Evaluator : Panel Reviewer : Group Manager : Group : J. W. Van Holst Ridley Nelson IEG ICR Review 1 IEGPS1 Pellekaan 2. Project Objectives and Components: a. Objectives: This project was the first phase of a 12-year three-phase Irrigation and Drainage Revitalization Adaptable Program Loan. The Program planned to rehabilitate and maintain, for the benefit of sustained agriculture, irrigation and drainage networks commanding 255,000 ha by: (i) establishing about 500 Amelioration Associations (AAs) through which farmers will participate in rehabilitation and O&M of irrigation or drainage systems covering 255,000 ha; and (ii) financing rehabilitation of selected higher -order infrastructure covering approximately 125,000 ha (91,000 ha irrigation and 34,000 ha drainage.) The Project Appraisal Document (page 3) states that the specific project development objectives of the first -phase project were: "to increase agricultural production and farm incomes on 110,000 ha by arresting further deterioration of irrigation and drainage infrastructure and keeping the operable infrastructure functional, through : (i) establishment and support to AAs that will take over O&M of irrigation and drainage systems within their service territories; (ii) repair and rehabilitation of irrigation and drainage infrastructure to improve irrigation water conveyance and distribution, and drainage discharge; and (iii) limited support to [the] Department of Amelioration and Water Economy of the Ministry of Agriculture and Food (DAWE) to improve its management of rehabilitated irrigation and drainage infrastructure ." The objective description in the Loan Agreement (Schedule 2) was: "to increase agricultural production and farm family incomes by arresting further deterioration of irrigation and drainage infrastructure and keeping the operable infrastructure functional on approximately 110,000 ha, through: (a) establishment of water management organizations (the AAs) that could perform operation and maintenance of irrigation and drainage systems; (b) repair and rehabilitation of irrigation and drainage infrastructure; and (c) limited support to DAWE." Both objectives point to the same outcomes . For the purpose of this Review the Loan Agreement's description of objectives will be used. An Additional Objective and financing was added following extensive flooding in Georgia in April, 2005 under OP8.50 Emergency Recovery Assistance : “to improve and secure the sustainability of river flood protection works and irrigation headworks and canals ". b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives /key associated outcome targets? Yes Date of Board Approval: 06/27/2005 c. Components (or Key Conditions in the case of DPLs, as appropriate): 1: Irrigation and Drainage Systems Rehabilitation and Maintenance (US$14.08 million at appraisal; $US$27.36 million at completion). This has three subcomponents : 1. Rehabilitation of Irrigation Infrastructure . This financed (i) rehabilitation of the main water supply systems of the three schemes covering approximately 16,330 ha, including the Tashiskari and the Kvemo Alazani schemes, but not the small Meshketi scheme; and other investments including the Poti diversion and the Ilto Siphon; and (ii) provision of technical assistance for the design and supervision of the rehabilitation works. 2. Rehabilitation of Drainage Infrastructure . This financed (i) rehabilitation of two drainage schemes, covering approximately 3,370 ha, including Rioni Choloki and Rioni Khiobi; and (ii) provision of technical assistance for the design and supervision of the rehabilitation works . 3. Targeted Rehabilitation for National Amelioration Associations Program . This financed (i) targeted rehabilitation irrigation and drainage infrastructure in urgent need of repairs supplying services to selected schemes under the National Amelioration Associations Program; and (ii) provision of technical assistance for the design and supervision of the rehabilitation works . 2: Amelioration Association Development and Strengthening for Improved Irrigation and Drainage Management (US$10.37 million at appraisal; US$3.67 million at completion). There were 6 subcomponents: 1. Amelioration Association Support for Rehabilitation Program and National Amelioration Association Program . Each Association was to prepare a Development Plan covering its area of operation . The project was to finance the preparation of the Plan through provision of technical assistance for organizational support, engineering design, supervision and training for the development of the Associations, as well as on principles and practices of irrigation and drainage in agriculture; the project was also to support provision of equipment and the Association's operation and maintenance costs including office items, tools, and remuneration for Association staff in accordance with the provisions of the Development Plan . In addition, this component financed rehabilitation of tertiary and secondary irrigation and drainage networks in Association territories on about 20,000 ha under the Rehabilitation Program and 40,000 ha under the National Amelioration Association Program. 2. Amelioration Association Establishment Program . This financed (i) establishment of new Associations covering approximately 50,000 ha; and (ii) technical assistance for organizational support and training for the development of the Associations, as well as on principles and practices of irrigation and drainage agriculture. 3. Support for the Department of Amelioration and Water Economy of the Ministry of Agriculture and Food . This financed the provision of (i) technical assistance to the Department of Amelioration and Water Economy (DAWE) which soon after project start up became the Department of Amelioration and Scheme Management (DASM); and (ii) equipment and vehicles to regional DAWE offices in charge of management and infrastructure under Component 1 of the project. 4. Dam Safety Action Program . This financed the provision of assistance to begin implementation of a Dam Safety Action Plan. 5. Environmental Management . This financed implementation of mitigation measures included in the Environmental Management Plan (EMP) and provision of technical assistance for the monitoring and evaluation of its implementation. 6. Training and Study Tours . This financed the provision of technical assistance, training and study tours to Irrigation and Drainage Project Implementation Unit (IDPIU) staff, Associations established under the project and DASM in the areas of, inter alia, financial management and accounting, operation and maintenance planning and project administration . 3: Project Management . (US$3.43 million at appraisal; US$ 4.67 million at completion). This financed the provision of (i) technical assistance including audit services and preparation of the second phase of the Program (ii) equipment; and (iii) financing of incremental costs for the Project Coordination Center Core Services Team and the IDPIU. 4: Project Preparation Facility (US$1.2 million at appraisal; US$1.22 at completion). This was financed by the IDA credit. A fifth component was added in 2005 when the additional financing was approved . 5: Reconstruction of Flood -Damaged Irrigation and River Embankment Infrastructure (additional financing of US $16.1 million at appraisal; US$ 14.92 million at completion). This had two subcomponents : 1. Reconstruction of River Protection and Irrigation Infrastructure . This financed (i) reconstruction of flood-damaged irrigation and headworks and conveyance canals to restore and guarantee the original design capacities (ii) reconstruction of flood-damaged river embankment work to provide flood protection as per applicable design parameters; and (iii) supervision of design and construction . 2. Management and Implementation Support . In the wake of a substantial privatization of Georgia ’s public sector in 2006 the implementing arrangements for part (3) of component 2 were changed. The DASM in the Ministry of Agriculture and Food was withdrawn from the role of operating and maintaining higher order irrigation infrastructure and this function was taken over by four state -owned limited water companies which were to take responsibility for operating and maintaining the higher order irrigation water delivery systems in four regions where irrigation infrastructure had been established . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost : Total project costs increased from $ 32.8 million at appraisal to $51.84 million at completion. This was because additional financing from the Bank and the government was added to the project to address the impact of flood damage to irrigation areas, headworks, conveyance canals , and river embankments. Substantial depreciation of the US$ against the Special Drawing Rights (SDR) produced an additional $4.51 million, of which US$3.54 million was spent on rehabilitation of irrigation and drainage systems . In addition, this infrastructure component also benefited from reductions in the budget for strengthening the Associations . Final expenditure on infrastructure was almost twice the amount planned at appraisal (excluding the rehabilitation financed from additional financing ). Financing : The IDA Credit, approved for US$27.00 million equivalent, grew to US$31.51 million by project closing because of depreciation of the US$ against SDR (the currency of the Credit), and at closing US$0.98 million was cancelled. Additional Financing was provided following an Agreement Amending the Development Credit Agreement dated July 27, 2005, which resulted from two Board decisions . The first decision was on the basis of a Project Paper dated June 6, 2005 that proposed an additional IDA Credit of US$ 3 million and a Grant of US$2 million. The second decision was based on an Addendum to a Memorandum from the President of the Bank dated June 27, 2005 on the basis of which the Board approved a revision of the earlier Additional Financing to an IDA Credit of US$ 7.8 million and a Grant of US$5.2 million. By the end of the project depreciation of the US$ increased the Credit to US$ 8.29 million and the Grant to US$5.35 million. US$0.41 million of the Credit and US$0.10 million of the Grant was cancelled at closing. Overall, Bank financing was US$41.38 million. Borrower Contribution : Between the project's appraisal and closure the Government contribution increased from the $4.6 million planned to $9.83 million. In addition farmers contributed US$ 0.53 million, or 53% of the planned amount (US$1.20 million). Dates : The project was extended twice . First by one year from April 30, 2007 and second by an additional year to April 30, 2009. This was to allow additional time for completion of all contracts because the project had been delayed in 2003/2004 by the Rose Revolution and in 2006 by the change in irrigation development policy . 3. Relevance of Objectives & Design: Original Objectives Relevance of Objectives : Substantial Project objectives were and remain relevant to addressing the challenges of the agricultural sector . Because of its relative size and role in employment, growth of the agricultural sector is critical to Georgia's overall economic growth and prosperity. While agriculture as a share of the economy had declined from about 36% to GDP in 1999 to 9% in 2008 it still provided over half of all employment in 2008. Much of agriculture depends on full or supplemental irrigation but poor operation and maintenance had reduced the area under irrigation from 470,000 ha in Soviet times to 160,000 ha in 2000. Over the same period drainage similarly contracted for similar reasons from 163,000 ha to 20,000 ha. In 2004 self-employed agricultural wages were the lowest in the economy - a third less than the average wage - and rural poverty had increased from 11% in 1998 to 19% in 2004. The Government's 2001 strategy for irrigation and drainage in its Letter of Development program affirmed to relevance of objectives at Board approval . Despite the Government’s changes in irrigation policy, the relevance of the project’s objectives remained substantial even though there were serious doubts about the Government's ownership of the means promoted by the project to achieve project objectives . Project objectives remained relevant to the Bank's Country Partnership Strategies . The first pillar of the Strategy for FY06-FY09 emphasized generating growth and job creation while the third pillar focused on strengthening public sector management, including infrastructure such as roads and irrigation . The second objective of the FY10-FY13 Strategy (strengthening competitiveness for post -crisis growth) includes the need to improve agriculture productivity and the irrigation framework . Relevance of Design : Modest Given the challenges of agriculture and irrigation in Georgia, the use of an APL was realistic as it recognized that institutional change takes considerable time and that mutual lessons would be generated for the Bank and the Borrower. While the results framework described in the Project Design Summary (PAD, Annex 1) was logical, there was a lack of explicit links between inputs, outputs and expected outcomes related to project objectives. The project was expected to increase farm incomes : most of the 33,000 potential beneficiaries of this project were small-scale farmers with between 0.5 and 1.2 ha and they are typically amongst the poorest groups in Georgia. The activities to enable deteriorated irrigation and drainage systems to become operable and sustainable through improving beneficiary management and measures to safeguard upstream water supplies were comprehensive. The selection of five pilot projects for rehabilitation to enable lessons to be learned and serve as a model of second -phase schemes was prudent and realistic . Building the capacity of agricultural stakeholders (the members of the Amelioration Associations ) by making them responsible for planning and contracting was key to increasing their ability to diagnose problems, and to repair and manage infrastructure effectively and efficiently. Their autonomy and ownership was expected to increase farmer's willingness to pay for water and operation and maintenance costs . Water supplies were assured through attention to dam safety and building/streamlining the capacity of the state agency responsible for maintenance of headworks and the major supply canals. The use of international study tours to Turkey and Kyrgyz Republic (where user management was successful) was to designed provide peer knowledge to Associations of how beneficiaries operate irrigation and drainage schemes and pay for them . The design of measures to achieve the original objectives was changed in 2006 when a newly-appointed Minister of Agriculture and Food reduced the scope of the project to only rehabilitation activities (of main and a few secondary canal and irrigation and drainage infrastructure ) and withdrew support for development activities for the Associations. At the same time the Department of Amelioration and Scheme Management was withdrawn from the role of operating and maintaining higher order irrigation infrastructure and distributing bulk water supplies and this function was taken over by four new state -owned limited water companies that had negligible experience of the sector. This reduced the relevance of the activities to the objectives, seriously compromised the original design, and jeopardized the ability to achieve the project objectives. The ICR (para 31) noted that the lack of support for the Associations as intermediaries in the supply and management of water would cause the government's new water supply companies to commence dealing directly with thousands of individual water users (instead of the Associations ), a task the water companies are ill -equipped to do. Revised Objectives Relevance of Objectives : Substantial The new objective added in 2005 was substantially relevant because very large floods had caused considerable damage to unprotected irrigation areas and river embankments protecting them . The original objectives remained substantially relevant . Relevance Design : Modest The design of flood mitigation works were technically innovative and substantially relevant to reducing the risks of future floods in the project area . However, given the design shortcomings under the original objectives that continued, the overall relevance is rated modest . 4. Achievement of Objectives (Efficacy): The original objective was: To increase (A) agricultural production and (B) farm incomes on 110,110 ,000 ha by arresting further deterioration of irrigation and drainage infrastructure and keeping the operable infrastructure functional . The additional financing objective (C) was: (C) to improve and secure the sustainability of river flood protection works and irrigation headworks and canals . (A) Increase Production (Original and Revised Objective ): Modest Outputs : The planned rehabilitation of irrigation and drainage system rehabilitation of 20,000 ha was achieved. However, the final supervision reports the quality of some of the completed works has been marginally satisfactory, due to inexperience among Associations . The assistance to develop Associations in areas that were not part of the rehabilitation program area exceeded the PAD objective (228,000 ha compared with 50,000 ha), but further progress was terminated after 2006. Rehabilitation assistance to the national Associations (those not part of the rehabilitation program ) was also stopped in 2006 by which time it had achieved only 25% percent of its target of 40,000 ha. Only 35% of the training program was implemented due to the Government's refusal to support the development of the Associations in 2006. This policy change was detrimental to farmers' and Associations' self -reliance and contrary to the agreement between the Government and the Bank that there would be support for the Associations in the irrigation rehabilitation schemes as well as 25% of national Associations (ICR paras 24 to 31). In addition, the absence of support services to farmers such as extension has meant that the use of improved agricultural technology on farms has progressed very little (ICR paragraph 50). The dam safety program was implemented satisfactorily as was the environmental management program . In consequence, risks from these sources to water supply have been addressed (see ICR, parag 117). Outcomes : It was planned to increase production over 110,000 ha by an average of 25% as a result of yield increases in rehabilitated areas. The results reported in the ICR do not state what the average production increase was for the whole project, instead they are reported for only one project - the Kvemo Alazani scheme - that had 7,933 ha under operation. The average yield increase weighted by the share of each major crop's production value is 43% (ICR page 41). If this holds for the whole of the benefited area of about 30,000 ha, the pro rata production increase over the total target area of 110,000 ha would be about 11.7% - or half the target. As a result of the constraints listed above, crop yields increased over only 25% of the target area in fully rehabilitated irrigation schemes. In contrast, most of the farmers benefited from increased yields in areas where drainage was rehabilitated: The yield of three major crops (wheat, maize and grapes) increased by much more than the 25% target increase compared with baseline level yields . On the other hand it was estimated that only “about 5,000 farmers on about 4,000 ha out of the irrigation area rehabilitated (16,300 ha) have practiced full irrigation. In these areas yields of wheat were estimated to have increased from 1.5 ton per ha to 2.95 ton per ha and grapes from 4 ton per ha to 6.4 ton per ha� (ICR paragraphs 50 and 61). In the drainage systems more than 6,000 farmers have benefited from improved growing conditions for crops as a result of the project . Maize yields in these areas increased from an average of 2 tons per ha to 3.1 tons per ha (ICR paragraphs 51 and 61). However, as indicated in Section 5, there was considerable doubt about the accuracy of yield estimates . The government's institutional change in the management of the higher order irrigation systems (such as main delivery canals) by the dissolution of the Department of Amelioration Scheme Management and its replacement by four state owned limited companies in 2006 had predictable consequences . The companies are bankrupt and incapable of providing an adequate service (ICR paragraph 26). Irregular water supplies account for some of the losses to potential production (for example the Mtkvari Company blocked water deliveries to the Taskiskari subproject (ICR para 157). Where water supplies are insecure farmers are typically unwilling to pay water charges. Income to maintain irrigation and drainage production systems is inadequate and the fee collection rate has steadily declined. Over the period 2005-2008, the collection rate fell in the Alazani scheme from 36% down to 18%; in the Sioni scheme from73% to 36%; in the Mtkvari 42% down to 6%; in the Kolkheti scheme from 16% percent to 2%. The ICR notes (para 178) that these declining Association membership fees coincide with the institutional changes made by the Ministry of Agriculture in 2006. As a result of the government's policy change, Associations are weak and unable to raise funds from water users to enable them to operate and sustainably maintain the tertiary irrigation systems (ICR paragraph 26). (B) Increase Farm Incomes (Original and Revised Objective ): Negligible Outcomes : The results of the financial analysis for typical farms in the project area presented in the ICR show that incremental annual net revenues on the very small farms (0.5 to 1.2 ha) in the irrigation areas were positive and ranged from 3% in the Rioni Choloki drainage scheme to 14% in the Tashiskari irrigation scheme . How far this is due to the project is unknown as there were no without -project controls. Most families supplement income from production on their irrigated land by keeping livestock (poultry, pigs or cattle) especially in the drainage areas where cattle are allowed to graze the arable land after harvest � (ICR para 167). However, these improvements in "net revenues from smaller holdings are insufficient to assure minimum living standards for a family� (ICR para166). Considering that farm incomes may only have been improved over a third of the target area, weighted average incomes may only have increased in the range 1% to 5%. (C) Improve and secure the sustainability of river flood protection works and irrigation headworks and canals (Revised Objective ): Modest Outputs : Works included protection of river banks with river training structures (spurs and embankments) using rocks, gabions, and flexible rock mattresses . In total, 16 km of earthen embankment was reconstructed, 6 km was strengthened with stone lining, and 3 km of spurs were constructed . Flood damage to various main structures at six irrigation schemes commanding in total about 36,000 ha was repaired. The ICR reports flood protection infrastructure was of "good quality and strength and is serving well ". Outcome : More than 20 villages with a population of over 40,000 people have increased security against flooding (ICR, para 53). The newly reconstructed infrastructure also provides protection to the water supply system of Tbilisi and roads. The ICR notes that "the completed infrastructure has been able to withstand floods with ease " (ICR, paragraph 117). While the Country Director has advised that these floods were "considerable", there is no evidence that the objective of securing the sustainability of river flood protection works was achieved . 5. Efficiency (not applicable to DPLs): Original Objectives : Modest The infrastructure investments were cost -effective. The counterfactual was that irrigated and drained areas would continue to decline without the project, and that crop yields would not increase . In practice, the project effectively stopped further deterioration, but on a much smaller area than planned, and in those areas it facilitated increased crop yields . As a result the ICR concluded that the overall economic rate of return (ERR) for the project infrastructure investments when completed (without the later flood control component ) was 20% compared with a calculated 24% at appraisal (PAD page 68). This is based on an analysis of the four rehabilitation projects: the ERRs for the Kvemo Alazani, and Tashiskari were respectively 27% and 21% at completion compared with 20% 35% at appraisal. For the drainage systems the ERRs for the Rioni Khobi-Chaladidi and Rioni Choloki-Lanchkuti were 39% and 40% at completion compared with 39% and 36% respectively for these two schemes at project appraisal . Despite the positive results reported above, the veracity of the economic analysis is questionable . Although a narrative is provided in both the PAD and the ICR on the basis for estimating rates of return for sub -projects, there is no information about how the overall ERR was calculated in either the PAD or the ICR . There is no evidence that a baseline survey was conducted for the areas where irrigation and drainage rehabilitation took place. In addition, the ICR (Annex 3) states that the "without the project baseline yield data " used for the ERR analysis came from the PAD (see ICR, page 39) but despite statements that PAD yields were adjusted "to allow for actual command areas rehabilitated ", the reliability of the PAD baselines is questionable because they were themselves estimates. The Task Team Leader clarified that the yield estimates "with the project" were based on assessments by monitoring and evaluation consultants and project management unit staff . In addition the rehabilitated area actually cultivated and achieving higher yields was estimated on the basis of “local reports and information collected during the ICR mission on April 2009� (ICR paragraph 147) and was not based on a formal survey. Notwithstanding the relatively good economic rates of return for infrastructure rehabilitation, the overall cost-effectiveness of the project was modest . Project funds were channeled into rehabilitation (that cost almost twice appraised costs) while support for essential capacity -building and institutional development to maintain this infrastructure was cut back by 70%. This severely compromised the overall efficiency of Georgia's irrigation and drainage institutions. The Government's decision to establish the new water companies and abandon the development of the Associations was not cost -effective. The companies are "grossly under-resourced and practically bankrupt and therefore unable to provide more than a fire -fighting service" (ICR para 26) and, at the project's close, the Associations were in a state of imminent collapse from lack of support (ICR para31). Revised Objectives : Modest The efficiency rating for the original objectives remains modest . There was no rate of return estimate for the flood repairs and prevention component . The apparent technical achievement of the objective of this component is not evidence of satisfactory cost -effectiveness because estimated losses avoided by the flood prevention works have not been estimated. Thus the efficiency of the additional objective is rated modest . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 24% 99% ICR estimate Yes 20% 60% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Original Objectives : The relevance of the original objectives was substantial, while the relevance of design was modest. The efficacy of the project in raising agricultural production was modest and the efficacy in terms of increased farm incomes was negligible . While the revised ERR estimates for the rehabilitated irrigation and drainage areas in the project were comparable to the ERRs estimated at appraisal, there are many questions about the reliability of the revised estimates . In addition, the project's use of funds was inefficient because the decision to reduce institutional support in favor of increased expenditures on infrastructure was not cost-effective. Overall efficiency is thus rated modest . On balance there were significant shortcomings in the project's achievements and hence its outcome is rated moderately unsatisfactory . Revised Objectives : The revised objectives were substantially relevant . While the flood protection activities enhanced the relevance of design to the revised objectives, this was outweighed by the modest relevance of design to the other two (original and revised) objectives, thus relevance of design is rated modest . The ratings for the efficacy of the two original objectives that remained to the end of the project are modest and negligible . The efficacy of flood mitigation was modest due to inadequate outcome evidence . Overall efficiency was modest. On balance. there were significant shortcomings in the project's achievements and hence its outcome is rated as moderately unsatisfactory . Given that the two outcome ratings under the original and revised objectives are identical, there is no need to consider a weighted average of these outcome ratings . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: There are a number of risks regarding the sustainability of the outcomes achieved from this project . Weaknesses of the state -owned water companies . The companies did not have adequate capacity to maintain the irrigation works and systems and there is "already evidence of the deterioration of some works that were rehabilitated under the project, and future sustainability is unlikely without additional resources from the Government" (ICR, paragraph 26). Uncertainty about the future support for the Amelioration Associations . It is not clear how much future support there will be for the existing Associations (ICR, paragraph 46). Unless there is a change in policy the irrigated sector will remain depressed, Associations will not be encouraged, small-scale farmers will not be in a position to improve their productivity and incomes, and the financial sustainability and maintenance of irrigation systems will continue to be jeopardized. Low water charge collections . Farmers are currently not contributing sufficiently towards operation and maintenance and hence the sustained operation of these systems is at risk . The Task Team leader advised that water charges by the water companies were probably justified in terms of actual costs even though fees were far higher than envisaged at appraisal . However, low water fee collection rates are influenced by a sudden increase in fees, poor water delivery services by companies, and because farmers can choose not to use irrigation water either by relying on rainfall or diversifying crops . Clearly, if low collection rates for water charges continue, sustainability of the irrigation schemes will be at substantial risk . The Government indicated its preparedness to reconsider changes in its policies on Associations . At the close of the project the Ministry and senior management of the state -owned water companies highlighted the importance of the Associations to the effective management of the irrigation and drainage systems (ICR, para 30). Flood mitigation works have been highly successful . The flood control measures (using a technology that has proven to be better than the traditional technology used in the former Soviet Union ) are expected to provide sustained flood protection benefits . On the other hand their efficiency is not confirmed . In light of these perceived risks the Bank put its support for the proposed next two years of the Irrigation and Drainage Revitalization on hold. This Review considers that, despite some positive indicators that damaging decisions on irrigation policy during this project's implementation will be reversed, there remains a significant risk that development outcomes from this project will not be sustained. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: Quality at Entry . The project was well prepared and it broke new ground in Georgia with its focus on the rehabilitation of irrigation and drainage systems and improving institutions with the development of Associations as a core organizing principle . Arrangements for procurement and financial management were sound . The financial management capacity was adequate and acceptable to the Bank due to strong staff and adequate internal control systems in the Borrower's project management unit (ICR, paragraph 38). The PAD's risk assessment was comprehensive . It mentioned the risk of Government interference in the management of Associations and concluded that it was unlikely that the Government would get involved in management of successful water users associations . In hindsight the Bank underestimated the level of irrigation fee increases required and its impact on farmer's participation in irrigation . The M&E arrangements at appraisal were deficient also and this delayed work on baseline data which subsequently resulted in inadequate documentation of the project's performance. Thus, overall, there were moderate shortcomings in the project's quality at entry . Quality of Supervision .There was continuity of the Task Team during the life of the project . The changes in Government policy early in 2006 led to an intensive series of discussions with Government counterparts which ended in a compromise that allowed the project to move forward . The ICR states that “In retrospect, stronger efforts to either close the project or elevate it in the Bank ’s policy dialogue might have been justified � (ICR, paragraph 78). On the other hand this Review does consider that, having patiently reached a compromise on the role of Associations, the Bank was provided with the opportunity to show the potential effectiveness of the Associations as well as the problems associated with the ill -considered establishment of the water companies by completing the project instead of closing it . Closing the project in 2006 would have eliminated the possibility of such an opportunity. The experience gained by completing the project should lead to a better irrigation policy in Georgia in the future. However, there were weaknesses in M&E by the project management unit that the Bank's supervision missions did not adequately address during project implementation . The project's performance in the irrigation areas could not be adequately assessed and no data were gathered for an evaluation of the benefits of the flood protection works. Thus, overall there were moderate shortcomings in the quality of the project's supervision. at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Moderately Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: Government Performance . The Government accepted the irrigation development model proposed at appraisal and effectively supported the project's implementation at a high standard during the first three years after project effectiveness. The Government's change in irrigation policy in early 2006 was was based on an ideology and an organizational arrangement that had not been analyzed or tested . While the Bank was able to negotiate a compromise on the rehabilitation of irrigation works and the continued (albeit limited) development of the Associations with the Government, the Bank was not able to convince the Government to undertake a pilot to test the viability of the changed arrangements for the management of higher order water distribution using the water companies. In the event the water companies proved incapable of undertaking the task and this failure also had serious negative implications for the Associations . The ICR (para 86) notes "it was clear in retrospect that the Government did not believe in the compromise and did not show adequate support to make it work " There were significant shortcomings in Government performance . Implementing Agency . During implementation the project management team was in the Ministry and made up of the Engineering Management Team and the Institutional Management Team . The fiduciary staff was located in the Project Coordinating Center which supported all Bank -funded projects in the Ministry . The staffing of engineering team was more or less constant but the staffing of institutional team was reduced following the 2006 change in irrigation policy. The ICR concluded that although the outcome of the institutional development component of the project has been unsatisfactory, this is not considered to be the fault of the institutional team, who endeavored to continue support to the Associations where they could, but with limited manpower resources or manpower (ICR, para 85). There were moderate shortcomings in the performance of the implementing agency. a. Government Performance :Moderately Unsatisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: Design . The PAD had stated that monitoring and evaluation should be kept separate with monitoring done by the PIU and evaluation done by an outside entity (PAD, page 22). On the other hand the ICR states that "The intention at the start of the project was that project monitoring and evaluation would be carried out by the M&E Specialist within the...Project Coordination Center, but for lack of sufficient capacity this was never set up " (ICR, para 33). No intermediate objectives or indicators were defined and neither was the timing of expected outputs or outcomes . There was also no evidence in the ICR of a baseline survey . For the objectives to increase agricultural production and farm incomes there was only one quantitative indicator, namely “crop yields for wheat, grapes and maize on irrigation and drainage schemes (would be) increased by 25 percent from pre-rehabilitation levels, two years after rehabilitation is completed.� Subsequently the Task Team defined an intermediate indicator relevant to the registration of Associations and the training of their members, final outcome indicators for crop yields, irrigation water supplied to Associations, as well as a final outcome indicator for the repair of flood damage to irrigation headworks, irrigation canals and social infrastructure . Implementation . Although the M&E responsibilities were transferred because of poor management from the project coordination center to the project management unit this arrangement was no better . The ICR states that "it was difficult to provide reliable and adequate results for the project's key performance indicators (paragraph 33). The detrimental implications of this performance have already been referred to in the assessment of the project's efficiency. Utilization . Although the results of the monitoring and evaluation have been used in the evaluation of the project's results they did not have a significant impact because of uncertainties about the veracity of the data . a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Safeguards : The original project and the Additional Financing triggered three safeguards : OP 4.01 Environmental Assessment in which the project was classified category B; OP 4.37 Safety of Dams; and OP 7.50 Projects on International Waterways. Environmental Assessment: The project complied well because there were specific objectives in the project which addressed Environmental Management (including water quality), and project contracts complied with environmental guidelines. Safety of Dams: A dam safety action program was financed and successfully . implemented under the project including action taken to secure one dam . International Waterways: Investments in relation to flood mitigation did not affect the flow or volume of the rivers and there was no additional abstraction . Fiduciary Issues : There were no fiduciary issues or problems during project implementation (see ICR, paras 35 to 39). There was no evidence in the ICR about qualified audits although they were received with some "minor delays" (ICR, para 39). Unintended Positive and Negative Effects : None 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Unsatisfactory Unsatisfactory Risk to Development High Significant This Review, although also concerned Outcome : about the risk to the future development outcome of this project, sees some scope for more optimism than the assessment in the ICR . There are two reasons for this: (a) the Government has indicated its preparedness to reconsider changes in its policies on Associations; and (b) the flood control techniques used in this project were considered to be better and more sustainable than traditional techniques . Bank Performance : Satisfactory Moderately Quality at entry was moderately Satisfactory satisfactory because of a weak results framework as well as an inadequate M&E design and implementation. The quality of supervision was also rated as moderately satisfactory because the M&E weaknesses were not satisfactorily resolved. Borrower Performance : Moderately Moderately Unsatisfactory Unsatisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The ICR provided a number of relevant lessons, of which the most important were : Government support is key for successful project implementation . Without a suitable institutional framework to manage, operate and maintain irrigation and drainage infrastructure, investments in rehabilitation will only have a short term impact. The following lessons could have been added : The project's history shows that it is well worth devoting adequate time at the appraisal stage to ensuring that a project is based on sound policy . Although the Government reversed policy during implementation the project, subsequent experience clearly demonstrated merits of the original policy and influential officials agreed at the project's close that policies should not have been changed . This suggests that solid project preparation and the defense of the original strategy paid off . Weak M&E can lead to serious problems in evaluating the project results . Strenuous efforts during supervision should be made to fix M&E weaknesses . A serious assessment of appropriate cost recovery rates for irrigation systems is essential to highlight the risks of not doing the needed institutional reform . Inadequate cost recovery for irrigation water was a chronic problem for the Associations but the draconian solution by the Government of establishing the water companies and then increasing cost recovery rates 12-fold was not a solution either . 14. Assessment Recommended? Yes No Why? It provides a classic example of Bank performance when Government policies change substantially during project implementation. Reviews of projects with similar policy challenges would provide an opportunity to see how the policy challenges between the Bank and the Government on irrigation policy played out and a basis for evaluating the Bank's resolution strategy. 15. Comments on Quality of ICR: The ICR arrived at well reasoned ratings of the project's performance according to the standard indicators . There was adequate evidence in the ICR on which to base its conclusions, although for some data (e.g. crop yield) it was not available because of the weak M&E performance in the project . Nevertheless, the annexes contained a considerable amount of relevant and useful data . The discussion in the ICR was comprehensive, particularly on the Government's policy changes which were major issues for the project . The ICR was, however, vague on the underlying problems and reasons for the weak M&E system and what should have been done to solve the problems .The economic analysis in the ICR was not clear on issues such as the incremental yields and areas that benefitted from irrigation rehabilitation which, together with prices, determined incremental benefits. As a result the basis for the ERR estimates was not transparent and could also not be checked, since summary worksheets were not available in the ICR . The basis for the overall rate of return for the project (presented in one sentence ) was completely unclear. No reason was given why the ICR did not undertake an economic analysis of the flood mitigation work despite the fact that, following the additional financing, it was a significant part of the project . While the ICR structure was consistent with the guidelines and was internally consistent, it tended at times to be too descriptive. For example the section on Implementation (section 2.2) devoted nine paragraphs to a description of implementation with only one quantitative fact (the shortfall in the implementation of the national Amelioration Associations development program ). The Bank's quick action in providing flood mitigation assistance was clearly successful and the lessons drawn from this experience should have been listed in the ICR . a.Quality of ICR Rating : Satisfactory