ICRR 12841 Report Number : ICRR12841 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 03/18/2008 PROJ ID : P043933 Appraisal Actual Project Name : Sichuan Urban US$M ): Project Costs (US$M): US$346.7 M US$179.1 M Environment Project Country : China Loan /Credit (US$M): Loan/ US$M ): US$150 M - IBRD US$44.2 M - IBRD and US$2 M - IDA and US$0.7 M - IDA Sector Board : WS US$M ): Cofinancing (US$M): Sector (s): Sewerage (51%) Water supply (38%) Sub-national government administration (8%) Solid waste management (2%) Other social services (1%) Theme (s): Other urban development (23% - P) Water resource management (22% - P) Pollution management and environmental health (22% - P) Municipal governance and institution building (22% - P) Municipal finance (11% - S) L/C Number : C3251; L4496 Board Approval Date : 06/17/1999 Partners involved : Closing Date : 06/30/2006 01/31/2007 Evaluator : Panel Reviewer : Group Manager : Group : John Redwood Roy Gilbert Soniya Carvalho IEGSG 2. Project Objectives and Components: a. Objectives: To provide a safe environmental setting for the long -term economic growth of urban areas in Sichuan Province (SP). b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): The Board-approved project was formally amended before loan signature to delete two previously planned components for the city of Zigong (also in the Tuo River Basin): (i) water supply (US$ 66.8 M according to the PAD) and (ii) watewater (US$ 35.2 M), as Zigong decided to withdraw from the project . The remaining components once the loan became effective were : A. Min River Basin 1. Chengdu Wastewater (appraisal: US$ 78.0 M; actual: US$ 90.9 M) - construction of a 300,000 m3/d wastewater treatment plant (WWTP), interceptors and secondary sewers and ancillary facilities . 2. Leshan Water Supply (appraisal: US$ 19.0; actual US$ 0) - augmentation of a new water supply source of 100,000 m3/d capacity including treatment, networks, and ancillary facilities . 3. Leshan Wastewater (appraisal: US$ 15.7 M; actual - US$17.0 M) - construction of interceptors, secondary sewers, and a WWTP of 100,000 m3/d capacity with preliminary treatment . 4. Leshan Municipal Solid Waste (appraisal: US$ 7.9 million; actual: US$ 8.5 M) - construction of a 300 tons/d sanitary landfill, collection and transfer facilities, and vehicles and equipment, including landfill equipment . B. Tuo River Basin 1. Deyang Wastewater (appraisal: US$ 27.2 M; actual: US$ 22.0 M) - construction of sewer networks, interceptors and a WWTP of 100,000 m3/d capacity. 2. Luzhou Water Supply (appraisal US$ 23.1 M; actual: US$ 11.5 M) - (a) Luzhou-Beijao - construction of a 50,000 m3/d treatment plant, transmission mains and distribution networks; (b) Luzhou-Daxihou - construction of a 50,000 m3/d treatment plant from a new intake, transmission mains, storage reservoir, and distribution networks C. Other Components 1. Urban Management and Environmental Management Information Systems (UMIS and EMIS)) (appraisal: US$ 3.9 M; actual: US$ 3.5 M) - development of an UMIS including decision support systems for the Sichuan Construction Commission (SCC), Deyang, Leshan, and Luzhou, and an EMIS for the Environmental Protection Bureau (EPB). 2. Environmental Protection and Rehabilitation (aprpaisal: US$ 6.4 M; actual - US$ 3.5 M) - improvement of infrastructure and facilities at the World Heritage Site of the Leshan Grand Buddha, including presentation and management of cultural heritage assets . 3. Institutional Development, Strengthening, and Training (aprpaisal: US$ 6.7 M; actual - US$ 0.9 M) - technical assistance (TA) comprising institutional and financial management systems for the utility agencies, design review, construction supervision services and training . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: The amended project cost after the two Zigong subcomponents were eliminated was reestimated at US$ 218.9 million. The corresponding Bank loan amount was reduced from US$ 150 M to US$ 100 M prior to signature and effectiveness. The amounts allocated (about US$ 42 M of the IBRD loan) for two other subcomponents (Leshan and Luxhou-Daxikou water supply) which were dropped during the early stages of project implementation were canceled, but only at the very end of the project, as borrower proposals for alternative investments to be financed with the unused balance arrived too late to be accommodated by the Bank . The Chengdu Drainage Company (CDC) also used large amounts of its own funds to finance works under its component . As a result of these changes, only US$ 44.2 M of the loan was disbursed, with US$ 54 M canceled in November 2006 and January 2007 and the remaining undisbursed balance of US$ 1.8 M was canceled at the time of loan closing . 3. Relevance of Objectives & Design: Project objectives were relevant given the rapid rate of urbanization in SP and the serious problems associated with the discharge of untreated domestic and industrial wastewater and agricultural run -off into the rivers in the Province, in particular the Min and Tuo Rivers which flow into the Yangtze . They were also consistent with provincial and central government policies on improving water quality in major rivers and lakes and promotion of development in the western part of the country . The focus on Chengdu, the largest city in SP with a population of about ten million, was also appropriate. The deteriorating water quality in the Province was not only seen as a possible limiting factor on future growth of its three largest cities but reportedly posed a threat to the Three Gorges Reservoir downstream . Project design was also relevant, although greater attention might have been given to on -site pretreatment of industrial effluents rather than exclusively on the construction of municipal wastewater treatment plants, whose effectiveness and longer-term sustainability would both be adversely affected if they would also have to process raw industrial pollutants together with domestic sewage . The project's overall development objective, however, as stated in the PAD, was excessively broad . Providing a "safe environmental setting for the sustainable long -term economic growth of urban areas in SP, " necessarily requires much more than addressing some -- no matter how high significant -- water pollution problems, although this was perhaps the most significant local priority from an environmental and a public health, as well as economic growth, perspective. However, there may also have been relevant air quality and land degradation problems in the Province, which were not specifically addressed by the project . A more realistic -- and monitorable -- project objective, therefore, would have been something like : 'to extend the provision of clean water and improve water quality and solid waste management in the largest urban agglomerations and adjacent river basins in SP ." Another important environmental issue mentioned in the ICR but not directly addressed in the project refers to adequate water flow regulation and seemingly excessive irrigation water use in the Min and Tuo Rivers, which, together with associated pollution, also contribute to poor water quality in downstream urban centers and below . Nor did the project address agricultural non-point pollution, which is specifically identified in the PAD as a major problem in SP . Project relevance would have been greater from an environmental quality perspective, therefore, had water quality and associated water resource management issues in the two river basins been addressed in a more holistic way, taking into account land and water use in these basins more generally . In short, as the project dealt with only a part of the broader problem, it is not surprising that water quality levels did not improve (Min River) or actually deteriorated (Tuo River). It is also unclear from the ICR how much the project itself contributed to a reduction in industrial pollution in SP, although some progress was made in this regard, including the reported closure of one highly polluting factory and substantial decreases in that of three others during the project implementation period . Thus, there is an attribution problem here that the ICR does not elucidate . 4. Achievement of Objectives (Efficacy): The project's objectives, as translated into specific investment targets, were only partially achieved . While it is important to recognize that a number of the dropped components were, in fact, iimplemented using non -project resources, the project itself achieved much less than it originally set out to do . Project technical assistance / institutional development measures also appear to have been only partly effective . From this perspective, the ICR's "Moderately Satisfactory" rating appears justified. It is also justified insofar as actual water quality improvement results were mixed; while Min River water quality at Chengdu and Leshan remained stable, that in the Tuo River at Deyang worsened. Similarly, while the project resulted in the expansion of water supply in Luzhou, it did not do so in Leshan (subcomponent dropped); however, it did generate solid waste collection and disposal improvements at Leshan. Finally, while the ICR reports that "GIS-based urban management informartion systems for the SCC and other cities, and the environment management information system for the EPB were completed, " it does not indicate how -- or even whether -- they were actually being used in practice . 5. Efficiency (not applicable to DPLs): The ICR notes that the subcomponent for which the ex -post ERR was calculated was not included in the ERR estimate at appraisal. Moreover, while an ERR of 10-12% for the original project (including the Zigong, Chengu, Deyang, Leshan, and Luzhou, subcomponents ) was indicated in the PAD, no ERRs as such are reported in the PAD for the individual subcomponents (although FIRRs before taxes were reported, ranging from 9.4 to 11.5 percent, covering nearly 93 percent of the base costs of the original project . The ICR also reports ex-post FIRRs ranging from 10.6 percent to 19.7 percent, all of which were higher than the PAD estimates for the respective subcomponents . The overall efficiency rating, therefore, is Satisfactory, although the economic analysis itself in the ICR is inadequate . The largest investment component, the Chengdu wastewater treatment plant, suffered a 44 percent cost overrun, but the reasons for this are not explained in the ICR . Finally, there was no attempt either ex -ante or ex-post to assess health-related benefits of decreased exposure of the population -- and especially those of lower incomes -- to urban water pollution even though such benefits were explicilty identified in the PAD as part of the main rationale for the investments supported. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 11% 92.8% ICR estimate Yes 20.5% 6.4% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: The ICR rating is appropriate since the project achieved only in part what it originally set out to do . In addition, the project's water quality targets were only partially met and nothing is said in the ICR about the project's impact on water quality in the Yangtze River, which presumably was minimal . Finally, the project's development objective as stated in the PAD was far too broad to be met by project investments alone and should have been framed in more specific and realistic terms. a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: Service operation appears to be proceeding well . However, the ICR correctly points to the need for beneficiary cities to further improve cost recovery by adjusting tariffs upward in order to assure adequate future O & M of the facilities constructed under the project . Further investments in wastewater treatment and consistent and perhaps stricter enforcement of industrial pollution abatement legislation by the provincial EPB as well as addressing water resource management in a more holistic way (including water extraction for agricultural purposes and associated run-off) will also be necessary to assure better water quality results in the future . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Rated satisfactory, although the Bank could perhaps have better anticipated the significant downsizing of project scope both before and after effectiveness . In addition, the project's development objective should have been defined more precisely or project design should have been broader to cover other relevant urban environmental problems including air quality and non -industrial and domestic sources of water pollution in the Min and Tuo River basins. For this reason, quality at entry is rated only "moderately satisfactory." The ICR also points to other shortcomings including the need for better communication with project implemeting agencies at the municipal level and for "more meaningful assessments of project outcomes and supervision with a focus on results [which] would have been possible had the project design included key performance indicators with quantitative targets and baseline conditions ." Supervision was satisfactory and reportedly especially helpful with respect to preparation of "the cultural heritage development plan and for solutions in developing the access walkway to the Leshan Grand Buddha site, " although one can question how high a priority this was from an urban environmental perspective. This component seems to have been an "add on" not very clearly associated with the project's development objective . at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Satisfactory 9. Assessment of Borrower Performance: The Bank may have been subject to a "bait and switch" strategy on the part of SP and /or one or more of the intended project beneficiary cities, whereby Bank assistance was utilized for project preparation, but significant investments, which would have been financed under the original project design, were subsequently dropped from it and funded from other sources . While this may have represented an understandable -- and laudable -- cost savings as far at the borrower was concerned, it nonetheless meant that some of the preparation /appraisal resources spent by the Bank were ultimately in vain . Although the ICR is more positive in this regard, the two year effectiveness delay, together with the significant deletions of project subcomponents and associated cancelations, suggests weak ownership on the borrower's part . The ICR also points to specific shortcomings on the government side that adequately justify the "satisfactory" implementing agency and "moderately satisfactory" overall borrower performance ratings . a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: Design of the M & E system "left much room for improvement," according to the ICR, but some enhancements were made during implementation. However, the ex-post economic analysis was incomplete, suggesting the lack of needed monitoring information, and better use could have been made of results -oriented performance indicators . It is unclear how systematic and comprehensive the water quality monitoring associated with the project were . According to the ICR, "the EPB regularly monitored industries and river water quality, " but few concrete monitoring results are reported. Some figures are presented on total municipal and industrial discharges to "regional rivers" over time, but these data in and of themselves are not particularly helpful in determining the project's actual impact on water quality near specific project cities . Furthermore, despite the fact that the project objective specifically refers to protecting public health, there appears to have been no effort to monitor and evaluate such benefits . a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Although this is a Category A project for purposes of Environmental Asssessment and also involved some involuntary resettlement, the ICR does not discuss these aspects of the project with any real substance . The ICR mentions that a PHRD implementation grant of US$ 400,000 was provided for independent monitoring of safeguard implementation, but only slightly more than half of these resources (US$ 221,765) were used. No reasons are given for this shortfall. Based on the limited concrete information provided int he ICR, it is, thus, impossible for IEG to make an independent judgment in this regard . Fiducuary measures were reportedly also adequate . However, the ICR observes that, while project financial management was satisfactory overall, "the auditors did highlight slow disbursement and shortage of counterpart funds in several reports ." No significant unintended positive or negative impacts are reported in the ICR, but it does affirm that the progressive decrease in project scope both before and after effectiveness was almost fully "unexpected." 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Moderate Moderate Outcome : Bank Performance : Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: In addition to lessons indicated in the ICR (and not repeated here): (i) project development objectives should be formulaed to make it very clear what the project is actually designed to accomplish and to provide a more realistic picture of what is achievable; (ii) projects such as this one that seek in part to protect "public health, especially of lower income groups" by reducing water pollution should include adequate monitoring and evaluation mechanisms to determine the extent to which they actually generate the desired public health benefits for low income groups and this should also be reflected in both the ex -ante and ex-post economic analysis; and (iii) greater attention should be given during preparation, implementation, and ex -post reporting to project evironmental and social aspects and results, including those associated with application of Bank safeguard policies . 14. Assessment Recommended? Yes No Why? The Bank's urban environmental portfolio in China is large and particularly interesting . Thus, this project could be part of a broader cluster assessment of several such recently completed projects . In addition, the ICR leaves a number of unanswered questions concerning the adequacy of the approach reflected in the project design from a broader environmental management and quality standpoint, both at the urban and provincial levels, the actual nature and extent of its results and impacts on water quality and public health, especially among the poor, both in and near the project cities, and in the larger Yangtze River basin and Three Gorges Reservoir, which could be usefully further explored, as could the implementation, actual use and possible impact of the non -physical investment components of the project. 15. Comments on Quality of ICR: This is a generally comprehensive and well -written ICR. But, it also has several shortcomings . It could have presented a more comprehensive ex -post economic analysis of the project -- although in fairness, it is not clear that the economic analysis at the time of appraisal was really much better -- and been more critical in its discussion of the project's development objective as stated in the PAD (too broad to be readily achieved through a single operation or even adequately monitored and evaluated ) and design (too narrow to achieve the stated develoment objective ). Greater attention could have been given to performance of the "software" or institutional development components and to the process and results of application of Bank safeguard policies to mitigate potential adverse environmental and social impacts. The ICR should also have explained the substantial difference in actual costs versus appraisal estimates for several subcomponents where there were significant differences between the two (e.g., Chengdu wastewater, where there was a significant overrun -- 144% actual vs. estimated costs), and Luzhou water supply (50%), environmental protection, and rehabilitation (57%), and insititutional development, strengthening, and training (just 16% of the costs estimated at appraisal ) where there were significant underruns . For the most part, however, its ratings seem correct and well-justified, although quality at entry from an environmental perspective was less satisfactory in practice than indicated by the ICR . Had this option been available, ICR quality would have been more accurately rated as "Moderately Satisfactory." a.Quality of ICR Rating : Satisfactory