Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-GEF-IF-Shanghai Agricultural &Non-Poi (P090376) Report Number : ICRR0020154 1. Project Data Project ID Project Name P090376 CN-GEF-IF-Shanghai Agricultural &Non-Poi Country Practice Area(Lead) China Water L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-97173 30-Jun-2014 34,865,000.00 Bank Approval Date Closing Date (Actual) 10-Jun-2010 30-Jun-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 4,788,000.00 Revised Commitment 0.00 4,788,000.00 Actual 0.00 4,788,000.00 Sector(s) General agriculture, fishing and forestry sector(50%):Public administration- Agriculture, fishing and forestry(21%):Wastewater Treatment and Disposal(13%):Agricultural extension and research(9%):General water, sanitation and flood protection sector(7%) Theme(s) Other rural development(63%):Rural services and infrastructure(25%):Pollution management and environmental health(12%) Prepared by Reviewed by ICR Review Coordinator Group Judith Hahn Gaubatz John R. Eriksson Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives According to the Project Appraisal Document (PAD, page 3) and the Grant Agreement (page 6), the project objective was as follows: • To demonstrate effective and innovative pollution reduction activities in Shanghai's rural areas in order to reduce the rural and agricultural pollution load (especially nutrients) in the surface water flowing to the East China Sea. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-GEF-IF-Shanghai Agricultural &Non-Poi (P090376) b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Components The project aimed to demonstrate eight technologies for four different types of waste management and pollution reduction. 1. Livestock Waste Management Technology Demonstration (Appraisal: US$ 9.75 million; Actual: US$ 3.75 million): This component aimed to demonstrate three technologies to more effectively manage livestock waste. 1) Large-scale farm demonstration: A dairy waste treatment facility was to be established on the Jinshan Dairy Farm, consisting of a solid/liquid waste treatment equipment and a biogas generator. Organic fertilizer and biogas energy were to be generated by this activity. 2) Medium-scale farm demonstration: A livestock waste treatment center was to be established for the Shenye Cooperative, consisting of a composting and pelletizing system and anaerobic liquid waste equipment. 3) Integrated Village-level demonstration: An integrated agricultural and livestock waste treatment center was to be established in Qianwei Village, consisting of equipment to treat livestock waste, crop straw, municipal solid waste, and rice chaff. 2. Wetland Demonstration for Pollution Reduction (Appraisal: US$ 3.34 million; Actual: US$ 4.59 million): This component aimed to demonstrate two technologies to mitigate water pollution due to agricultural and household waste. 1) Rural Town River-Network demomstration: An artificial wetland was to be constructed on a sub-project site comprised of 66.5 hectares in a rural area, while also restoring the existing river wetland ecological network (including riverside belts and vegetation buffers to stop sediment run-off and river erosion); 2) Village Wetland Sewage Treatment System demonstration: Household waste treatment facilities were to be established for four villages, including restructuring improper septic tanks, constructing of sewage pipe networks and waste treatment facilities. 3. Integrated Agricultural Pollution Reduction Techniques (Appraisal: US$ 20.4 million; Actual: US$ 20.4 million): This component aimed to demonstrate three pollution reduction technologies in agriculture. 1) Use of organic fertilizer: This activity was to demonstrate innovative techniques to reduce utilization of chemical fertilizers, including accurate fertilizer application; 2) Use of agricultural chemicals: This activity was to promote reduction of pollution from insectides and pesticides by using low toxic, eco-friendly biological pesticides, and non- chemical technologies; 3) Monitoring and extension: This activity was to regularly monitor samples to assess the effect of the technologies being demonstrated, and to train farmers and technical staff to identify successes and best practices and promote new technologies. 4. Project Management and Dissemination (Appraisal: US$ 1.18 million; Actual: US$ 1.14 million): This component aimed to to develop implementation capacity of the various implementing entities (including through the establishment of a monitoring and evaluation system), to develop replication strategies based on demonstrated successes, and to provide training to farmers and participating entities. Under the July 2013 project restructuring, the following revisions were made: • Component 1: The biogas power generation activity was dropped, as the client considered it too technologically advanced and expensive. Instead, the wastewater treatment plant was expanded in capacity from 450 tons per day to 800 tons per day, and the cattle waste composting yard was expanded. • Component 2: The original location of the rural river-network demonstration site (66.5 hectares) was changed to a new site (2.2 hectares) due to a change in the government's land use master plan. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost • The total appraised cost of the project was US$ 34.68 million. Due to project restructuring and dropping of several sub-components, the total Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-GEF-IF-Shanghai Agricultural &Non-Poi (P090376) actual cost was US$ 29.88. • The actual cost of Component 1 was significantly less than the appraised cost, due to restructuring of the activities, including dropping of one sub-component. • For Component 2, although the scale of the rural river-network demonstration site was reduced from 66.5 hectares to 2.2 hectares, the cost of the activity increased from US$ 300,000 to US$ 650,000 due to limited availability of local counterpart funding at the new site. Also, the village wetland sewerage treatment activity was financed by government funds, instead of project funds due to slow project preparation and government funds becoming available. Financing • The project was financed by a grant from the Global Environment Facility (GEF) in the amount of US$ 4.788 million, as well as a Borrower contribution of US$ 14.2 million (Government of China) and Beneficiary contribution of US$ 15.65 million (various project implementing agencies and participating farmers), for a total of US$ 34.73 million. Borrower contribution • The government of China was expected to provide US$ 14.2 million in counterpart funds. The actual amount provided was significantly higher at US$ 23.67 million, due to the government's decision to provide additional financing for several activities, increasing the actual project cost to US$ 44.1 million. Dates • July 2013: The project was restructured, with several project activities revised or dropped. In Component 1, the bio-gas generation technology was dropped and replaced with an expanded composting facility. In Component 2, the site of the activity was changed, including significant reduction in hectare coverage. Targets for several key indicators were revised accordingly to reflect the changed scope, as well as to rectify some inaccuracies: the original targets had only measured solid waste but were updated to include liquid waste; the nitrogen targets were correctly calculated. • June 2014: The project was restructured, including minor revisions of several key indicators to reflect project progress to date. In Component 1, the integrated waste management activity was dropped, as the land use clearance (for "scientific research" purposes) expired before the activity could be initiated. The closing date was extended from June 2014 to June 2015 to allow for completion of activities. 3. Relevance of Objectives & Design a. Relevance of Objectives Objectives under original targets Shanghai municipality (comprised of the city core and surrounding non-urban areas) was being significantly impacted by the deteriorating water quality of the two major rivers that are the main sources of raw water. The water quality of the nearby East China Sea was also being negatively affected, as significant areas of seawater were failing to meet national standards of clean water. Four major sources of pollution were identified in Shanghai: livestock waste, untreated household wastewater, crop straw, and chemical fertilizer/pesticides. In 2010, the national government officially recognized that agricultural and rural sites had surpassed industry and urban areas as “the most important sources of water pollution in the country,” with crop waste, livestock waste, aquaculture contributing 44%, 57%, and 67% of the country’s total chemical oxygen demand, total nitrogen, and total phosphorus discharges, respectively (as reported in the ICR, page 13). Therefore, the project objective is highly relevant to the country conditions. The intent to "demonstrate" potential technologies to address the pollution problem was also highly appropriate, given the magnitude of the problem. The project objective is also consistent with the Borrower and Bank priorities. The national strategy for the agriculture sector and Shanghai municipality’s own multi-year Environmental Action Plans emphasized reduced fertilizer and pesticide use and improved basic management of Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-GEF-IF-Shanghai Agricultural &Non-Poi (P090376) livestock and agriculture, as well as agricultural non-point source pollution control overall as priorities. The Bank’s Country Partnership Strategy for FY 2013-16 includes support for “greener growth” (such as demonstrating pollution management) as the first of three strategic engagement areas. Objectives under revised targets The relevance of the objectives remains highly relevant under the revised targets. Rating Revised Rating High High b. Relevance of Design Design under original targets The primary project interventions were likely to achieve the objective to demonstrate the effectiveness of various pollution reduction technologies. The selected technologies had been implemented with positive results in other parts of the world and therefore likely to be proven effective in the Shanghai context. However, these technologies were considered innovative in the country, and therefore the project aimed to demonstrate results before being scaled up. These interventions addressed the multiple sources of rural/agricultural water pollution, including livestock waste (on large and medium scale farm contexts, and also the village level context), agricultural and rural discharge (due to lack of wetland to control non-point pollution), household waste, and improper fertilizer/pesticide use. The exception to relevance of the activities was the biogas power generation technology, which was costly and therefore not feasible for adoption by farmers in this context. The project activities also included, importantly, training for stakeholders to ensure acceptance of the technologies, a robust monitoring plan to demonstrate results, as well as development of a strategy to ensure replication. Design under revised targets The relevance of the project design remained the same under the revised targets. Rating Revised Rating Substantial Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To demonstrate effective and innovative pollution reduction activities in Shanghai's rural areas Rationale Outputs • Introduction and demonstration of six pollution reduction technologies (target: 8), including: • Expansion of the capacity of the wastewater treatment plant (with an aerobic reactor and composting yard) at the large-scale demonstration farm, from 450 tons/day to 800 tons/day; • Construction of composting and pelletizing systems with a series of anaerobic and facultative lagoons at the medium-scale demonstration farm. • Construction of 13 village sewage treatment systems, with a total capacity of 260 m3/day. • Construction of an artificial river-network wetland, including ecological restoration of river embankment and beds. The Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-GEF-IF-Shanghai Agricultural &Non-Poi (P090376) original location comprised of 66.5 hectares was dropped due to a change in the land use master plan. The new location was comprised of 2.2 hectares. This activity was completed one month before project closing and therefore no data on effectiveness of this technology was available. • Training of 1,100 farmers on the use of fertilizer, insecticides and pesticides (target: 1,100). Usage of the demonstrated organic fertilizer, low-residue and low-toxicity pesticides, and green pest control techniques reached 100%. • Production of 24,000 tons and 6,977 tons of organic fertilizer in 2014 by two demonstration farms, respectively. For the latter farm, the fertilizer was applied to the farmland, resulting in savings of about US$ 168,000 that would have been spent on chemical fertilizer. • Development of replication strategy for disseminating demonstrated technologies, including recommendations on financing and institutional arrangements. However, • The biogas power generation technology was not introduced at the large-scale farm at the request of the implementing unit due to high investment costs and low likelihood of adoption. • The integrated agricultural and livestock waste management demonstration at Qianwei Village was dropped as the land use clearance for “scientific research” use had expired. Outcomes Livestock Waste Management • The demonstration site is treating an average of 360.0 tons of livestock solid and liquid waste per day, surpassing the target of 126.0. • The total nitrogen (TN) pollution load from the demonstration site was reduced for liquid waste by 73.73 tons/year, surpassing the target of 30.6. • The total phosphorus (TP) pollution load from the demonstration site was reduced for liquid waste by 4.95/year, falling short of the target of 7.8. • The biological oxygen demand (BOD) pollution load from the demonstration site was reduced for liquid waste by 398.56 tons/year, falling short of the target of 606.5. • The chemical oxygen demand (COD) pollution load from the demonstration site was reduced by 638.92 tons/year, falling short of the target of 1,347.5. Wetlands Note: According to the ICR (page 16), some of the original targets were set before the M&E consultant was hired and therefore “ might have been conservatively set due to uncertainties regarding the performance of the technology.” • The demonstration site is treating an average of 199 m3 of rural household wastewater per day, falling short of the target of 504. • The ammonia nitrogen (NH3-N) pollution load from the demonstration sites was reduced by 16.6 tons/year, surpassing the target of 1.87. • The total phosphorus (TP) pollution load from the demonstration sites was reduced by 2.2 tons/year, surpassing the target of 0.57. • The biological oxygen demand (BOD) pollution load from the demonstration site was reduced by 90.7 tons/year, surpassing the target of 33.21. • The chemical oxygen demand (COD) pollution load from the demonstration sites was reduced by 173.4 tons/year, surpassing the target of 59.3. Achievement is rated Modest due to failure to implement two of the eight demonstration technologies as well as significant shortfalls in achieving targets for reduced pollution load. Rating Modest Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-GEF-IF-Shanghai Agricultural &Non-Poi (P090376) Revised Objective Objective was unchanged, while targets were revised. Revised Rationale Outputs • Introduction and demonstration of six pollution reduction technologies (target: 6) - see above. Outcomes Livestock Waste Management • The demonstration site is treating an average of 360 tons of livestock solid and liquid waste per day, surpassing the target of 243.0 tons/ day. • The total nitrogen (TN) pollution load from the demonstration sites was reduced by 536.5 tons/year, surpassing the target of 425.0. • The total phosphorus (TP) pollution load from the demonstration sites was reduced by 163.8 tons/year, surpassing the target of 129.0. • The biological oxygen demand (BOD) pollution load from the demonstration sites was reduced by 2,442.9 tons/year, surpassing the target of 1,983.0. • The chemical oxygen demand (COD) pollution load from the demonstration sites was reduced by 4,828.5 tons/year, surpassing the target of 4,198.0. Wetlands • The demonstration site is treating an average of 199 m3 of rural household wastewater per day, surpassing the revised target of 157. • The ammonia nitrogen (NH3-N) pollution load from the demonstration sites was reduced by 16.6 tons/year, surpassing the revised target of 2.29. • The total phosphorus (TP) pollution load from the demonstration sites was reduced by 2.2 tons/year, surpassing the revised target of 0.7. • The biological oxygen demand (BOD) pollution load from the demonstration sites was reduced by 90.7, surpassing the revised target of 28.03. • The chemical oxygen demand (COD) pollution load from the demonstration sites was reduced by 173.4, surpassing the revised target of 40.9. Revised Rating Substantial PHEFFICACYTBL Objective 2 Objective To reduce the rural and agricultural pollution load (especially nutrients) in the surface water flowing to the East China Sea. Rationale Outputs See outputs listed above. Outcomes See outcomes listed above. The ICR (page 17) also reports on total reduction of rural and agricultural pollution loads from all project sites. (Note: There were no targets set, as these were not key outcome indicators). • 536.5 tons/year of total nitrogen pollution was removed. • 166.0 tons/year of total phosphorus pollution was removed. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-GEF-IF-Shanghai Agricultural &Non-Poi (P090376) • 2,533.6 tons/year of BOD pollution load was removed. • 5,001.9 tons/year of COD pollution was removed. • 16.6 tons/year of NH3-N pollution was removed. • The ICR (page 17) reports that the use of organic fertilizer, compound fertilizer and high-concentration dedicated fertilizer increased, while that of chemical fertilizer decreased. No specific details are provided. However, given the small scope of the project and the geographic location of Shanghai at the estuary of the Yangtze River, where it receives pollution from further upstream towns, the actual attributable impacts of the project in improving water quality of the East China Sea are likely marginal. Rating Modest Revised Objective No change in objective or targets Revised Rationale No change Revised Rating Not Rated/Not Applicable 5. Efficiency The economic analysis at project appraisal provided a separate analysis for each of the eight demonstrated technologies. Livestock Waste management Benefits were considered as follows: organic fertilizer produced, biogas energy produced, and reduction of carbon dioxide pollutants. Costs were represented by the project cost for these activities. The combined economic internal rate of return was estimated at 15% (15% for the large scale farm, 13% for the medium scale farm, and 17% for the integrated livestock and agricultural waste management). At project closing, the benefits realized were the production of organic fertilizer and reduction in pollution. The production of biogas energy did not occur. The economic internal rate of return was calculated at 14.1% for the large scale farm and 12.2% for the medium scale farm. These rates were lower than appraised, and the ICR (Annex 3) suggests that the three-year delay in implementation was a contributing factor. These activities comprised 12.5% of total project cost. The financial sustainability analysis also indicated that the annual financial revenue for these activities (due to the sale of organic fertilizer) exceeded the total annual costs. Wetlands The economic analysis for these activities consisted of a cost-effectiveness analysis, comparing the restored wetland intervention to an alternative intervention, namely construction of an artificial wetland. The estimated cost of the restored wetland was 0.59 RMB/ton, while the estimated cost of the artificial wetland was 0.78/ton. Thus the more cost effective option was selected for this project. At project restructuring, the selected intervention site was changed and the scope was significantly reduced from 66.5 hectares to 2.2 hectares. The actual cost of the activity was calculated at 7,941 thousand RMB/hectare. This is far higher than the appraised estimate of 109 thousand RMB/hectare (re-calculated in hectares, not tons). Increased benefits were also expected from the reduction in pollution; however, as the outcomes for this activity were yet to be measured due to delayed implementation, these additional benefits could not be Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-GEF-IF-Shanghai Agricultural &Non-Poi (P090376) accounted for in the economic analysis. These activities comprised 9.0% of project cost. Village Sewage Treatment Similar to the wetlands analysis, a cost-effectiveness analysis compared the boil-filter technology to an alternative intervention, namely energy aeration technology. The estimated costs for each intervention indicated that the energy aeration cost 2.08 RMB/ton, while the bio-filter cost 1.68 RMB/ton. Thus the more cost effective option was selected. At project closing, the ICR reports that the total project costs for this activity were close to the appraised amount and thus similar in cost effectiveness as the estimated amount; however, there was no updated analysis. These activities comprised 6.4% of project cost. Agricultural Pollution Reduction At project appraisal, the cost-effectiveness analysis compared the economic value of organic fertilizer to standard fertilizer. The economic value includes the value of reduced COD, using price estimates for trading of pollutant emission rights in China. The total economic value of organic fertilizer is RMB 485/ton, which compares favorably to RMB 345/ton for standard fertilizer. According to the ICR, the updated economic value of organic fertilizer is RMB 364/ton, which still compares favorably to the economic cost of 345/ton. These activities comprised 64.9% of project cost. Overall There were also some inefficiencies in project implementation. There were implementation delays (atleast 12 months) at the start of the project, which led to incomplete implementation of some activities and therefore shortfalls in results alongside full project costs. Two of the eight planned demonstrations did not take place. However, overall, the use of cost-effective interventions was a key feature of the project design, and the economic return for the activities that comprised the majority of project costs were also favorable to the alternative options. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 27.30 Appraisal  15.00 Not Applicable 12.50 ICR Estimate  13.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Project under original targets – Moderately Unsatisfactory Relevance of the project objectives is High and relevance of the project design is Substantial. Achievement of the objective to demonstrate effective and innovative pollution reduction activities is rated Modest due to failure to implement two of the eight demonstration technologies as well as significant shortfalls in achieving targets for reduced pollution load. Similarly, contribution to the broader objective to reduce the rural/agricultural pollution load (especially nutrients) to the East China Sea is rated Modest due to limited evidence and attribution. Project under revised targets – Moderately Satisfactory Ratings remain the same, except that achievement of the objective to demonstrate effective and innovative pollution reduction activities is rated Substantial due to meeting of targets in reducing pollution loads. Efficiency over the project period is rated Substantial. Overall outcome The overall outcome rating of a restructured project is determined by weighting the outcome ratings for the original and restructured projects, according to the proportion of the Grant that disbursed during each of those periods. In the case of this project, US$ 1.01 million of the Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-GEF-IF-Shanghai Agricultural &Non-Poi (P090376) Grant, or 21.1%, had disbursed at the time of the first project restructuring (July 2013). Therefore, the combined overall outcome rating is Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating The national and municipal governments are likely to continue prioritizing the overarching project objective to reduce pollution. Sustainability of actual operations is likely due to demonstrated effectiveness and low cost of the project interventions, although these will also depend on plans for operations and maintenance (O&M). The Qingpu district government took measures to establish O&M standards, provide funding, and outsource O&M functions to a private company, in order to sustain the village sewage treatment systems. The large scale dairy farm also extended the contract of its O&M operator. The preparation of the replication strategy, which included recommendations for financing and institutional and O&M arrangements, will also contribute to the scaling up of the demonstrated technologies. a. Risk to Development Outcome Rating Modest 8. Assessment of Bank Performance a. Quality-at-Entry The project design drew upon findings from the Upper Huangpu Catchment Management Study (2005), which had identified the four major sources of pollution to the Yangtze River and the East China Sea. The project design, as well as the results framework, included features that supported the demonstration objective (i.e. focus on short-term results, dissemination of findings, replication strategy). Some flexibility or uncertainty was to be expected given the innovativeness and demonstration purpose of the project; however, it appears that while the technologies were overall technical sound, the feasibility of implementation in the project's specific contexts was not adequately assessed (for example, the biogas technology activity was requested by the large-scale farm, but was later deemed inappropriate due to high investment cost and low technical capacity of the operating company and was subsequently dropped). There was a lengthy period between approval of the project by the Global Environment Facility Board (October 2007) and approval by the Bank Board (June 2010); the ICR (page 8) suggests that the parallel Bank project (Shanghai Urban Environment APL) was being prepared at the same time, and therefore both the client and the Bank team (which experienced two changes in task team leadership during preparation) were more focused on delivering the larger APL project. Also, the multitude of implementing agencies and the geographic dispersion of the activities posed a coordinating challenge. Risk assessment was considered "Moderate" overall, although specific risks such as low commitment of smaller implementing agencies, inappropriate choice of technology, and weak coordination were all rated Moderate but subsequently materialized and effective mitigation measures were not identified. The M&E framework had some shortcomings as well, including lack of accurate target figures. Due to the multiple shortcomings discussed above, Quality-at-Entry is rated Moderately Unsatisfactory. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The project experienced several implementation challenges at the start, as the technologies were innovative and untested in the Shanghai context. For example, the biogas technology was changed to an expanded composting activity, the village-level integrated agricultural and livestock waste activity was dropped due to the expiration of the land use permit for scientific research; and the site for the rural river network system was changed due to changes in the master land use plan. Project supervision ratings were candid during this period, reflecting the slow progress in implementation and low disbursement levels, and the ICR (page 8) suggests that turnover in Bank team leadership was a factor in maintaining commitment level of the smaller implementing agencies. However, the Bank team worked closely with the municipal government to restructure the project components and with the various implementing agencies to improve their capacity. Project performance improved significantly and the revised project was mostly implemented as planned. The results framework was revised to reflect the Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-GEF-IF-Shanghai Agricultural &Non-Poi (P090376) restructured activities, as well as to address inaccuracies in the target figures. Fiduciary performance was overall satisfactory with no major problems reported. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance The municipal government of Shanghai demonstrated strong commitment to the project, as the project objectives were strongly aligned with its own environmental strategy and action plan. It was supportive of attempting new technologies and worked closely with the Bank team to revise activities as needed. Counterpart funding was provided well beyond appraised amounts. However, the ICR (page 23) reports that commitment at the district and county levels was low, as reflected in the varying performance of smaller implementing agencies. Government Performance Rating Moderately Satisfactory b. Implementing Agency Performance The project management office (PMO) under the Shanghai Development and Reform Commission was the primary implementing agency. While it had some prior experience with Bank projects, it lacked technical knowledge of agricultural development and non-point source pollution. Implementation challenges included slow procurement (procuring equipment for medium and small farm demonstrations), and weak project management. In addition, there were seven smaller project implementing agencies (PIAs) that lacked any experience with Bank projects. The ICR (page 8) reports several challenges with these institutional arrangements: difficulty in maintaining commitment from some PIAs, lack of prior working relationships with the PMO; limited leverage over the PIAs due to small grant amounts. Implementation performance improved significantly through project restructuring, the increased participation of the Shanghai Agricultural Commission Working Group (which provided technical support with the agricultural technologies) and with intensive Bank team support. Fiduciary performance was overall satisfactory, in part due to the hiring of an experienced procurement team that improved the quality and timeliness of bidding documents. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The M&E design aimed to capture the project's impact on pollution and on capacity, the latter of which was critical given the demonstration purposes of the project. Key indicators were relevant and measurable, although some baseline and target figures were not correctly identified. Indicators for pollution loads measured liquid waste only and did not include solid waste, and total nitrogen pollution load was incorrectly calculated. Also, for the wetlands component, some of the original targets were set before the M&E consultant was hired and therefore “ might have been conservatively set due to uncertainties regarding the performance of the technology” (ICR, page 16). The M&E design included dissemination of results and development of a replication strategy. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-GEF-IF-Shanghai Agricultural &Non-Poi (P090376) b. M&E Implementation The shortcomings in calculation of indicators were addressed at the first project restructuring. The implementing agencies for Components 3 and 4 were notable in providing updated monitoring records in a timely manner. The ICR (page 11) reports some difficulties of the PMO and some PIAs with data collection due to "lack of clear definitions" but that following the hiring of an M&E consultant in June 2014, data collection and analysis improved. c. M&E Utilization Disseminating results from the technology demonstrations was a critical aspect of the project's objectives. Following the village sewage treatment system demonstration, the district government replicated the technology in other parts of the district. Also of note, M&E data were used by the large and medium scale farms under the livestock waste management component to secure their business licenses from the Environmental Protection Bureau. M&E Quality Rating Substantial 11. Other Issues a. Safeguards The project was classified as an Environmental Category "B" project due to potential negative impacts from construction, use of pesticides and insecticides, and resettlement for the wetland sewage construction. Safeguard policies were triggered on Environmental Assessment (OP/BP 4.01), Pest Management (OP 4.09) and Involuntary Resettlement (OP/BP 4.12) An Environmental Assessment report was prepared that included an Environmental Management Plan for each sub-somponent, including preventive measures for each sub- component, as well as a Pest Management Plan and an Abbreviated Resettlement Action Plan. According to the ICR (page 12), an external party conducted environmental monitoring (including noise, ambient air quality, farmland runoff, groundwater, surface water quality, and soil quality) of both the construction and operation phases, and compliance with the EMPs and the Bank's safeguard policies was satisfactory. Compliance with pest management safeguards was also satisfactory, with the project's implementation experience considered good practice. Due to the change in location for the wetland sewage sub-component, no resettlement took place. b. Fiduciary Compliance Financial management: The ICR (page 12) reports that there were no major issues, with financial management performance of the PIAs generally satisfactory despite initial lack of experience with Bank's processes. There were some delays in submitting interim financial reports due to the need to consolidate reports from the multiple implementing agencies. The ICR does not report whether audit reports had qualifications. Procurement: The ICR (page 12) reports no major issues in procurement although PIAs had low capacity and had difficulty preparing qualified bidding documents. An experienced consulting team was subsequently hired by the PMO, which led to improved quality and timeliness of bidding documents. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-GEF-IF-Shanghai Agricultural &Non-Poi (P090376) c. Unintended impacts (Positive or Negative) --- d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Moderately Satisfactory Moderately Satisfactory --- Risk to Development Outcome Negligible Modest --- Bank Performance Moderately Satisfactory Moderately Satisfactory --- Borrower Performance Moderately Satisfactory Moderately Satisfactory --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons Lessons drawn from the ICR, adapted by IEG: • For a relatively small project, the design should be less complicated, including involving a small number of implementing entities. In the case of this project, coordination challenges arose due to the use of seven project implementing agencies at the sub-component level, some of which were more experienced and/or committed than others. • For a highly technical (and innovative) project, the implementation arrangement should be aligned with institutional mandates and experience. In the case of this project, the selected project management office was under the Shanghai Development and Reform Commission, which, although it had significant experience with Bank projects in the water and wastewater sectors, did not have technical expertise in agricultural and non-point pollution issues. The increased involvement of the Shanghai Agricultural Commission in restructuring the agricultural sub-components was a significant factor in improved project performance. • Selected technologies should be aligned with beneficiaries' ability to pay and technical skills. In the case of this project, the large-scale farm originally requested biogas power generation technology but later changed to more conventional technology due to the complexity. A comprehensive assessment of the adequacy and affordability of this technology for the specific implementing agency, taking into account the agency's technical and financial capacities, was not undertaken. 14. Assessment Recommended? No 15. Comments on Quality of ICR Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-GEF-IF-Shanghai Agricultural &Non-Poi (P090376) The ICR was results-oriented, with quality evidence and a strong focus on both technical outcomes (for example, reductions in pollution loads from liquid and solid waste) and on demonstration outcomes (for example, outputs to promote dissemination and replication). Lessons were specific and well-drawn from the project experience, particularly for highly technical projects. However, the ICR did not adhere to guidelines for restructured projects - although the PDO did not change, the key outcome targets were revised at the time of the first project restructuring and therefore a split outcome rating should been assessed. Also, in financial management, there was no information provided in the ICR on project audits and whether there were any qualifications. a. Quality of ICR Rating Substantial