Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-Sustainable Dev. in Poor Rural Areas (P099751) Report Number : ICRR0020306 1. Project Data Project ID Project Name P099751 CN-Sustainable Dev. in Poor Rural Areas Country Practice Area(Lead) China Agriculture L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-79100 31-Dec-2015 206,615,000.00 Bank Approval Date Closing Date (Actual) 10-Jun-2010 31-Dec-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 100,000,000.00 0.00 Revised Commitment 99,920,360.16 0.00 Actual 99,920,360.16 0.00 Sector(s) General agriculture, fishing and forestry sector(45%):General water, sanitation and flood protection sector(20%):Agro-industry, marketing, and trade(20%):Rural and Inter-Urban Roads and Highways(15%) Theme(s) Rural services and infrastructure(50%):Poverty strategy, analysis and monitoring(25%):Other environment and natural resources management(25%) Prepared by Reviewed by ICR Review Coordinator Group Ridley Nelson John R. Eriksson Christopher David Nelson IEGSD (Unit 4) PHPROJECTDATATBL Project ID Project Name P101844 CN-GEF-Sust Dev in Poor Rural Areas ( P101844 ) Country Practice Area(Lead) China Agriculture L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-Sustainable Dev. in Poor Rural Areas (P099751) TF-97197 31-Dec-2015 147,465,000.00 Bank Approval Date Closing Date (Actual) 10-Jun-2010 31-Dec-2015 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 4,265,000.00 Revised Commitment 0.00 4,219,958.81 Actual 0.00 4,219,958.81 Sector(s) Agricultural extension and research(10%):General agriculture, fishing and forestry sector(80%):Water supply(10%) Theme(s) Natural disaster management(10%):Poverty strategy, analysis and monitoring(20%):Social Protection and Labor Policy & Systems(20%):Rural services and infrastructure(30%):Climate change(20%) 2. Project Objectives and Components a. Objectives Project Portal Project Development Objective: The project's development objective is to explore and pilot more effective and innovative ways of providing poverty reduction assistance to the poorest communities and households in Henan Province, Shaanxi Province and Chongqing Municipality (the Project Provinces) through community driven development and participatory approaches. Financial Agreement Development Objective: The objective in the Loan Agreement was: “(a) to explore and pilot more effective and innovative ways of providing poverty reduction assistance to the poorest communities and households in Henan Province, Shaanxi Province, and Chongqing Municipality through CDD and participatory approaches; and, (b) to pilot Sustainable Land Management and adaptation measures to address vulnerability to climate change in poor rural areas. The Global Environmental Objective was formulated as for (b) above, “to pilot Sustainable Land Management and adaptation measures to address vulnerability to climate change in poor rural areas.” In this review, the project is assessed against the Loan Agreement objectives. This encompasses both PDO and GEO. While there was no change in the objectives during the period of the project, there was a substantial change in the outcome indicators. These were substantially revised at the October 2013 project restructuring. Ten new indicators were added and six were revised. Only two of the original sixteen indicators remaining unchanged. The three indicators associated with the Rural Migrants Support Component were eliminated with the dropping of that component at restructuring. The ICR argues that since no data was reported for the original results framework it is not possible to undertake a split evaluation of the original and restructured results framework. While lack of data is not a reason not to do a split rating, in this review IEG has also not adopted a split rating approach on different grounds. A number of key indicators were, indeed, changed, but these did not constitute a de facto change in direction for the project. The objectives remained unchanged and the new indicators mainly changed the way the achievement of objectives was to be measured, giving a greater focus on cost effectiveness. The overall intent of the project remained. b. Were the project objectives/key associated outcome targets revised during implementation? No Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-Sustainable Dev. in Poor Rural Areas (P099751) c. Components There were five components at the time of approval. Component 1: Community Driven Development (CDD) (Appraisal US$90.6 million; Actual US$120.9 million) This was to construct or improve basic rural infrastructure in poor villages through investment and technical assistance. Areas of investment were to include: access roads, drinking water supply, rural sanitation, electrification, communications infrastructure, school construction and repair, agricultural production infrastructure, and housing repair. Component 2: Community Development Financing (CDF) (Appraisal US$35.7 million; Actual US$22.0 million) This was to provide financing to community cooperatives to support household level productive activities, technical assistance, and capacity building at village level. Component 3: Rural Migrants Support (Appraisal US$10.4 million; Actual US$0.0 million (the component was dropped)) This was to address, only in Chongqing Municipality, large-scale rural to urban migration by improving migrant workers dormitories and service centers, providing vocational training and job placement, establishing migrant community service centers, providing migrant laborers micro-enterprise support financing, strengthening M&E related to the welfare of migrant workers, and undertaking research on migration and poverty reduction. Component 4: Sustainable Land Management and Adaptation (SLMA) (Appraisal US$4 .3 million; Actual US$4 .2 million) This was to support sustainable land management pilot activities by promoting adaptation measures in small infrastructure and agricultural improvement, the provision of technical assistance, the dissemination of experience from these pilot activities, integration of the SLMA activities into the CDD activities, policy studies and consultation workshops. Component 5: Project Management, Monitoring and Evaluation (Appraisal US$11 .9 million; Actual US$9 .4 million) This was to strengthen the capacity of the project implementation agencies in project management, monitoring and evaluation and the design and implementation of a comprehensive project MIS system including physical progress, financial management, and impact assessment. The component also supported training, and workshops. When the Rural Migrants Support component was eliminated at the time of the MTR, the funds were reallocated to the Municipality’s CDD component. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Costs Total Project Costs at appraisal was US$158.8 million and actual at closing was US$156.5 million. Financing. At appraisal, the Bank financed US $100 million which remained the same at closing. At appraisal, the GEF financed US$4.27 million with an actual of US$4.23 million at closing. There was no additional financing. Borrower Contribution. The borrower planned to contribute US$54.88 million at appraisal. This became US$52.57 million at closing. Dates. There was an unusually long delay between appraisal and effectiveness. Appraisal was on October 20, 2008 and effectiveness was more than 2 years later on November 16, 2010. This was due to clearance and processing issues on the side of the borrower. This resulted in a number of policy shifts and changed circumstances which meant that the original design became dated. The project closed on schedule on December 31, 2015. 3. Relevance of Objectives & Design a. Relevance of Objectives As noted by the ICR (p.14), the objectives were and remain highly relevant for addressing poverty reduction and increased incomes in the rural areas of China. However, in discussing relevance, the ICR is not quite accurate in asserting (p.14) that the project key objective was “to introduce the CDD and CDF approaches in order to increase (a) the share of poverty reduction funding of the village level and, (b) poor villagers’ role in determining the use of available poverty reduction funds”. This was not, in fact, the language of the stated project development objective. To be accurate the PAD language under the section on rationale for Bank involvement, while similar, was: “The rationale for Bank involvement … is to assist government in overcoming the institutional and administrative barriers to the effective use of poverty reduction program funding so that … a greater share … actually reach the poorest communities, and … the villages themselves play a greater role in determining the poverty reduction measures and forms of assistance …” The Loan Agreement stated objective was highly relevant to both the borrower’s and Bank’s strategy at the time of appraisal and closing. With respect to the Bank and borrower strategy, the China Country Partnership Strategy, Pillar 2, aims at, “reducing poverty, inequality and social Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-Sustainable Dev. in Poor Rural Areas (P099751) exclusion”. The ICR adds that the challenge of easing rural migrants’ transition to urban employment was also of great concern to the government at the time and a concern of civil society. The project was consistent with China’s 11th Five Year Plan and Western Region Development Strategy. It also contributed to China’s obligations under the Convention on Climate Change, the Convention to Combat Degradation, and the UN National Action Plan for Desertification Control. The project was consistent with the GEF Climate Change Operational Strategy and the GEF Land Degradation Focal Area strategy. Rating High b. Relevance of Design The original results framework, later substantially altered, is given in Annex 8 of the ICR. It included the following as main indicators to measure “the extent to which the project enabled the exploration and piloting of more effective and innovative ways of providing poverty reduction assistance to the poorest communities”: 1 The Chinese government acknowledges the importance of CDD and participatory approaches for future poverty alleviation. 2 80% satisfaction among beneficiaries regarding project impact on income levels and well-being and on the CDD approach itself. 3 For Component 1, indicators included: percentage of the available funds actually transferred (funds transferred to the poorest villages were to be equal to the average for all villages), the percentage of women participants in project management groups; the percentage of infrastructure works assessed as satisfactory; the percentage of investments with maintenance and management arrangements; and the percentage of villages with multiyear development plans. 4 For Component 2, there was a single indicator which was that 70% of the poorest households would benefit from the CDF fund. 5 For Component 3, the indicators included numbers of trainings completed, employment rate after training, and the establishment of two migrant workers service centers. (As noted, this component was never implemented.) 6 For Component 4, the GEF component, the main indicators were the following. 70% of pilot villages completed resource mapping and identified needs. 50% of villages implemented innovative adaptation measures. The total number of adaptation innovations introduced into the CDD menu were to be measured but no target was set. The carbon stock increased by 5% across all pilot SLMA villages. This original framework and the associated indicators were weak in that the extent to which "exploration and piloting" was achieved was not explicitly measured. The revised results framework following restructuring represented a shift towards measuring cost efficiency and towards indicators that were easier to measure. The main indicators under the revised framework were: 1 Percentage cost savings on infrastructure investments (which were carried out under the CDD activity) 2 Percentage of directly benefiting poor households in both CDD and CDF activities in relation to the percentage of poor households in project villages. (i.e. 100% would mean all poor households participated). 3 The financial rate of return on CDD investments and CDF loans. 4 For the GEO objective, the main indicator was a qualitative statement: “Disseminate the improved SLM approach through innovative community pilots mainstreaming the CDD model while addressing the vulnerability of poor rural areas to climate change”. This gave no real milestones for quantitative assessment. In addition, there were some “core sector” indicators added at this stage (referring to Bank core sector indicators recently proposed). These included measurement of awareness of the project, rate at which grievances were addressed, the extent of operations and maintenance, beneficiary assessment of the extent to which project investments reflected their needs, and the share of vulnerable and marginalized households in total project beneficiaries. For the environment-focused SLMA activity, revised indicators included: the percentage of completion of village assessments, resource mapping, and adaptation needs; the percentage of villages implementing innovative adaptation measures; and the changes in carbon sequestration stock. In both the original and revised framework indicators, it is not easy to discern how the measurement of “exploration and piloting” or “more Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-Sustainable Dev. in Poor Rural Areas (P099751) effective and innovative ways” was to be done. There is an implied testing and piloting in the language of the objectives that seems not to be well reflected in the indicators. Questions that arise from the objectives statement include: (i) to what extent were alternative institutional relationship structures explored and piloted? Which were the most effective and efficient? (ii) to what extent were alternative processes of community planning explored and piloted? Which were found to be the most effective and efficient? (iv) to what extent were alternative mechanisms for a community role in managing infrastructure contractors explored and piloted? (vii) to what extent were alternative approaches to participatory M&E explored and piloted? Neither before nor after restructuring, do there appear to be indicators that would contribute much to learning from testing institutional and process innovations. The ICR picked up on this weakness noting (p.2): “It is important to highlight that the emphasis of the PDO was to explore and pilot (italics added here) the CDD and participatory approaches in the project provinces, and the primary beneficiaries’ satisfaction with: (a) the CDD approach, and, (b) its impact on the income levels and well-being. The PAD does not state that the PDO was either poverty reduction or increased incomes among the primary beneficiaries …” The ICR goes on later to say (p.3), “The ICR team believes that the extensive changes to the PDO, GEO and intermediate outcome indicators were excessive and also incorrectly changed the focus of the PDO indicators from “exploring and piloting (italics added here) the CDD and participatory approaches in the Project provinces” to assessing the effectiveness of the CDD approach.” This IEG assessment agrees. IEG concludes that: (i) the original Results Framework indicators were weak. While the Revised Indicators offered some additional value in the grievance measurement and the O&M measurement, overall, both before and after restructuring, the results frameworks and associated indicators were weak but in different ways. Rating Modest 4. Achievement of Objectives (Efficacy) PHREVISEDTBL Objective 1 Objective To explore and pilot more effective and innovative ways of providing poverty reduction assistance to the poorest communities and households in Henan Province, Shaanxi Province, and Chongqing Municipality through CDD and participatory approaches. Rationale The following were the achievements drawing mainly from the limited scope of the main indicators: 1 From the ICR, and based on communication with the project team, there is some evidence that the Chinese government acknowledged the importance of CDD and participatory approaches for future poverty alleviation and rural development. However, it is not clear to IEG how deep this acknowledgement really is. Government strategy appears to have, in some respects, undermined the strategy adopted by the project. As discussed further in the Risk section, a government-launched national program of revolving credit funds was started during the project that had no limits on loan size, a repayment rate of about 67% and subsidized interest rates. This seems to be a rather different approach with the potential to undermine what was learned and developed under the project and to threaten sustainability. Indeed, in terms of sustainability, the ICR expressed concerns about the project shifting under the umbrella of the larger program. 2 Based on the impact study, as targeted, 80% of beneficiaries were satisfied with the impact on income and well-being and with the CDD approach. 96.7% of beneficiaries felt the project investments reflected their needs. 3 There was achievement of the targeted wide participation in design, implementation and management of development plans. 92% of households participated in CDD activities (the revised target was 75%) and 66% of project village women participated (above the revised target of 60%). 4 Access to infrastructure appears to have improved. Although “access” was not directly measured, 94% of households were satisfied with the improvement of infrastructure. This suggests substantial access although there are other more direct ways of measuring access. 5 The original target had been for 70% of the poorest households to benefit from the Community Development Fund. The actual achievement was 71% percent. 6 Awareness of the project was found to be 100%, grievances addressed was 100%, community engagement in operations and maintenance was 100%. A high level of 97% of beneficiary households, against a 100% target, felt that project investments reflected their needs. The share of Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-Sustainable Dev. in Poor Rural Areas (P099751) vulnerable and marginalized people in the total project beneficiaries was 69%, above the revised target of 50%. 7 100% of pilot villages successfully completed the Sustainable Land Management and Adaptation village assessments and resource mapping; 100% of pilot villages are reported to have implemented innovative adaptation measures, and the carbon stock increased by 19% against a 5% target but the carbon sequestration calculation appears to have been based on data from elsewhere. 8 The target for the Rural Migrants Support activity, which was to be implemented only in the Chongqing Municipality, was not achieved since this was dropped from the project at restructuring because of a change in the government institutional mandate for migrant workers affairs. The ICR notes that the Chongqing Provincial Project Management Office was understaffed, suggesting that it may have been unable to implement it. A main focus of the post-restructuring indicators was on efficiency: 1 As discussed in the Efficiency section and shown in the PDO indicators table, the cost savings under the CDD component was 26% (ICR p. viii). The scale of the improved infrastructure appears substantial since, on road access (by far the largest component) based on the surveys, project villages reported having “better all-weather road access” in 66.5% of villages compared to 36.6% of non-project villages. 2 Rates of return for irrigation construction were estimated to have been more than 20% and on terracing were estimated to have been more than 10%, an average of 15% is quoted in the ICR (p.viii). The revised target for financial rates of return for CDD investments and CDF loans was 12%. 3 The financial rate of return of the CDD investments and the CDF loans was estimated as 15% against the target of 12%. However, it is not entirely clear where this 15% figure comes from since, based on the data from household surveys by the Chinese Academy of Social Sciences team, the only rates of return given in the ICR Annex 3, Economic and Financial Analysis are for farming activities under the CDF giving ERRs ranging from 16 to 38%. 4 In the CDF fund, the turnover rate of lending was 160%, above the 100% revised target and the repayment rate was a very high 99%, above the 95% revised target. Notwithstanding a number of outcome achievements, a significant weakness was the limited evidence, either quantitative or qualitative, of the extent to which the project achieved “the exploration and piloting of more effective and innovative (approaches) …”. While it seems probable that there was a degree of piloting and testing of alternatives within communities as the project approach unfolded, we do not know from the ICR, partly due to weak M&E, what alternative institutional arrangements were tested and explored or how alternative piloted institutional or procedural innovations performed. A similar concern is raised in the ICR which noted that the weakness in the third party monitoring and evaluation contributed to some of the “weakness in the project piloting various innovations that would require rigorous and systematic arrangements for measuring progress, performance and results. …. This had major implications for the project accounting for piloting of innovations …” (ICR p. 25). In conclusion, despite a number of achievements in terms of welfare and poverty impact, the achievement of objectives is rated Modest on balanced due to: (i) the absence of evidence on the exploration and piloting; (ii) concerns about the extent to which the CDD approach has been really internalized (a key indicator) given the national program offering subsidized credit with no limits on loan size and low repayment rates at 67% (ICR p.16 and 21); and, (iii) related concerns about sustainability (see Section 7). Rating Modest PHREVISEDTBL Objective 2 Objective To pilot Sustainable Land Management and adaptation measures to address vulnerability to climate change in poor rural areas. Rationale In the GEF supported Sustainable Land Management and Adaptation component, the three quantitative targets were: (i) for 70% of all pilot villages to successfully complete assessments and resource mapping and identified needs; (ii) for 50% of pilot villages to implement innovative adaptation measures; and, (iii) for carbon stock to increase by 5% across all pilot villages. These were reportedly more than achieved. All pilot villages are reported to have completed assessments, mapping and the needs identification. All pilot villages are reported to have implemented innovative adaptation measures. Calculations done by the ICR reports a 19% increase in carbon stock. However, the carbon achievement appears to be based on data from elsewhere rather than measured within the project area. The evidence reported is that there was completion of the vulnerability analyses, satisfactory quality of implementation of the pilot activities Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-Sustainable Dev. in Poor Rural Areas (P099751) and that experience was documented and disseminated and incorporated into the CDD component design. Policy recommendations fed into the China 13th five year poverty reduction agenda. However, it is not clear where the evidence for these assertions comes from, particularly the quality of implementation of the pilot activities. Again, there seems to be limited evidence of the results of the exploration and piloting. There is limited evidence of the impact of sustainable land management but the objective was set at an output level, to pilot the measures. This appears to have been done, although it would have been informative for future progress to know more about what was learned during this piloting that could be applied in future. On balance, achievement is rated Substantial. Rating Substantial 5. Efficiency In the PAD, the project as a whole was projected to achieve a 31% ERR, with the CDD component yielding 32%, the CDF component 53%, and the Rural Migrants Support component (in the end not implemented) 22%. In the ICR, the economic and financial analysis presented is narrower. It argues (p. 31) that the benefits generated from the CDD component are not readily quantifiable and that therefore a cost effectiveness analysis has been adopted to compare unit cost savings with parallel government programs. (However, it is not clear to IEG why road improvement, the major investment, could not have been analyzed on a sample basis through the well-established model of operational cost reductions based on traffic volumes. In a subsequent exchange, the Project Team argued that the main benefit was not the usual saving of operational costs but improved access to markets, decreased post- harvest losses and facilitation of new home construction. While this may be also true, such benefits are presumably quite closely associated with reduced operational costs for transporters.) On the cost effectiveness of infrastructure, the cost savings under the CDD component based on PMO and CAAS team data suggest cost savings on roads of 19% to 34% “against other similar works .. under .. other programs” (ICR p. 18). Savings on footbridges are reported as 28%, on water supply systems 12%, on pumping stations 25%, on water ponds 35%, on water harvesting pools 34%, and on terracing 4%. The Economic and Financial Analysis Annex reports cost savings on roads “as high as 20% - 39%”. The scale of the investment achievements was substantial. Some 1.76 million people, of which 0.55 million were in the category of poor, benefitted from CDD component activity which included 1,160 km of between village roads and 4,467 km of within village roads. 92,355 people benefitted from CDF activities that included productive investment. 28,720 households joined cooperatives. 134 villages completed resource mapping and vulnerability assessments for the SLM activity. On road access, which was the largest component, project villages had “better all-weather road access” (ICR p.36) in 66.5% of surveyed villages compared to 36.6% of non-project villages. (It is not entirely clear how “better” was defined in the survey but it seems to have meant better than at the time of the starting year of the project.) It is reported in the ICR (p. 39) that 67% of the houses in project villages were on cement roads by the end of the project, 30% higher than in the control villages. The ratio of households with water in project villages was 16% higher than in control villages. On incomes, there is evidence from the State Statistical Bureaus indicating that project villages outperformed the non-project villages despite the fact that the latter generally had better initial conditions. Per capita net income of surveyed households showed an annual increase of about 20% compared to non-project village households with a 10% annual increase. It is reported that the poverty incidence fell from 54% in 2010 to 27% in 2014. With respect to employment, project villages had 18% more outside work than control villages. In terms of beneficiary satisfaction, 80% of interviewees in project villages were “satisfied or very satisfied with their life”. This was somewhat better than control villages which were 15% lower. Qualitatively, the Borrower’s ICR reports that the CDD approach gave communities decision-making rights so that the communities selected specific subprojects themselves, strengthening their sense of ownership. The opening of project accounts at community level reportedly improved fund utilization efficiency. The GEF-funded SLMA activity is reported to have strengthened the consciousness of project communities on vulnerability, climate change and adaptation. The ICR claims a number of other qualitative achievements, but with limited evidence, including improved food security, increased agricultural productivity and farm job opportunities, improved access to markets, safe drinking water, improved transportation, improved electric power, improved access to technology and capacity building activities, and stabilization and reversal of environmental destruction through the land management activities. Institutionally, Project Management Offices (PMO) improved their capacity to manage projects. Through the implementation of disclosure policies and grievance mechanisms, and with innovative arrangements of supervision and post-project management, project communities were reportedly motivated to take care of project outputs with a sense of ownership. On farm enterprise profitability under the CDF component, the economic rates of return based on data collection and household sampling by the CAAS team, ranged between 16% and 38%. These included a limited range of enterprises such as pig, cattle, and goat raising, apple Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-Sustainable Dev. in Poor Rural Areas (P099751) production, and mushroom production. With respect to overall project operational efficiency, there were no obvious significant inefficiencies beyond the long lag from appraisal to effectiveness. The project closed on schedule. During implementation there was a long delay over restructuring and both the Bank and borrower note the difficulty of monitoring the original results framework. However, there does not seem to be any evidence that this significantly slowed implementation. Overall, Efficiency is a mixed story, but given the quite strong cost effectiveness evidence, the adequate enterprise rates of return, the high levels of satisfaction with infrastructure and some plausible qualitative benefit claims, Efficiency is rated, on balance, Substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  31.00 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of Objectives is rated High but Relevance of Design Modest due to weak indicators that did not sufficiently address the stated piloting objective. Efficacy of the first element of the objective is rated, on balance, Modest although there were a number of important achievements including particularly the community managed processes themselves and the substantial investments through the CDD and CDF activities. The extent to which the objective of "exploration and piloting more effective and innovative ways" was met is unclear partly due to limited M&E evidence. Better M&E could have fed back piloted results into strategy and implementation adjustments. The second element of the objectives is rated Substantial. Efficiency is rated, on balance, Substantial due to the cost effectiveness evidence although there was limited evidence on the economic returns to infrastructure. There are also concerns about sustainability, particularly the extent to which the project approach has been really internalized given concerns about the wider government program with subsidized interest rates and low repayment. The project is rated Moderately Satisfactory on the basis of the following main positive and negative considerations: (i) unweighted averaging of the rating of the two elements of the objectives; (ii) allowing for significant achievements in terms of poverty impact; (iii) remaining concerns about Risk which could impact the rating of the indicator on the extent to which CDD approaches have really been internalized; (iv) the clearly High rating of Relevance of Objectives but balanced by weaknesses in Relevance of Design ; and, (v) the Substantial rating of Efficiency due to cost effectiveness. Overall, the project had moderate rather than significant shortcomings. a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating The main risks identified in the PAD (p. 14) were related to financial risks, internal control and funds flow. The ICR, in discussing risk, rightly focused predominantly on the operation and maintenance aspects of sustainability, on the risk of pressure from higher levels of government to achieve quick results, and on recent shifts of the project activities to the national program with concern about pressure to harmonize procedures. There is some concern evident in the ICR that the established CDD approach tested under the project may become distorted within the larger Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-Sustainable Dev. in Poor Rural Areas (P099751) program (ICR p.16, 21-22, 26). No significant financial risks and problems with internal control for funds flow were reported. The ICR notes that communities have demonstrated a strong sense of ownership of civil works suggesting a likelihood that operations and maintenance will be sustained. While the ICR believes that operations and maintenance at a normal level should be adequate, it notes that external support is likely to be needed for damage from disasters such as earthquakes, landslides or floods. Sustainability of the CDF component is considered promising by the ICR as it will be shifted under the new national program: the “Mutual Help Financial Groups”. However this is an area where the concern about pressure to harmonize procedures with the national program may affect the sustainability of the project-supported CDF program. The CDF component appears to have been undermined in strategy by this program of revolving credit funds that reportedly have no limits on loan size, a repayment rate of only 67% and subsidized interest rates (ICR p. 16). This has the potential to undo key elements in the strategy developed under the project. There is also some risk that villages may have difficulty continuing adequate financial records without the more intensive help of community facilitators that they had access to during the project but may now lose. It is of some concern that the ICR reports that mitigating the risk of pressure from higher levels of government to achieve quick results “will require acknowledgment by senior levels of government that CDD approaches, while taking longer to implement, ultimately produce more effective and sustainable poverty alleviation achievements” (ICR p. 21). The concern is that the main indicator originally set in the Results Framework by the PAD was that, “Chinese Government acknowledges importance of CDD and participatory approaches for future poverty alleviation and rural development work”. If this remains a significant risk, it raises questions about the extent to which the CDD approach as an exploratory and piloting intervention actually succeeded. Thus, the two main risk concerns are pressures still from government to by-pass CDD approaches for quick results and the need for facilitators to help properly implement the tested approaches. Overall, risk is rated Substantial. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry Project design drew substantially from: earlier poverty reduction projects; from a Chinese Poverty Assessment by the Leading Group for Poverty Reduction; and from World Bank experience globally. However, there was a one and a half year delay before negotiations due to slow approval processes by government. This left some of the design elements dated including the Rural Migrants Support component that was canceled. The ICR also reports that the third party monitoring and evaluation work suffered from insufficient attention at the design stage. It turned out that there was not a willingness to pay for this activity by the Implementing Agency. The ICR rates Quality at Entry as only Moderately Satisfactory on these grounds. However, the delay was largely due to slow processing on the borrower’s side and the Bank did conduct several missions to attempt to keep the process on track. While this delay in processing does not appear to have been due to weak performance by the Bank, IEG finds other weaknesses. In particular, as noted above, the Results Framework and the associated indicators at the outset were weak in making provision to answer the main question about the extent to which the project explored and piloted more effective and innovative ways of providing poverty reduction. Quality at Entry is rated Moderately Satisfactory. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision As noted by the ICR (p. 22), the project required extensive engagement with the provinces which were unfamiliar with CDD and CDF approaches to poverty alleviation. However, the ICR notes several weaknesses. First, the monitoring indicators were not provided in the project supervision documents until the end of 2014, shortly before the project closed. Several monitoring indicators were not updated during the first three and a half years of the project. Second, there was some disruption in leadership with the World Bank Task Team Leader changing three times over the life of the project. Third, the restructuring process at midterm was lengthy, taking eighteen months. The reallocation of costs using an outdated COSTAB format was reported as the reason for some of this delay. It is not clear why a different format could not have been used or simply Excel. As noted, the revised indicators at the time of the MTR were not, overall, an improvement although they added two or three necessary new Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-Sustainable Dev. in Poor Rural Areas (P099751) elements. The ICR reports that supervision improved considerably in the last two years with the quality of monitoring and implementation support improving. On balance however, the Quality of Supervision is rated Moderately Satisfactory. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance Government and the participating provinces and municipality exhibited quite strong commitment to the project and coordination was generally strong. However, as noted by the ICR, there were serious delays in government’s processing of the project from the date of appraisal to the date of negotiations. There were serious staffing issues on the government’s side particularly in Chongqing Municipality which partly caused the dropping of an important poverty-focused component. There was no government consolidated ICR at the time of the Bank ICR mission. The two M&E systems, despite considerable input from the Bank, could not present a consolidated M&E report until the penultimate year of the project. For a project with a main objective to "explore and pilot" this was a serious weakness. There are also some questions about the extent to which government really internalized the CDD approach since the larger program had subsidized interest rates, specified no loan limits, and had a weak repayment record. Government was not able to present a consolidated implementation completion report for the project in time for the Bank’s ICR mission despite the availability of an independent assessment report. Overall, Borrower Performance is rated Moderately Unsatisfactory. Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance The Central Project Implementation Office (CPIO) was responsible for coordination of the design and supervision. This office coordinated between the relevant ministries and commissions and the provincial level Provincial Project Management Offices (PMO) as well as the Chongqing Municipality PMO. Below the center, responsibility lay with the PMOs. The project management system included procurement systems, project funds managed by special community accounts, and separate procedures for CDD and CDF funds. There was initially slow implementation progress but this gradually picked up. There was reluctance to fund M&E which was a significant omission given the exploratory and pilot nature of the project. There were substantial delays in obtaining registration approval for the CDF cooperatives as not-for-profit organizations. This was most serious in Henan Province. This was a main reason for the reduced allocation to the CDF component by about US $12 million. Another reason was that the government’s Mutual Aid Cooperatives Program became operational after project appraisal but before effectiveness, which opened an alternative option for funding. Understaffing in all the PMO’s was an issue in the earlier years, particularly at Chongqing. There was also a further long delay at restructuring. The CPIO blames this on the heavy workload of restructuring, the difficulty of the original results framework, and the need for the improvement of the indicators. Overall, implementing agency performance is rated Moderately Satisfactory. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-Sustainable Dev. in Poor Rural Areas (P099751) Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The ICR reports that the internal and independent external M&E systems were well designed but, as noted earlier, IEG finds that the original indicators were weak in not clearly providing for measurement of the stated objective. b. M&E Implementation The lead implementing agency was initially reluctant to spend funds on monitoring and evaluation despite the fact that it represented less than one half of 1% of total project costs.The ICR notes that the project internal monitoring and information system was initially hampered by software problems but was functioning by the second year. However, it also reports that, despite strong efforts by the Bank, the two M&E systems did not provide a consolidated report on the project development objectives and the global environmental objectives until late 2014, the year before project closing. As noted earlier, during implementation, the original indicators were substantially revised with 10 new indicators added and 6 substantially changed, leaving only 2 of the original 16 indicators unaltered. The ICR notes that several of the formally revised indicators were ambiguous or difficult to understand although they also argue that the restructured Results Framework represented a consistent and logical means for assessing development impact. There was also an implementation issue on the Bank’s side in that there was no reporting in the Bank supervision documents of the achievement of the indicators until the final supervision report in November 2015. c. M&E Utilization The ICR notes that the information was not systematically utilized to evaluate the achievement of the project objectives during the first four years of implementation. The ICR believes that the output and outcome data collected made it possible to carry out a credible evaluation of development impact but that is not evidence of utilization of the M&E system within the project. IEG notes that, for a project with an objective to explore and pilot, the lack of use of data for redirecting implementation in the first four years was a serious weakness. M&E Quality Rating Modest 11. Other Issues a. Safeguards Environmental The project was classified as a Category B. It triggered safeguard policies on Environmental Assessment (OP/BP 4.01), Involuntary Resettlement (OP/BP 4.12), and Indigenous People’s (OP/BP 4.10). The ICR mentions only the Environmental Assessment safeguard (ICR p 11). The ICR reports that the Environmental and Social Management Framework was properly implemented in all three provinces. While the ICR asserts that project implementation was in compliance with the Bank’s safeguard and fiduciary requirements, there is limited information provided on what was done with respect to the Involuntary Resettlement and Indigenous People’s safeguards. But see under Social below. Social The project objective was substantially focused on piloting innovative approaches to poverty alleviation through the CDD and CDF components. The ICR reports that “no involuntary resettlement issues occurred as any arrangements that would potentially impact their living standard were dealt with on a voluntary basis, and any negative impact was compensated through collectively agreed arrangements”. There is no evidence that the arrangements that were made on a voluntary basis were monitored or supervised by either the borrower or the Bank. It is possible that the grievance mechanism established allowed complaints related to such issues as land taken from a Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-Sustainable Dev. in Poor Rural Areas (P099751) household for improvement of a road. But there is no evidence presented of any form of supervision to ensure that voluntary arrangements were fair and were consistent with Bank safeguards. Similarly, the ICR assures that the project made sure that ethnic minority households had equal opportunity but little evidence is presented. b. Fiduciary Compliance Procurement supervision and post-reviews indicate that required procurement procedures were followed. This included procurement at the village level. At the higher levels, procurement was through provincial or municipal or county offices and reportedly the procedures outlined in the legal agreements were followed. At the community level, there had been concern at the time of appraisal that there were fiduciary risks since this was the first time such a CDD project had been implemented in these areas. Procedures for procurement were put in place. The processes followed at community level included: decision-making and supervision committees in “administrative” villages (those formally recognized for local government); implementation and supervision groups in “natural” villages (local aggregations of households) to implement and supervise procurement and contract management; an operational manual; procurement training including hands-on training for community facilitators and villages provided by the Project Management Offices. Villages were free to decide on procurement methods. Both shopping and community force account were applied. At the village level, all expenditures were publicly notified to all community members. The ICR reports that no significant financial management issues were noted by the Bank missions or uncovered by external auditors. CDF funds had their own special arrangements with an implementation and a financial manual. External auditors audited project villages on a sample basis. c. Unintended impacts (Positive or Negative) None reported. d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment See Section 6 for the balancing of the case. Efficacy on the first element of the objective is rated Modest mainly due to the lack of evidence on the extent to which Outcome Satisfactory Moderately Satisfactory the project achieved the objective of exploring and piloting approaches and questions about the extent to which government has internalized the CDD approach. Due to indications that the CDD strategy may not have been Risk to Development Outcome Negligible Substantial internalized sufficiently to sustain progress initiated. Bank Performance Moderately Satisfactory Moderately Satisfactory --- Borrower Performance Moderately Satisfactory Moderately Satisfactory --- Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-Sustainable Dev. in Poor Rural Areas (P099751) Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The following lessons are drawn largely from the ICR with some amendment of language. 1 An effective M&E system from the outset is critical, particularly for an innovative CDD operation with multiple indicators and many agencies. Unfortunately, in this case, the lead implementing agency was initially reluctant to spend funds on monitoring and evaluation despite the fact that it represented less than one half of 1% of total project costs. 2 Community facilitators can play an important role particularly in a situation where many educated villagers and youth migrate to cities for employment. In this case, community facilitators were able to fill a number of roles including: mobilizing villagers; liaising with the project office, local government and village groups; maintaining records, particularly financial records; and ensuring transparency. 3 A CDD project can help to greatly improve and change the relationship between government officials and households at community level. In this case, in a situation where past approaches had been predominantly top-down, the project was able to develop and demonstrate processes and procedures to involve beneficiaries through the entire process of identifying and selecting activities, contracting, supervising implementation, and operation and maintenance. The ICR notes in the lessons section that it was unfortunate that the Rural Migrants Support component was dropped and the Municipality concerned did not take the opportunity to test innovative solutions to this substantial social challenge. The ICR notes that there remains an urgent need to test and develop solutions to address challenges faced by poor rural migrants in China. 14. Assessment Recommended? Yes Please explain There are a number of questions remaining including: (i) the extent to which the project was able to test and influence national CDD strategy; (ii) the sustainability of the project; (iii) the extent to which the stated objective of piloting and exploring new ways was implemented and what was learned and adopted; (iv) the economic returns to the investments. The trajectory of the post-project phase of this project has implications for the trajectory of CDD support for the whole of China. 15. Comments on Quality of ICR The ICR is strong in some areas but less strong in a few.. It has quite good coverage of some of the issues and there is evidence in two sections that it clearly grasped the weakness in the logical chain between the objectives statements and the indicators, both before and after the substantial changes in the indicators. It offers useful lessons. However, it is less strong on following through to the logical conclusions on that weakness with the indicators and what this meant for project performance. It also could have followed through further on the impact of the government's wider subsidized credit program and the low repayment issue and its implication for sustaining CDD approaches. The Efficiency analysis could have gone further. It is not clear why an overall rate of return could not have been attempted as the PAD did, or at least a rate of return attempted on the major roads component using a well-established methodology. On the other hand, there was some useful cost effectiveness analysis. The ICR offers no clear evidence on whether the Involuntary Resettlement safeguard was met through the seemingly hands-off approach of voluntary arrangements within villages, although it is possible that the grievance mechanism covered this. Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review CN-Sustainable Dev. in Poor Rural Areas (P099751) The ICR coverage of M&E utilization was weak. This may be partly attributable to the fact that M&E itself was weak. While sources of survey data are given, there is no assessment of the validity of the methodologies, e.g. sample size, frequency of enumeration, independence of enumerators, methods of assessing yields, selection of controls, methodology for prices, etc. The ICR offers limited evidence on how well the GEF-funded activities impacted, or were expected to impact, environmental objectives. The carbon sequestration evidence seems to come from data outside the project. a. Quality of ICR Rating Substantial