ICRR 13710 Report Number : ICRR13710 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 03/14/2012 Country : China Project ID : P065035 Appraisal Actual Project Name : Gansu And Xinjiang US$M ): Project Costs (US$M): 110.93 110.91 Pastoral Development Project L/C Number : L4718 Loan/ US$M): Loan /Credit (US$M): 66.27 65.41 Sector Board : Agriculture and Rural Cofinancing (US$M): US$M ): 10.50 10.50 Development Cofinanciers : GEF Board Approval Date : 09/09/2003 Closing Date : 06/30/2010 06/30/2010 Sector (s): Animal production (50%); Agricultural extension and research (25%); Agricultural marketing and trade (25%) Theme (s): Land administration and management (34% - P); Other environment and natural resources management (33% - P); Rural markets (33% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : Ridley Nelson Robert Mark Lacey Soniya Carvalho IEGPS1 2. Project Objectives and Components: a. Objectives: The project development objective was stated in the same language in the Project Appraisal Document (PAD) and the legal agreement, as follows: "To promote sustainable natural resource management by establishing improved livestock production and marketing systems that would increase the incomes of herders and farmers in the project areas." The global environmental objective (GEO) (there was a component funded by the Global Environment Facility, GEF ) was stated as follows: "To maintain and nurture natural grassland ecosystems to enhance global environmental benefits." b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: This was a complex project with five components and many subcomponents, as follows : Component 1. Grassland Management and Forage Improvement . (Appraisal US$13.98 million, Actual US$11.62 million). This included forage and fodder production, grassland management and improvement, applied research, training, and extension support . It also supported an inventory of grassland ecosystems in selected areas to assess biodiversity changes; preparation and implementation of group -based grassland resource management plans; community-based integrated grassland management and pastoral development; strengthening existing grassland ecological monitoring systems; capacity building; and TA . Component 2. Livestock Production Improvement . (Appraisal US$67.75 million, Actual US$82.24 million). This included: fine wool and mutton nucleus breeding stations and multiplier stations; support for wool and mutton breeding households and fattening activities, beef cattle breeding households and fattening activities; and, household dairy production . These subcomponents were supported by investments in artificial insemination and veterinary stations and by applied research, training and extension . It included technical assistance to conserve native livestock breeds, inventory and assessment of native livestock, training and institutional capacity building, public awareness on agro-biodiversity, and selective breeding and maintenance of small flocks of native carpet -wool sheep, mutton sheep and Yak breeds . Component 3. Market Systems Development. (Appraisal US$10.20 million, Actual US$4.34 million). This component provided support for improved efficiency in marketing systems to provide better prices for farmers /herders. It included new and renovated markets, shearing stations, milk delivery infrastructure, loans for rural enterprises and entrepreneurs, support for farmers and herders groups, and development and establishment of mechanisms for public goods provision such as market information systems, market research, quality standards, and quality promotion. Component 4. Applied Research, Training and Extension . (Appraisal US$13.25 million, Actual US$4.99 million). This supported the first three components to introduce and spread new technologies . It included production of forage, breeding and raising livestock, production of high -quality livestock, and public extension services . The component included multiplication of indigenous grassland species, grassland ecology and ecosystem management, ecological surveys and environmental workshops, and applied research into conservation of wildlife habitat of global significance. Component 5. Project Management , Monitoring and Evaluation . (Appraisal US$5.75 million, Actual US$6.26 million). This included overall project management and M&E activities including operational costs, strengthening of the provincial, city, county and township level Project Management Offices . It included monitoring processes for adaptive integrated ecosystem management, and development and implementation of monitoring tools to measure changed carbon sequestration and biodiversity status in managed grasslands . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost. Close to 100% of project funds were utilized within the planned timeframe . US$1.47 million were canceled, US$0.85 million due to ineligible expenditures and US$ 0.62 million were unutilized. Financing and Borrower Contribution. The approximate share of actual financing was 30% Government/beneficiaries, 60% IBRD, and 10% GEF. Counterpart funding had been identified at appraisal as a substantial risk. The risk materialized in some counties where counterpart funding was either inadequate or not provided on a timely basis. This resulted in counterpart funding being only applied to components considered the highest priority of project management . These were mainly the directly productive investments that would increase livestock production quickly . However, for the project as a whole, counterpart funding reached 102% of that projected at appraisal (ICR Annex 1). There was a substantial reallocation of funds to Component 2. This came mainly from Component 4, reduced by 64%, which, being a more medium term research activity proved to be of less interest to the borrower than the more immediately productive Component 2. It also came from the marketing component, Component 3, which was reduced by 57% partly for a similar reason and partly because the wool component declined due to a global price decline . 3. Relevance of Objectives & Design: a. Relevance of Objectives: Rated High . The project was consistent with the 2006-2010 Country Partnership Strategy which aimed to manage resource scarcity and environmental challenges through improved land management and protection of global environmental goods. This land management objective is one of the listed ten highest priorities (CPS p. 16). The focus on enhancement of rural livelihoods and social infrastructure was consistent with the government's 11th five-year plan which called for increasing farmers incomes through sustainable agricultural development while mitigating environmental degradation . The project was relevant to the significant environmental degradation in the pastoral grazing areas of China . b. Relevance of Design: Rated Substantial . While there were some design weaknesses, including, particularly, complexity, these lie largely in the area of Quality at Entry rather than in the area of the relevance of design to the objectives and they are addressed in Section 8. The design was holistic, in keeping with the wide -ranging objectives, but it is partly this that led to the complexity. At the technical level the design appears to have been generally sound, addressing many of the areas suggested by the lessons of experience in livestock projects and consistent with the objectives . However, with respect to the sustainability element of the PDO, there is insufficient detail in the PAD to answer the question of whether the project design could have been expected to adequately address the conflict between shorter term productivity for income maximization and longer term land use sustainability, particularly in relation to poorer beneficiaries. 4. Achievement of Objectives (Efficacy): The extent to which the project promoted sustainable natural resource management . Rated Modest . In rating this objective, much depends on the interpretation of the word "promoted". This assessment makes the assumption that a program that merely promoted sustainable management without actually achieving it would not meet this objective. In other words, the assumption is that this objective called for an actual enhancement of natural resource management on the ground in the two project areas or at least early steps in this direction . Due to weak M&E, as noted by the ICR, there is limited evidence to assess this objective although data from government sources and research institutes and studies provided by the project team in a Working Paper on Technical Interventions filled some of the gaps . There is some evidence in this working paper on environmental enhancement but it is difficult to connect this with the project inputs or to relate changed vegetation biomass with changed animal productivity. For example, in Bole County biomass increased by 170%, and infiltration rate by 10% while sheep liveweight increased by 11% following 3 years of controlled grazing. These are plausible and promising changes. But to evaluate this for project impact one would need to know more about the results chain to project attribution including for example, what happened to any animals taken off to reduce stocking rates (slaughtered or moved) and what was their impact and was there a productivity or environmental change surrounding the controlled area. Similarly, in A'ding County, soil erosion is reported reduced by 70% after fencing and plant density rose from 268 to 392 plants/m2 with biomass rising from 3.7 to 5.3 tons per ha but it is not clear how that was linked to animal productivity increase or decrease . (To take the extreme case of total exclosure, ground cover can often be greatly increased but at the opportunity cost of zero total animal offtake . Evidence on such trade-off relationships was not provided). It is claimed in the working paper that carbon sequestration increased in a number of degraded grasslands. Again, this is plausible but this evidence is from research papers and the link to project investments is not shown. While under the standard guidelines the GEO is not rated, the impact in the direction of the global environmental objective is of evaluative interest . There is little evidence presented in the ICR that enables an assessment of how the interventions in natural grasslands resulted in the enhancement of global environmental benefits . However, there is some evidence from studies in the working paper cited above . Data quoted from Provincial PMOs and available information from local Grassland Station records, Animal Husbandry Bureau records, and County records indicate an aggregate reduction in gross land degradation averaging about 5% across the two provinces . However, although the ICR says that "the data quality is believed to be reasonably good ", it is not clear how this change in degradation was measured and to what extent, through a results chain, it could be attributed to the activities supported by the project . The extent to which the project improved livestock production that would increase the incomes of herders and farmers in the project areas. Rated Modest partly due to lack of evidence linking outcomes with project inputs and outputs. Due to weaknesses in M&E, there is limited evidence on the improvement of livestock production and on the results chain link between outputs and outcomes . There is some data from other sources outside the M&E system shown in the Technical Annex but here there are questions about attribution to the project inputs . There is evidence presented from institutional and national statistics that productivity and a number of livestock improvement indicators, such as calving rates, improved. However, the survey designs and analytical steps used to gather this evidence are not described in the ICR or the Technical Annex and again attribution to the project investments is unclear . For example, increases in the rate of cattle weight gains are reported but it is not clear how these weights were recorded and what the sample was. The ICR reports (p.15) that by the end of the project, per capita income of project beneficiaries had risen 160% higher than the baseline in Xinjiang but a comparable income increase percentage is not available for Gansu. However, in Gansu, the average net income increase for cattle breeding households was "up to" 1,870 Yuan (about US$280). An upper bound is not very useful for assessing outputs or outcomes because it is entirely influenced by the distribution . The models calculated for the financial rates of return indicate a range of 16% to 33% across the various enterprises except for fine wool for which prices declined and many farmers could not compete with import prices from more efficient systems . The extent to which the project improved marketing systems that would increase the incomes of herders and farmers in the project areas. Rated Modest . In this area, there was limited achievement because the project focused predominantly on productivity and did substantially less than planned in the development of marketing systems . The total cost of this component was reduced by 59% in 2007. This was partly due to the exogenous event of a decline in wool prices which had an impact on incentives and therefore on the wool marketing subcomponent but it was also due to greater borrower focus on the more immediate technology -driven productivity gains. 5. Efficiency: Rated Modest on balance given the uncertainties due to the limited outcome evidence and questions of attribution noted above. The overall ex post ERR for the project was estimated in the ICR at 24%. The provincial ERRs were similar at 26% for Gansu and 22% for Xinjiang. This compared with lower ex ante ERRs in the PAD of 15% for Gansu, 19% for Xinjiang and 17% for the whole project. However, the reason for the increases over the PAD ERRs was largely the very conservative original assumptions on technical parameters, e .g. weight gains, and higher prices for livestock products particularly beef and mutton . The ex post analysis involved 21 household models of different size and scale covering beef cattle fattening, beef cattle breeding, calf raising, fine wool production, dairy production, and mutton production. However, there is a question about how realistic and representative these are given the limited attributable outcome evidence presented in the ICR and the difficulty of connecting outcomes to inputs discussed in Section 4 above. At the PAD stage, prices were adjusted in the analysis to traded parity prices but, with China's accession to the WTO, the ICR concluded that domestic prices adequately reflected opportunity cost . (There has been debate among economists around the issue of whether parity traded values are adequately reflected in the exchange rate .) Rather than assert that the sector has achieved global parity prices, some calculations demonstrating this would have been useful. There is no evidence that would allow assessment of whether interventions were least cost . It could be concluded that, since the marketing and wool activities, and the applied research did not go very far, the initial resources put into those activities were probably not efficiently used . However, on wool, it would have been efficient to reduce the production with falling prices rather than continue at the same level . Thus that sub-component bears evidence of being responsive to price signals even if the initial preparation investments did not pay off . There were no project delays. The project closed as originally scheduled suggesting overall management efficiency . This is unusual for Bank projects globally that address pastoral rangeland issues where often social issues have overwhelmed productivity and environmental objectives . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal Yes 17% 100% ICR estimate Yes 24% 100% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Overall, Outcome is rated Moderately Unsatisfactory . Relevance of Objectives was High and Relevance of Design was Substantial. However, Efficacy of each of the three objectives was rated Modest, largely on the grounds of : (i) limited productivity and attribution evidence due to poor indicators and weak M&E implementation; (ii) limited evidence on environmental gains; (iii) weak or slow progress in marketing and applied research; and (iv) questions on productivity evidence related to how data reported in the ICR and in the technical paper was collected and analyzed. For similar reasons, due to these uncertainties about the benefit stream, Efficiency is rated Modest . While the GEO is not rated, there is limited evidence on global environmental impact attributable to the project . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: This Review rates risk as Moderate . On environmental risks, there are some doubts about whether the project really achieved meaningful gains on natural resource management sustainability (see Section 4 above) which was an important aspect of the objectives . On financial/profitability risks at the producer level, while no evidence is available on farm level financial sustainability, the free adoption of new livestock production technologies, which are reported to have resulted in productivity gains such as improved calving intervals, seem unlikely to regress . On social sustainability, there is insufficient evidence to assess the likely social sustainability of changed grazing practices some of which involved changes in land access . In most such situations there are likely to be both gainers and losers from legislative or community imposed controls over pasture utilization but evidence on this was not offered in the ICR. On political and ownership sustainability and on associated institutional support, it seems probable that commitment at both central and local levels will be sustained since improved natural resource management has been an important element of the borrower's strategy . a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: a. Quality at entry: The development objectives were consistent with the Government's and the Bank's strategies and there was good technical and policy skills mix provided . However, as noted in the ICR (p. 22), there was a long gestation period from identification to effectiveness of nearly three years, due to slow resolution of issues in a complex project, which increased project risks given the rapid evolution of the livestock sector . There was a lack of baseline data which raises questions about how the design responded to the situation on the ground . Also, design was complex and ambitious . Overall, M&E design was weak. Partly due to the preparation delays, there were shortcomings in the production models proposed in the PAD due to the changing market conditions . at -Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: There was a strong mix in the supervision teams of technical expertise and knowledge of China's livestock sector. The ICR reports (p. 22) that the missions were proactive in resolving emerging issues . These included procurement management issues and institutional capacity generally but also technical guidance . However, the ICR reports a number of serious lapses in supervision including failure to ensure a baseline study early enough, a particular problem where there has been no baseline established during preparation, failure to include in the supervision reports updated indicators, and failure to carry out a formal Mid Term Review . A formal MTR could have focused on the key performance indicators and could have improved the M&E system to enable the later assessment of project impact . In addition to the lapses mentioned in the ICR, it is not clear to IEG to what extent the compulsion in grazing regulations associated with project investments called for mitigation action under the Bank's Involuntary Resettlement safeguards policy (see Section 11 below). As noted above, usually in such resource control situations, there are gainers and losers . Quality of Supervision Rating : Unsatisfactory Overall Bank Performance Rating : Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: Government provided an enabling environment, making a number of policy changes which enhanced incentives. The Government at both central and local levels generally provided adequate resources during project preparation. There was commitment during implementation in the provision of good project staff . However, there were issues with the provision of counterpart funding from local governments who faced challenging financial constraints, although they were usually able to meet their commitments with some delay . This delay was generally covered by Central Government when needed . Government Performance Rating Satisfactory b. Implementing Agency Performance: The project management offices in both participating provinces were generally well staffed . There were dedicated professionals supported by a good team at the national level . There was good preparation and use of implementation manuals, guidelines, and plans . Training in management, procurement, and accounting practices enhanced capacity. There was generally good coordination with government departments and project management units were proactive in identifying and addressing problems . The ICR reports good cooperation with the Bank task teams. However, there were important weakness in the implementation and utilization of M&E partly caused by the complexity of project design and unclear indicators and targets . While this is partly attributable to the Implementing Agency it is also attributable to the Bank's design and supervision support . There was somewhat weak attention to training and adaptive research . This may have impacted on progress in the direction of environmental sustainability. There was one incident of noncompliance with fiduciary policies over a withdrawal claim covering the wrong livestock enterprise but this appears to have been relatively minor . Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The PAD indicators were excessively demanding in scale, over -designed, and too complex. There were eight Key Performance Indicators and some 30 output indicators. In implementation these were expanded further to 61 output indicators. Moreover, the quantified targets were not set at the design stage . The intention was to have a baseline survey which the PAD proposed should be achieved by assembling existing information from research station, government, and county records combined with data collected at project preparation . This was not done early enough so that the baseline data were missing . Moreover it is difficult to see how it could have been known at appraisal whether such information would be sufficient and relevant for baseline purposes and project design without at least partially assembling it. These weaknesses made implementation difficult . Responsibility for M&E was placed within the project management unit . For projects of this type, with longer -term environmental implications and potential benefits, this is problematic given the likelihood that the unit will be disbanded after project closing . b. M&E Implementation: There was routine project progress monitoring by the project management units . This was presented in progress reports and consolidated by the central unit into semiannual progress reports . At both the provincial levels there was a computerized MIS system but this only became operational in 2007 and was therefore of limited value . For environmental monitoring, there was an environmental coordinator in each of the project management units . Each sub-project in the Market Systems Development component was environmentally screened . There were surveys for social monitoring . There was supposed to be an annual project impact assessment by both the provincial project management units and an overall project impact assessment in the first, third and sixth years of implementation by an independent organization . Most of this did not happen. There was insufficient buy-in from Government partly due to the lack of baseline data . Even the social surveys undertaken in 2001, as reported by the ICR (p.11), appeared to be, in some instances, taken from recommendations by extension agents rather than from sampled farmer responses. A contract was awarded to an international consulting firm for monitoring but this only took place in 2008. The methodologies developed for M&E monitoring were not sound so that the surveys were of limited value. Overall, the lack of interest in M&E simply compounded the problems of design . c. M&E Utilization: There was limited utilization of the generally weak data . The M&E reports prepared by the consultants were little utilized. As a consequence of these problems project impact data was "well below appraisal expectations " (ICR p. 12). M&E Quality Rating : Negligible 11. Other Issues a. Safeguards: This category B project triggered safeguard policies on Environmental Assessment (OP 4.01), Indigenous Peoples (OP 4.10) , and the Safety of Dams (OP 4.37). No resettlement or land acquisition was anticipated as a result of project activities . However, based on discussions with the project team and statements in the ICR and PAD, IEG is unclear about the level of compliance with the Involuntary Resettlement safeguard policy . Unlike most other countries where grassland interventions have been supported by the Bank, there is a high level of legislative compulsion in range management in China through grazing bans and access restrictions . It is not clear from the ICR to what extent there were losers and gainers from such controls, which controls were associated with the project or were supported by it, and to what extent any losers through such regulation were adequately and formally compensated through the participatory processes and compensatory project investments . Apparently no mitigation plan was considered necessary and the PAD (p. 39) indicates that OP 4.12 on Involuntary Resettlement was not triggered . The ICR (p.13) finds that the environmental impacts of the project were largely positive . There were some difficulties in obtaining regular reports on environment from the two provinces . According to the ICR, in 2009, in Gansu, a survey provided a comprehensive assessment of the project impact on the environment . The report largely reported positive environmental impacts but made suggestions for some improvements . However, the evidence supporting those findings is not reported in the ICR . 39% of the project beneficiaries were indigenous peoples, the majority in Xinjiang Province . Multi Ethnic Group Development Strategies were prepared by both provinces and the ICR reports that they were closely followed . Minorities were consulted and participated in design and implementation . Training was organized in the languages of participating minorities. On dam safety, both provinces prepared Dam Safety Review Guidelines with procedures and responsibilities . While no dam water was used for livestock production, the Borrower assured the Bank that the guidelines had been followed. Since no dam water was used this activity appears to have been unnecessary . b. Fiduciary Compliance: The ICR reports that the Bank's fiduciary requirements were generally followed . However, since the counties had no prior experience of Bank procurement procedures there were some initial problems . The ICR reports that these were satisfactorily resolved and that, overall, procurement guidelines were followed and were in compliance with the Project Agreement. A Financial Management Manual was prepared by each province . The ICR reports that financial management practices were in compliance with the financial covenants . There is no evidence in the ICR about the status of audits . There was experience of slow reimbursements (the period from farmers pre-financing investments to receiving reimbursement was typically one year ) and one case of ineligible expenditure mentioned above . c. Unintended Impacts (positive or negative): None reported. d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately There is insufficient evidence of Satisfactory Unsatisfactory enhanced productivity and natural resource sustainability. This results in ratings of modest efficacy and efficiency. Risk to Development Negligible to Low Moderate There are questions about progress in Outcome : the direction of environmental sustainability, partly due to lack of evidence, which have implications for longer-term risk. Bank Performance : Moderately Unsatisfactory There were significant weaknesses in Satisfactory project design including complexity, M&E, and lack of a baseline and there were also weaknesses in supervision, including the lack of a formal MTR and continuing uncorrected issues with M&E. Borrower Performance : Moderately Moderately Note: Borrower Performance is rated Satisfactory Satisfactory Satisfactory in the ICR text but Moderately Satisfactory in the Data Sheet.. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR offers a number of lessons of which the following are the most important ( with some adjustment of language): 1. A project with wide coverage exacerbated by complex design overlaid with multiple productivity, income and environmental objectives, while useful in principle for a holistic approach, in practice presents a heavy burden for project management and supervision and runs the risk that, in implementation, some subcomponents will be given priority by the borrower and others neglected . 2. There are risks in project design and implementation in the inflexible application of production models and associated prescriptions specified in advance due to changing prices and market demand and crowding out by alternative financing. Project design needs to be flexible and nimble to accommodate such changes . 3. Investments in public good supporting services such as livestock breeding improvement, veterinary services and artificial insemination services can be very effective in enhancing livestock productivity and in benefiting large numbers of poor farmers/herders. 4. For effective M& E, indicators, baseline data and responsibility for data collection and analysis should be clearly defined at project appraisal. 14. Assessment Recommended? Yes No Why? This project may have had a substantial impact that could not be adequately demonstrated due to weak data arising partly from weak M&E. More evidence is needed to understand the outcomes . There are a number of questions about environmental impact that would benefit from further analysis . There may be Involuntary Resettlement safeguard questions related to grassland management controls . There may be lessons for the Bank related to incentives for Borrower interest in M&E and impact assessment . 15. Comments on Quality of ICR: This was a generally satisfactory ICR which had to deal with weak M&E evidence . It was open about the problems of the project and drew some useful lessons . There is an error in the project rating table (or in the text) with different rating of Borrower Performance. Somewhat more analysis of the possible environmental impacts would have been useful although this again was constrained by the limited data . The Technical Annex provided on request was a useful adjunct to the ICR, some more of this, as well as the 2009 Gansu study, could have been placed in an ICR annex. Some analytical evidence beyond an assertion to support not applying parity price adjustments as in the PAD economic analysis would have been useful . There was no information on project audits . a.Quality of ICR Rating : Satisfactory